Model training method and device, readable storage medium and computer program product

Abstract

The invention provides a model training method and device, and the method comprises the steps: an execution end device sends abnormal information to a control end device, the control end device determines a fault card number based on the abnormal information, the fault card number indicates a training card having a fault in a training cluster, and the fault card number indicates the training card having the fault in the training cluster; the training cluster is divided into n data parallel units for data parallel training, and n is an integer greater than 1; the control end equipment confirms parallel grouping information based on the fault card number, and the parallel grouping information is used for training new communication links of training cards in a cluster and training task redistribution of the training cards except the fault card in a data parallel unit where the fault card is located; and the execution end equipment controls the training cluster to carry out continuous training on the model based on the received parallel grouping information sent by the control end equipment. By implementing the embodiment of the invention, the model training efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a model training method and apparatus, a computer-readable storage medium, a computer program product, and a chip. Background Technology

[0002] The purpose of training an artificial intelligence (AI) model is to enhance its performance on real-world tasks by iteratively updating the model parameters (model weights) through inputting a large number of samples.

[0003] Since artificial intelligence opened the door to large language models (LLMs), the number of parameters in LLMs has continued to grow, reaching hundreds of billions. Since the size of a large language model can be measured by the number of parameters (i.e., parameter scale), a large language model with hundreds of billions of parameters is enormous. Training such a large language model consumes enormous computing and storage resources. A single GPU is far from meeting the computing power and memory requirements for training large models, making parallel training an inevitable method.

[0004] Parallel training refers to the process of training a model simultaneously on multiple processing units. Parallel training strategies are divided into data parallelism and model parallelism. Data parallelism addresses the problem of insufficient computing power on a single GPU; model parallelism addresses the problem of insufficient video memory on a single GPU. In practical applications, a combination of model parallelism and data parallelism is often used within a training cluster. How to improve training efficiency and more effectively utilize the computing resources of computing cards (or training cards) during parallel training is a key concern for technical personnel. Summary of the Invention

[0005] This application provides a model training method and device, a computer-readable storage medium, a computer program product, and a chip, which can improve training efficiency and more effectively utilize the computing resources of a computing card (or training card) during parallel training.

[0006] Firstly, a model training method is provided. This method can be executed by a control device (i.e., a computer device, such as a server) or by a module (e.g., a processor, chip, or chip system) applied to the control device. It can also be implemented by a logic node, logic module, or software capable of implementing all or part of the control device's functions. In this model training method, the control device first identifies the faulty card number, which indicates the training card that has failed in the training cluster. The training cluster is divided into n data parallel units for data parallel training; where n is an integer greater than 1. Then, the control device confirms parallel grouping information based on the faulty card number. This parallel grouping information is used to establish new communication links between training cards in the training cluster and to reallocate training tasks for training cards other than the faulty card in the data parallel unit where the faulty card resides. The control device then sends this parallel grouping information to the execution device.

[0007] As can be seen from the above embodiments, when the control device determines that a faulty card has occurred, it can reallocate the training tasks of the training cards other than the faulty card in the data parallel unit where the faulty card is located. This enables asymmetric parallel training to continue on the training cards other than the faulty card in the data parallel unit. This solves the problem that parallel training in the prior art requires the reliance on redundant AI servers and does not require the removal of the computing cards (or training cards) in the entire data parallel unit where the faulty card has occurred. This not only improves training efficiency but also makes more effective use of the computing resources of the computing cards (or training cards).

[0008] In one possible implementation, the parallel grouping information includes reference training card numbers required for repartitioning the training task. These reference training card numbers indicate that the training cards and faulty cards are located in different data parallel units but perform the same training task. The training cards indicated by these reference training card numbers provide model parameters for subsequent training of the training cluster. The training cards indicated by these reference training card numbers may include one or more training cards.

[0009] As can be seen from the above embodiments, when performing fault-based training, it is not necessary to reread the checkpoint. Instead, the model weights and other parameters that change with the training process can be obtained from other cards. This can solve the problem of wasted step training time between the most recent checkpoint and the current step, reduce the loss of training steps caused by reading the checkpoint, and further improve training efficiency.

[0010] In one possible implementation, each of the n data parallel units is divided into m pipelined parallel units; each pipelined parallel unit includes s tensor parallel units; the parallel grouping information includes the first reassignment grouping information of the training cards in the training cluster, the first reassignment grouping information includes k×s groups of first training card numbers, each group of first training card numbers includes n teams of first training card numbers, the first training card indicated by the first training card number in each team of the same group of first training card numbers undertakes the same training task, and the first training card indicated by the first training card number in different teams belongs to different data parallel units; m is an integer greater than 1; k is an integer greater than or equal to 1 and less than m; s is an integer greater than or equal to 1;

[0011] Each group of first training card numbers includes the first team training card numbers. The first training card indicated by the k×s first team training card numbers in the k×s group of first training card numbers undertakes the same training task as the target pipeline parallel unit. The target pipeline parallel unit includes pipeline parallel units that are in different data parallel units than the faulty card and undertake the same training task.

[0012] As can be seen, in the above embodiments, the control device can determine at least one pipeline parallel unit from the data parallel unit where the faulty card is located, and undertake the same training task as the target pipeline parallel unit. It generates corresponding first reassignment grouping information to instruct the execution device to implement asymmetric parallel training. This solves the problem that parallel training in the prior art requires the reliance on redundant AI servers, and does not require the removal of computing cards (or training cards) in the entire data parallel unit where the faulty card occurs. This not only improves training efficiency, but also makes more effective use of the computing resources of the computing cards (or training cards).

[0013] In one possible implementation, each of the multiple data parallel units is divided into m pipeline parallel units; each pipeline parallel unit includes s tensor parallel units; the parallel grouping information includes the second reassignment grouping information of the training cards in the training cluster, the second reassignment grouping information includes i×s groups of second training card numbers; each group of second training card numbers includes n teams of second training card numbers, and the data parallel units in which the second training cards indicated by the second training card numbers of different teams are located are different; the second training cards in each team of second training card numbers in the same group of second training card numbers undertake the same training task; m is an integer greater than 1; i is the number of groups after regrouping the number of layers of the training model based on the number of layers of the training model and the training cards in the pipeline parallel units without faults in the data parallel unit where the faulty card is located; i is greater than or equal to m; s is an integer greater than or equal to 1.

[0014] Each group of second training card numbers includes a second team training card number. The second training card indicated by the i×s second team training card numbers in the i×s group of second training card numbers is in the same data parallel unit as the faulty card, but not in the pipeline parallel unit where the faulty card is located. The i×s group of second training card numbers includes a target group of second training card numbers. The second training card indicated by the second team training card numbers in the target group of second training card numbers undertakes the same training task as the target pipeline parallel unit. The target pipeline parallel unit includes pipeline parallel units that are in different data parallel units from the faulty card but undertake the same training task.

[0015] As can be seen from the above embodiments, the control device can regroup the training model layers in the remaining pipeline parallel units based on the data parallel unit where the faulty card is located. For example, the total training tasks of the original data parallel unit can be divided equally and allocated to the training cards in the remaining pipeline parallel units. Accordingly, the training model layers are regrouped. Based on the number of regrouped groups, corresponding second reassignment grouping information is generated to instruct the execution device to implement asymmetric parallel training. This solves the problem that parallel training in the prior art requires redundant AI servers and does not require removing the computing cards (or training cards) in the entire data parallel unit where the faulty card is located. This not only improves training efficiency but also makes more effective use of the computing resources of the computing cards (or training cards).

[0016] In one possible implementation, each of the n data parallel units is divided into m pipelined parallel units, and each pipelined parallel unit includes s tensor parallel units; the faulty training card corresponds to at least one tensor parallel unit in a pipelined parallel unit; the parallel grouping information includes the third reassignment grouping information of the training cards in the training cluster, the third reassignment grouping information includes q groups of third training card numbers, each group of third training card numbers includes n teams of third training card numbers, and the third training card numbers of different teams indicate that the third training card is located in a different data parallel unit; m is an integer greater than 1; q is an integer greater than or equal to 1 and less than s; s is an integer greater than 1.

[0017] Each group of third training card numbers includes the third team training card number. The third training card indicated by the q third team training card numbers in the q groups of third training card numbers is in the same pipeline parallel unit as the faulty card. The third training card indicated by the q third team training card numbers undertakes the same training task as the target tensor parallel unit. The target tensor parallel unit includes tensor parallel units that are in different data parallel units than the faulty card and undertake the same training task.

[0018] As can be seen from the above embodiments, for the case where each of the m pipeline parallel units is divided into s tensor parallel units, the control device can determine at least one tensor parallel unit from the pipeline parallel unit where the faulty card is located, and have it undertake the same training task as the target tensor parallel unit. It generates corresponding third-level allocation grouping information to instruct the execution device to implement asymmetric parallel training. This solves the problem that parallel training in the prior art requires the reliance on redundant AI servers, and does not require the removal of the computing card (or training card) in the entire data parallel unit where the faulty card occurs. This not only improves training efficiency, but also makes more effective use of the computing resources of the computing card (or training card).

[0019] In one possible implementation, each of the n data parallel units is divided into m pipelined parallel units, and each pipelined parallel unit includes s tensor parallel units; the faulty training card corresponds to at least one tensor parallel unit in a pipelined parallel unit; the parallel grouping information includes the fourth reassignment grouping information of the training cards in the training cluster, which includes j groups of fourth training card numbers, each group of fourth training card numbers includes n teams of fourth training card numbers, and the data parallel units in which the fourth training cards in different teams of fourth training card numbers are located are different; the fourth training cards in each team of fourth training card numbers in the same group of fourth training card numbers undertake the same training task; m is an integer greater than 1; s is an integer greater than 1; j is the number of groups after regrouping the training tasks undertaken in the target tensor parallel unit based on the training cards in the pipelined parallel unit where the faulty card is located without faults; j is greater than or equal to s.

[0020] Each group of fourth training card numbers includes the fourth team training card number. The fourth training card indicated by the j fourth team training card numbers in the j groups of fourth training card numbers is in the same pipeline parallel unit as the faulty card. The j groups of fourth training card numbers include the target group fourth training card number. The fourth training card indicated by the fourth team training card number in the target group fourth training card number undertakes the same training task as the target tensor parallel unit. The target tensor parallel unit includes tensor parallel units that are in different data parallel units than the faulty card and undertake the same training task.

[0021] As can be seen in the above embodiments, for the case where each of the m pipeline parallel units is divided into s tensor parallel units, the control device can regroup the training tasks undertaken by the remaining tensor parallel units in the target tensor parallel unit (which can also be understood as the training tasks undertaken by the pipeline parallel unit where the faulty card is located before the fault) based on the pipeline parallel unit where the faulty card is located. For example, the total training tasks of the original pipeline parallel unit are divided equally and allocated to the training cards in the remaining tensor parallel units. Accordingly, the training tasks undertaken by the target tensor parallel unit are regrouped. Based on the number of regrouped groups, the corresponding fourth reassignment grouping information is generated to instruct the execution device to implement asymmetric parallel training. This solves the problem that parallel training in the prior art requires the reliance on redundant AI servers and does not require the removal of the computing cards (or training cards) in the entire data parallel unit where the faulty card is located. This not only improves training efficiency but also makes more effective use of the computing resources of the computing cards (or training cards).

[0022] Secondly, a model training method is provided. This method can be executed by an execution-end device (i.e., a computer device, such as an AI server) or by a module applied to the execution-end device (e.g., a processor, chip, or chip system). It can also be implemented by a logic node, logic module, or software capable of implementing all or part of the functions of the execution-end device. In this model training method, the execution-end device sends an exception message to the control-end device; the execution-end device receives parallel packet information sent by the control-end device. This parallel packet information is used to establish new communication links between training cards in the training cluster and to reallocate training tasks for training cards other than the faulty card in the data parallel unit where the faulty card is located; the training cluster performs data parallel training through n data parallel units; where n is an integer greater than 1; the execution-end device controls the training cluster to continue training the model based on the parallel packet information.

[0023] As can be seen from the above embodiments, after the control device determines that a faulty card has occurred, it sends parallel grouping information to the execution device. By reallocating the training tasks of the training cards other than the faulty card in the data parallel unit where the faulty card is located, asymmetric parallel training can be achieved on the training cards other than the faulty card in the data parallel unit. This solves the problem that parallel training in the prior art requires the reliance on redundant AI servers and does not require the removal of computing cards (or training cards) in the entire data parallel unit where the faulty card has occurred. This not only improves training efficiency but also makes more effective use of the computing resources of computing cards (or training cards).

[0024] In one possible implementation, each of the n data parallel units is divided into m pipelined parallel units; each pipelined parallel unit includes s tensor parallel units; the parallel grouping information includes the first reassignment grouping information of the training cards in the training cluster, the first reassignment grouping information includes k×s groups of first training card numbers, each group of first training card numbers includes n teams of first training card numbers, the first training card indicated by the first training card number in each team of the same group of first training card numbers undertakes the same training task, and the first training card indicated by the first training card number in different teams belongs to different data parallel units; m is an integer greater than 1; k is an integer greater than or equal to 1 and less than m; s is an integer greater than or equal to 1;

[0025] The method of controlling the training cluster to continue training the model based on the parallel grouping information includes: establishing a connection between the target first training card and the target pipeline parallel unit;

[0026] Each group of first training card numbers includes the first team training card number. The target first training card is the first training card indicated by the k×s first team training card numbers included in the k×s group of training card numbers. The target first training card and the target pipeline parallel unit undertake the same training task. The target pipeline parallel unit includes pipeline parallel units that are in different data parallel units than the fault card and undertake the same training task.

[0027] As can be seen from the above embodiments, at least one pipeline parallel unit can be determined from the data parallel unit where the faulty card is located through the first reassignment group information. This unit undertakes the same training task as the target pipeline parallel unit, thereby achieving asymmetric parallel training. This solves the problem that in the prior art, parallel training requires the reliance on redundant AI servers and does not require the removal of computing cards (or training cards) from the entire data parallel unit where the faulty card is located. This not only improves training efficiency but also makes more effective use of the computing resources of the computing cards (or training cards).

[0028] In one possible implementation, the parallel grouping information further includes a reference training card number required to repartition the training task, the reference training card number indicating a training card (one or more) that is in a different data parallel unit and performs the same training task as the faulty card; the method further includes:

[0029] The model parameters of the current model training in the training card corresponding to the reference training card number are sent to the first training card indicated by the k×s first team training card number.

[0030] As can be seen from the above embodiments, when performing fault-based training, it is not necessary to reread the checkpoint. Instead, the model weights and other parameters that change with the training process can be obtained from other cards. This can solve the problem of wasted step training time between the most recent checkpoint and the current step, reduce the loss of training steps caused by reading the checkpoint, and further improve training efficiency.

[0031] In one possible implementation, controlling the training cluster to continue training the model based on the parallel grouping information further includes:

[0032] The training sample of the faulty card is input into the first training card that undertakes the training task in the first training card of the k×s first team training card number;

[0033] After completing the gradient calculation, the first training card indicated by the k×s first team training card number is synchronized with the training card in the target pipeline parallel unit.

[0034] As can be seen, in the above embodiments, the execution device can complete the training of the training samples of the original faulty card based on the first training card indicated by the k×s first team training card numbers configured by the first redistribution grouping information, and perform gradient synchronization with the training card in the target pipeline parallel unit after completing the gradient calculation, so as to achieve asymmetric parallel training. This solves the problem that parallel training in the prior art requires the reliance on redundant AI servers, and does not require the removal of the computing card (or training card) in the entire data parallel unit where the faulty card has occurred. This not only improves training efficiency, but also makes more effective use of the computing resources of the computing card (or training card).

[0035] In one possible implementation, each of the multiple data parallel units is divided into m pipeline parallel units; each pipeline parallel unit includes s tensor parallel units; the parallel grouping information includes the second reassignment grouping information of the training cards in the training cluster, which includes i×s groups of second training card numbers; each group of second training card numbers includes n teams of second training card numbers, and the second training card numbers of different teams indicate that the second training card is located in a different data parallel unit; the second training card numbers of each team in the same group of training card numbers indicate that the second training card undertakes the same training task; m is an integer greater than 1; i is the number of groups after regrouping the number of layers of the training model based on the number of layers of the training model and the training cards in the pipeline parallel units without faults in the data parallel unit where the faulty card is located; i is greater than or equal to m; s is an integer greater than or equal to 1.

[0036] The method of controlling the training cluster to continue training the model based on the parallel grouping information includes: establishing connections between the second training cards of each team indicated by the second training card number in each group;

[0037] Each group of second training card numbers includes a second team training card number. The second training card indicated by the i×s second team training card numbers in the i×s group of second training card numbers is in the same data parallel unit as the faulty card, but not in the pipeline parallel unit where the faulty card is located. The i×s group of second training card numbers includes a target group of second training card numbers. The second training card indicated by the second team training card numbers in the target group of second training card numbers undertakes the same training task as the target pipeline parallel unit. The target pipeline parallel unit includes pipeline parallel units that are in different data parallel units from the faulty card but undertake the same training task.

[0038] As can be seen from the above embodiments, the connection between the second training cards of each team indicated by the second training card number in each group of the second training card number is established through the second reassignment grouping information, so as to realize asymmetric parallel training. This solves the problem that the parallel training in the prior art needs to rely on redundant AI servers, and does not need to remove the computing cards (or training cards) in the entire data parallel unit where the card has failed. This not only improves training efficiency, but also makes more effective use of the computing resources of the computing cards (or training cards).

[0039] In one possible implementation, the parallel grouping information further includes a reference training card number required to repartition the training task, the reference training card number indicating a training card that is in a different data parallel unit than the faulty card and performs the same training task; the method further includes:

[0040] Send the model parameters of the current model training in the training card corresponding to the reference training card number to the second training card corresponding to the second training card number in the same group.

[0041] As can be seen from the above embodiments, when performing fault-based training, it is not necessary to reread the checkpoint. Instead, the model weights and other parameters that change with the training process can be obtained from other cards. This can solve the problem of wasted step training time between the most recent checkpoint and the current step, reduce the loss of training steps caused by reading the checkpoint, and further improve training efficiency.

[0042] In one possible implementation, controlling the training cluster to continue training the model based on the parallel grouping information further includes:

[0043] The training sample of the fault card is input into the first training card in the second training card of the second training card number, which is included in the second training card number of the target group, and is responsible for the training task.

[0044] After completing the gradient calculation, the second training card indicated by the second training card number included in the second training card number of the target group is gradient synchronized with the training card in the parallel unit of the target pipeline.

[0045] As can be seen from the above embodiments, the execution device can complete the training of the training samples of the original faulty card and other training samples based on the second training card indicated by the second training card number of the i×s group configured by the second redistribution grouping information, so as to achieve asymmetric parallel training. This solves the problem that parallel training in the prior art requires the reliance on redundant AI servers and does not require the removal of the computing card (or training card) in the entire data parallel unit of the faulty card. This not only improves training efficiency, but also makes more effective use of the computing resources of the computing card (or training card).

[0046] In one possible implementation, each of the n data parallel units is divided into m pipelined parallel units, and each pipelined parallel unit includes s tensor parallel units; the faulty training card corresponds to at least one tensor parallel unit in a pipelined parallel unit; the parallel grouping information includes the third reassignment grouping information of the training cards in the training cluster, the third reassignment grouping information includes q groups of third training card numbers, each group of third training card numbers includes n teams of third training card numbers, and the third training card numbers of different teams indicate that the third training card is located in a different data parallel unit; m is an integer greater than 1; q is an integer greater than or equal to 1 and less than s; s is an integer greater than 1.

[0047] The method of controlling the training cluster to continue training the model based on the parallel grouping information includes: establishing a connection between the target third training card and the target tensor parallel unit;

[0048] Each group of third training card numbers includes the third team training card number. The target third training card is the third training card indicated by the q third team training card numbers included in the q group of training card numbers. The target third training card and the fault card are in the same pipeline parallel unit and undertake the same training task as the target tensor parallel unit. The target tensor parallel unit includes tensor parallel units that are in different data parallel units than the fault card and undertake the same training task.

[0049] As can be seen from the above embodiments, for the case where each of the m pipeline parallel units is divided into s tensor parallel units, a connection is established between the target third training card and the target tensor parallel unit through the third allocation grouping information to achieve asymmetric parallel training. This solves the problem that in the prior art, parallel training requires the reliance on redundant AI servers and does not require the removal of computing cards (or training cards) in the entire data parallel unit where a faulty card has occurred. This not only improves training efficiency but also makes more effective use of the computing resources of the computing cards (or training cards).

[0050] In one possible implementation, the parallel grouping information further includes a reference training card number required to repartition the training task, the reference training card number indicating a training card (one or more) that is in a different data parallel unit and performs the same training task as the faulty card; the method further includes:

[0051] Send the model parameters of the current model training in the training card corresponding to the reference training card number to the third training card indicated by the q third training card numbers.

[0052] As can be seen from the above embodiments, when performing fault-based training, it is not necessary to reread the checkpoint. Instead, the model weights and other parameters that change with the training process can be obtained from other cards. This can solve the problem of wasted step training time between the most recent checkpoint and the current step, reduce the loss of training steps caused by reading the checkpoint, and further improve training efficiency.

[0053] In one possible implementation, each of the n data parallel units is divided into m pipelined parallel units, and each pipelined parallel unit includes s tensor parallel units; the faulty training card corresponds to at least one tensor parallel unit in a pipelined parallel unit; the parallel grouping information includes the fourth reassignment grouping information of the training cards in the training cluster, which includes j groups of fourth training card numbers, each group of fourth training card numbers includes n teams of fourth training card numbers, and the data parallel units in which the fourth training cards in different teams of fourth training card numbers are located are different; the fourth training cards in each team of fourth training card numbers in the same group of fourth training card numbers undertake the same training task; m is an integer greater than 1; s is an integer greater than 1; j is the number of groups after regrouping the training tasks undertaken in the target tensor parallel unit based on the training cards in the pipelined parallel unit where the faulty card is located without faults; j is greater than or equal to s.

[0054] The method of controlling the training cluster to continue training the model based on the parallel grouping information includes: establishing connections between the fourth training cards of each team indicated by the fourth training card number in each group;

[0055] Each group of fourth training card numbers includes the fourth team training card number. The fourth training card indicated by the j fourth team training card numbers in the j groups of fourth training card numbers is in the same pipeline parallel unit as the faulty card. The j groups of fourth training card numbers include the target group fourth training card number. The fourth training card indicated by the fourth team training card number in the target group fourth training card number undertakes the same training task as the target tensor parallel unit. The target tensor parallel unit includes tensor parallel units that are in different data parallel units than the faulty card and undertake the same training task.

[0056] As can be seen in the above embodiments, for the case where each of the m pipeline parallel units is divided into s tensor parallel units, the connection between the fourth training cards in each group indicated by the fourth training card number in the fourth training card number of each group is established through the fourth reassignment grouping information to achieve asymmetric parallel training. This solves the problem that in the prior art, parallel training requires the reliance on redundant AI servers and does not require the removal of computing cards (or training cards) in the entire data parallel unit where a faulty card has occurred. This not only improves training efficiency but also makes more effective use of the computing resources of the computing cards (or training cards).

[0057] In one possible implementation, the parallel grouping information further includes a reference training card number required for repartitioning the training task, the reference training card number indicating a training card that is in a different data parallel unit than the faulty card and performs the same training task; the method further includes:

[0058] Send the model parameters of the current model training in the training card corresponding to the reference training card number to the third training card indicated by the fourth training card number in the same group.

[0059] As can be seen from the above embodiments, when performing fault-based training, it is not necessary to reread the checkpoint. Instead, the model weights and other parameters that change with the training process can be obtained from other cards. This can solve the problem of wasted step training time between the most recent checkpoint and the current step, reduce the loss of training steps caused by reading the checkpoint, and further improve training efficiency.

[0060] Thirdly, the model training device includes units or modules for implementing the execution steps of the control device or execution device as described in any of the possible embodiments of the first to second aspects. That is, the model training device may be a control device or execution device, or the aforementioned modules (e.g., processors, chips, or chip systems), or it may be a logic node, logic module, or software capable of implementing all or part of the aforementioned control device or execution device.

[0061] Fourthly, a model training apparatus includes at least one processor; wherein the at least one processor is configured to execute the methods in any of the possible implementations of the first to second aspects. The model training apparatus may be a control device or an execution device. The at least one processor may execute computer programs or instructions stored in memory to cause the described methods to be executed. The memory may be included in the model training apparatus or located externally to the model training apparatus. Furthermore, the model training apparatus may also include an interface.

[0062] Fifthly, a communication system is provided, comprising a control device and an execution device. The control device is used to perform the method as described in any one of the first aspects, and the execution device is used to perform the method as described in any possible embodiment of the second aspect.

[0063] In a sixth aspect, a computer-readable storage medium is provided, which stores computer instructions that, when executed, cause a computer to perform a method as described in any possible implementation of the first or second aspect.

[0064] In a seventh aspect, a computer program product is provided, comprising: computer program code, which, when executed by a computer, causes the computer to perform a method as described in any possible implementation of the first or second aspect.

[0065] Eighth aspect, a chip is provided, the chip including at least one processor and an interface, the processor being configured to read and execute instructions stored in a memory, which, when executed, cause the chip to perform a method as described in any possible implementation of the first or second aspect. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0067] Figure 1 This is a schematic diagram of a training cluster employing a hybrid parallel grouping strategy provided in an embodiment of this application;

[0068] Figure 2 This is a schematic diagram of a training cluster employing a hybrid parallel grouping strategy, provided in another embodiment of this application;

[0069] Figure 3 This is a schematic diagram of a model training process using a hybrid parallel grouping strategy provided in an embodiment of this application;

[0070] Figure 4 This is a schematic diagram illustrating a training cluster failure as provided in an embodiment of this application;

[0071] Figure 5 This is a schematic diagram illustrating resource waste caused by reading fault points, provided in an embodiment of this application.

[0072] Figure 6 This is a schematic diagram of a system architecture for training using a hybrid parallel grouping strategy, provided in an embodiment of this application.

[0073] Figure 7A This is a schematic diagram of a model training method provided in an embodiment of this application;

[0074] Figure 7B This is a schematic diagram of a newly established communication connection and the training task partitioning results provided in an embodiment of this application;

[0075] Figure 7C This is a schematic diagram of a newly established communication connection and the training task partitioning result provided in another embodiment of this application;

[0076] Figure 8A This is a schematic diagram of a model training method provided in another embodiment of this application;

[0077] Figure 8B This is a schematic diagram of a newly established communication connection and the training task partitioning result provided in another embodiment of this application;

[0078] Figure 8C This is a schematic diagram of a newly established communication connection and the training task partitioning result provided in another embodiment of this application;

[0079] Figure 9 This is a schematic diagram of a newly established communication connection and the training task partitioning result provided in another embodiment of this application;

[0080] Figure 10 This is a schematic diagram of a newly established communication connection and the training task partitioning result provided in another embodiment of this application;

[0081] Figure 11 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application. Detailed Implementation

[0082] The technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more, and "multiple types" refers to two or more.

[0083] It should be understood that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0084] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0085] In practical applications, a hybrid parallel training strategy that combines model parallelism and data parallelism can often be used in a training cluster to achieve efficient parallel training.

[0086] like Figure 1 The diagram illustrates a training cluster employing a hybrid parallel training strategy, where the training data is divided into three parts and distributed across three different data parallel units to perform the training task. Figure 1 A training cluster consisting of three data parallel units is shown. The data parallel numbers of the three data units are dp1, dp2, and dp3, respectively. In this embodiment, for ease of description, the data parallel numbers are used to refer to the corresponding data parallel units. Each data parallel unit stores a complete model, which is used for model training.

[0087] Each data parallel unit partitions the model into different layers, and then distributes these partitions to different pipeline parallel units. It's important to note that while the number of layers in the model can be used for pipeline parallel unit partitioning, there isn't a one-to-one correspondence between each layer and a pipeline parallel unit. For example, a model with 5 layers can be divided into three parts: the first part contains one layer, and the second and third parts each contain two layers. These three parts are then assigned to different pipeline parallel units. In a training cluster employing a hybrid parallel training strategy, each data parallel unit partitions the model into the same way, such as... Figure 1As shown in the diagram, in the three data parallel units, the model is similarly divided into three parts, each part being assigned to a different pipeline parallel unit, i.e., assigned to three pipeline parallel units. The pipeline parallel unit numbers are pp1, pp2, and pp3, respectively. In this embodiment, for ease of description, the pipeline parallel unit number is used to refer to the corresponding pipeline parallel unit in the following text. Each data parallel unit in the same training cluster is completely identical except for the training data it needs to process. Figure 1 The pp1 of dp1 and pp1 of dp2 are completely identical except for the training data they need to process. For example, the initial model parameters and calculation methods are the same.

[0088] Each pipeline parallel unit can also contain multiple tensor parallel units, such as Figure 1 As shown, each pipeline parallel unit is divided into two tensor parallel units, with tensor parallel numbers tp1 and tp2 respectively. In this embodiment, for ease of description, the tensor parallel number is used to refer to the corresponding tensor parallel unit. It can be understood that the tensor parallel units in pipeline parallel units with the same pipeline parallel number in different data parallel units are the same, such as tp1 in pp1 in dp1 and tp1 in pp1 in dp2 are the same.

[0089] The following is an example Figure 1 The example shown is a training cluster employing a hybrid parallel training strategy, which will be used to introduce data parallelism and model parallelism:

[0090] I. Data Parallelism (DP)

[0091] Data parallelism can refer to dividing the training data into N parts and distributing them to N data parallel units to perform the training task, where N is a positive integer. For example, the training data can be divided into three parts and distributed to N data parallel units such as... Figure 1 In the three data parallel units dp1, dp2, and dp3 shown, each data parallel unit processes one set of training data, and all data parallel units store the complete model. Data parallelism can distribute the training task among multiple data parallel units. This strategy reduces the amount of training data processed by a single GPU in each data parallel unit, thus reducing the computational power requirements of the training task on a single GPU. However, the trade-off is that when the model updates parameters, it needs to aggregate the gradients from all devices, introducing additional communication. Gradient aggregation can be achieved by averaging the gradients calculated by devices performing the same training task in different data parallel units. For example, in... Figure 1In this algorithm, gradient aggregation is performed on dp1, dp2, and dp3. This involves calculating the average of the gradients of pp1 in dp1, pp1 in dp2, and pp1 in dp3. This average is used as the gradient of pp1 in each data parallel unit for updating model parameters. Similarly, the average of the gradients of pp2 in dp1, pp2 in dp2, and pp2 in dp3 is calculated and used as the gradient of pp2 in each data parallel unit for updating model parameters. The gradient can be a vector representing direction, with the same shape as the model parameters. Each element in the gradient represents the partial derivative of the loss value calculated by the loss function with respect to the model weights.

[0092] II. Model Parallelism

[0093] Model parallelism can refer to dividing the model into segments within each data parallel unit and distributing them to different devices (such as devices in different pipeline parallel units and / or tensor parallel units) to collaboratively perform training tasks. This reduces the number of models distributed to each device and decreases the consumption of single-card memory on each device during model training. However, the cost is the introduction of additional communication during each forward and backward computation in the model training process.

[0094] The goal of forward computation is to calculate the training loss of the model. Each data parallel unit acquires different training data and inputs it into the computation card (a type of training card) pp1 in each data parallel unit. The input on pp1 is calculated according to the model structure (i.e., operators that specify the computation method) and the weights of the model pre-loaded onto the computation card (e.g., matrix multiplication where the input is multiplied by the weights). Each pipeline parallel unit can have multiple layers of models, and the output of the last layer is the output of the current pipeline parallel unit. Figure 1As shown, taking pp1 as the first part of the training task, pp2 as the second part of the training task, and pp3 as the third part of the training task as an example, the second part of the training task is carried out after the first part of the training task is completed, and the third part of the training task is carried out after the second part of the training task is completed. Each of the first, second, and third parts of the training task can contain a multi-layer model. In the forward computation, the output of pp3 is ultimately obtained. This output is then input into the loss function to obtain the forward computation loss (a floating-point number). The activation values ​​from the forward computation (intermediate variables in the forward computation process, such as the input of a matrix multiplication) are retained for backward computation. The purpose of backward computation is to calculate the partial derivatives of the model weights with respect to the loss (i.e., to calculate the gradient). Starting from the loss, and based on the structure of the backward computation, the inputs are the loss, the activation values ​​calculated by pp3, and the model parameters on pp3. The outputs are the gradients of the model weights and the backward output of pp3. The gradients of each model weight are calculated sequentially from back to front. Each pipeline parallel unit can have multiple layers of models. After a pipeline parallel unit has calculated the gradients of all layers, the backward output of the first layer becomes the backward output of the current pipeline parallel unit. For example, pp2 receives the backward output of pp3. Based on the activation values ​​of pp2 and the backward output of pp3, the gradient and backward output of pp2 are calculated, and so on, calculating the gradients of all pipeline parallel units in the same way.

[0095] Model parallelism can be divided into pipeline parallelism (PP) and tensor parallelism (TP). The following sections will introduce pipeline parallelism and tensor parallelism respectively.

[0096] 1. Pipeline parallelism refers to dividing a model into different layers and assigning them to different pipelined parallel units for processing. When the parallelism of pipelined parallelism is M, the model is divided into M groups according to the neural network layers and assigned to M different pipelined parallel units for processing. Each pipelined parallel unit can also include multiple layers of the model, where M is a positive integer. For example, ... Figure 1 As shown, each data parallel unit is divided into 3 pipeline parallel units (i.e., M equals 3). If the model has 5 layers, the three pipeline parallel units can include one layer model, two layers model, two layers model, or they can include one layer model, one layer model, and three layers model, respectively.

[0097] 2. Tensor parallelism can refer to dividing the same layer of a model into s groups and distributing them to s different tensor parallel units for processing, where s is a positive integer. For example, ... Figure 1 As shown, pp1 in dp1 can be divided into two tensor parallel units (i.e., s equals 2).

[0098] In practical applications, a hybrid parallel training strategy that combines tensor parallelism, data parallelism, and pipeline parallelism (TP / DP / PP) is often adopted.

[0099] Hybrid parallel training strategies significantly improve computational efficiency through a distributed approach, making the training of large language models possible. However, in parallel computing, various computing units (such as tensor parallel units, pipelined parallel units, or data parallel units where computing cards reside) are interdependent. If one computing card fails, the entire cluster cannot continue training. As the number of computing cards in the cluster increases, even if the probability of a single computing card failing remains the same, the failure rate of the entire cluster will increase. Moreover, the increase in cluster size also leads to a greater likelihood of communication failures, such as optical module failures and switch failures. In a scenario with thousands of computing cards, a single training session can result in hundreds of failures, severely impacting the efficiency of training large models. Large models refer to machine learning models with large-scale parameters and complex computational structures; large models include large language models.

[0100] When a distributed parallel training cluster experiences an abnormal interruption, it is generally handled manually. After the fault is identified and resolved manually, all computing cards in the cluster load the most recent checkpoint from the checkpoint (ckpt) file saved during training and restart training. This process of troubleshooting, resolving the fault, and restarting training is time-consuming, sometimes taking several hours or even days, which seriously affects the training efficiency of large-scale clusters.

[0101] A schematic diagram of the training cluster for the most mainstream and standard parallel training framework for large language models in the world is shown below. Figure 2 As shown, the training dataset is divided into three parts and distributed to three different data parallel units to perform the training task. Figure 2 The diagram illustrates a training cluster consisting of three data-parallel units, designated dp1, dp2, and dp3. Each unit contains the complete model required for training. Within each unit, the model is divided into three parts based on its multi-layered structure. Each part can include multiple layers. The partitioned model is then distributed across multiple pipelined parallel units, such as... Figure 2As shown, the model is divided into four parts, each of which can contain multiple layers; for example, pp1 can contain three layers. Each part is assigned to a pipelined parallel unit, i.e., four pipelined parallel units. The pipelined parallel units are numbered pp1, pp2, pp3, and pp4. Each pipelined parallel unit can include only one training card (including the computation card) or multiple training cards. Each data parallel unit within the same training cluster is identical except for the training data it processes. Figure 2 In dp1, pp1 is completely identical to pp1 of dp2, except for the training data they need to process; for example, the initially allocated model parameters and calculation methods are the same. Each pipeline parallel unit can also contain multiple tensor parallel units, as detailed in the reference. Figure 1 As shown, it can be understood that the tensor parallel units in the same pipelined parallel encoding pipelined parallel unit in different data parallel units correspond to the same type. For example, such as... Figure 1 The tp1 of pp1 in dp1 is the same as the tp1 of pp1 in dp2.

[0102] as follows:

[0103] S301: Initialize the training cluster used for model training.

[0104] Each training card in the training cluster has an assigned data parallelism number, pipeline parallelism number, and tensor parallelism number. The execution device divides the model parameters of the large language model according to the assigned encoding and stores them on the corresponding training card. Each data parallelism unit contains a complete set of model parameters (or model weights).

[0105] For example, such as Figure 2 As shown, the training cluster is divided into 3 data parallel units, and the data parallel codes of the training cards in these 3 data parallel units are dp1, dp2, and dp3, respectively. Each data parallel unit is further divided into 4 pipeline parallel units, and the pipeline parallel numbers of the training cards in these 4 pipeline parallel units are pp1, pp2, pp3, and pp4, respectively. It can be understood that each pipeline parallel unit can also be divided into multiple tensor parallel units, and the training cards in each tensor parallel unit will also be assigned a corresponding tensor parallel number.

[0106] S302. Based on sample text, train the cluster to train the model.

[0107] The sample text is input into a training cluster consisting of N data parallel units, and then distributed among the N data parallel units. For example, X sample texts are divided into three parts and distributed among the N data parallel units. Figure 2The data parallel encoding is divided into three data parallel units: dp1, dp2, and dp3, where X is a positive integer, and the allocation method can be equal distribution. Each data parallel unit trains the training cluster based on the allocated sample text.

[0108] It should be noted that the training samples in the various embodiments of this application are exemplified by text samples, but are not limited to text samples. They can also be image or picture samples, video samples, audio samples, etc.; this application does not impose any restrictions.

[0109] The process of training a model based on sample text in the training cluster is a loop. The loop stops when all sample texts have been trained or when a training card in the cluster fails. The specific process of this loop is as follows:

[0110] S3021, N data parallel units compute gradients in parallel.

[0111] The training cluster is divided into N data parallel units. X sample texts are divided into N sample text sets. Each of the N data parallel units receives one sample text set and computes the gradient of the model weights in parallel based on the received sample text sets. For example, at the beginning of each node (step), the sample texts used for cluster training (i.e., the sample texts in the received sample text sets) are input to pp1 in each data parallel unit. For instance, the X sample texts are divided into 3 sample text sets and input to pp1 respectively. Figure 2 In dp1, dp2, and dp3, the input of each pipelined parallel unit depends on the output of the previous pipelined parallel unit. For example... Figure 2 The input of pp2 in dp1 depends on pp1 in dp1; after forward and backward computation, each pipeline parallel unit calculates the gradient of the model weights.

[0112] S3022, gradient synchronization between N parallel data units.

[0113] Each data parallel unit receives the gradients of the model weights calculated by all other data parallel units. The gradients of the model weights calculated by each data parallel unit include the gradients calculated by each pipeline parallel unit within that data parallel unit. After receiving the gradients of the model weights calculated by all other data parallel units, the data parallel unit averages the gradients calculated by the pipeline parallel units with the same pipeline parallel number. For example, such as... Figure 2As shown, dp1 receives the gradients of the model weights sent by dp2 and dp3. The gradient of the model weights sent by dp2 includes the gradients calculated by pp1, pp2, pp3, and pp4 in dp2. The gradient of the model weights sent by dp3 includes the gradients calculated by pp1, pp2, pp3, and pp4 in dp3. The gradient calculated by pp1 in dp1, the gradient calculated by pp1 in dp2, and the gradient calculated by pp1 in dp3 are averaged to obtain the synchronized gradient.

[0114] S3023. Update model weights and periodically save checkpoint files.

[0115] The training cluster updates model weights using synchronized gradients. To prevent the loss of model weights stored on nodes due to cluster failure, the training cluster saves parameters such as model weights that change during training as a checkpoint file every fixed step.

[0116] As described above, during cluster training, the cluster training executes steps S3021 to S3023 in a loop until all sample data has been trained. However, when a training card in the cluster malfunctions, the above loop is interrupted, and the operation of step S303 begins.

[0117] S303. If a card in the cluster fails, all cards stop training and fault handling is performed. The fault handling steps are as follows:

[0118] S3031. If a certain card malfunctions and becomes a faulty card, all training cards in the cluster will stop computing.

[0119] When a training card in the training cluster malfunctions (i.e., a faulty card appears), the normal training cards connected to the faulty card in the training cluster stop computing due to communication timeouts caused by their inability to communicate with the faulty card. The normal training cards in the training cluster also stop computing because their communication with the faulty card stops, causing communication timeouts. Therefore, all training cards in the training cluster stop computing. For example, ... Figure 4 As shown, during the training process, the training card on pp1 in dp2 malfunctions. pp1 of dp1, pp1 of dp3, and pp2 of dp2 cannot communicate with pp1 of dp2, resulting in a communication timeout and stopping computation. pp2 of dp3, which is communicating normally with pp1 of dp3 and pp2 of dp2, also experiences a communication timeout because pp1 of dp3 and pp2 of dp2 stop computation. At this point, pp1 of dp3 and pp2 of dp2 also stop computation, and so on, until all training cards in dp1, dp2, and dp3 stop computation.

[0120] S3032, The training cluster decides whether to add a redundant card at the location of the faulty card.

[0121] The control device determines the location of the faulty card by receiving the responses from each training card, and decides whether to add a redundant card (i.e., a spare training card) at the location of the faulty card; if yes, proceed to step S3033b; if no, proceed to step S3033a.

[0122] S3033a, The training cluster deletes the data parallel unit where the faulty card is located. For example, such as... Figure 4 As shown, if pp1 of dp2 fails, all training cards in dp2 will be deleted.

[0123] S3033b: Add redundant cards to the location of the faulty card in the training cluster. For example, such as... Figure 4 As shown, if pp1 of dp2 fails, a redundant card is added to pp1 of dp2 to replace the original faulty card.

[0124] S3034. All training cards read the model parameters stored in the most recent checkpoint file.

[0125] S3035, Restart training in the training cluster.

[0126] Restarting the model training in the training cluster completes the fault handling. At this point, the training cluster will repeat step S301, initialize the training cluster based on the model parameters read from the most recent checkpoint file, and then continue the subsequent model training.

[0127] As described above, in the fault handling scheme of this hybrid parallel training strategy, if the control device decides not to replenish the redundant card at the location of the faulty card, it will delete the data parallel unit where the faulty card is located in the training cluster, resulting in significant resource waste. If the control device decides to replenish the redundant card at the location of the faulty card, but the redundant card cannot be replenished (for example, there is no redundant card in the training cluster), the training task of the training cluster will stop until the redundant card is replenished before the training cluster can continue training (i.e., resume training). Furthermore, because during resume training, the model parameters on the redundant card need to be read from the stored checkpoint file, and the model parameters stored in the checkpoint file are used for subsequent training, this process will waste the training between the step of the most recent checkpoint file and the current step. For example, as shown... Figure 5As shown, when the training cluster fails at step 250, the node that saved the checkpoint file closest to the time of the failure is step200, which has completed 200 steps of training. When the training cluster resumes training, it needs to use the parameters from step200, which results in a waste of 50 steps.

[0128] To address the issue of training clusters relying heavily on redundant equipment and needing to read past checkpoint files during training continuation in the event of a training failure in a large model cluster, resulting in wasted training time, this application provides a model training method. After the cluster training of the model is interrupted, the training cards on the node where the faulty card is located (i.e., the pipeline parallel unit or tensor parallel unit where the faulty card is located) are removed, and all the remaining training cards are used to continue training.

[0129] Figure 6 The system architecture to which this application can be applied is shown.

[0130] The following is based on Figure 6 The system architecture 600 shown in the figure is used as an example for specific illustration. It should be understood that system architecture 600 can have more or fewer components than those shown in the figure. The various components shown in the figure can be implemented in a combination including one or more servers. System architecture 600 includes a control terminal device and a training cluster, which includes multiple server nodes. Figure 6 The example shows server node 1, server node 2, and server node 3; each server node may include an execution end device and multiple neural network processing unit cards (NPU cards), for example, Figure 6 The server node 1, server node 2, and server node 3 shown in the example each include an execution device and eight NPU cards.

[0131] In the system architecture, the control device can run a program on a server and communicate with all NPU cards in the training cluster and the AI ​​server where the NPU cards reside. The control device can receive exception requests from the execution device (i.e., exception reception) and provide necessary information for the training cluster to continue training through computation (i.e., group computation), such as parallel grouping strategies and the training card numbers that can be used to copy model parameters, and then distribute these groups. The training cluster can be a system composed of AI servers (such as Ascend AI servers). Each AI server can include multiple computing cards (such as NPU cards). A computing card is a type of training card, a physical entity responsible for model training. Each AI server has a unique identification (ID), and each computing card has a unique global ID. Computing cards with training tasks communicate with each other during training. An NPU card is a high-speed computing device installed on the AI ​​server. The NPU card is responsible for completing the training tasks distributed by the execution device. The execution device can be deployed on a software entity on the AI ​​server, responsible for distributing tasks to the NPU cards, reporting exception information, and executing specific strategies for resuming training after failure based on the information distributed by the control device. For example, a server node (such as Figure 6 Server node 1, server node 2, or server node 3 can be used as a data parallel unit, i.e., server node 1 is a data parallel unit, denoted as dp1, server node 2 is a data parallel unit, denoted as dp2, and server node 3 is a data parallel unit, denoted as dp3; an NPU (any one of NPU0 to NPU7 in the figure) can be used as a pipelined parallel unit, for example, NPU0 can be used as a pipelined parallel unit, denoted as pp1.

[0132] The control device can be deployed on a server outside the computing cluster, while the execution device can be deployed on the AI ​​server in the computing cluster. Each AI server can contain multiple training cards. It should be noted that the training cards in a data parallel unit can be deployed on one AI server or multiple AI servers. The deployment of training cards in a data parallel unit is not fixedly related to the AI ​​server.

[0133] To reduce resource waste during resume training after a training cluster based on a hybrid parallel grouping strategy experiences a failure, this application provides a model training method. After a failure occurs, the execution device sends an anomaly message to the control device. The control device identifies the faulty card number based on the anomaly message. This faulty card number indicates the training card in the training cluster that has failed. The training cluster is divided into n data parallel units for data parallel training, where n is an integer greater than 1. The control device confirms parallel grouping information based on the faulty card number. This parallel grouping information is used to establish new communication links between training cards in the training cluster and to reallocate training tasks for the training cards other than the faulty card in the data parallel unit where the faulty card resides. Based on the parallel grouping information received from the control device, the execution device controls the training cluster to resume model training.

[0134] The above model training method is illustrated below through four embodiments. It should be noted that the above model training method is not limited to the following four embodiments.

[0135] In one embodiment, each of the n data parallel units is divided into m pipelined parallel units, and each pipelined parallel unit includes s tensor parallel units; where m is an integer greater than 1; and s is an integer greater than or equal to 1.

[0136] The control device confirms the faulty card number based on the anomaly information and then confirms the parallel grouping information based on the faulty card number. The parallel grouping information includes the first reassignment grouping information of the training cards in the training cluster. The first reassignment grouping information includes k×s groups of first training card numbers, where k is an integer greater than or equal to 1 and less than m. Each group of first training card numbers includes n teams of first training card numbers. The first training card indicated by each team of first training card numbers in the same group undertakes the same training task. The first training cards indicated by first training card numbers in different teams are located in different data parallel units. Each group of first training card numbers includes the first team of training card numbers. The target first training card is the first training card indicated by the k×s first team of training card numbers included in the k×s group of training card numbers. The target first training card undertakes the same training task as the target pipeline parallel unit. The target pipeline parallel unit includes pipeline parallel units that are in different data parallel units than the faulty card but undertake the same training task. The parallel grouping information also includes the reference training card number required for repartitioning the training task. The training card (one or more) indicated by the reference training card number is in a different data parallel unit than the faulty card but undertakes the same training task. n is an integer greater than 1.

[0137] The execution device establishes a connection between the target first training card and the target pipeline parallel unit based on the parallel grouping information sent by the control device.

[0138] The execution end device control reference training card number corresponds to the training card that sends the model parameters of the current model training to the first training card indicated by the k×s first team training card number.

[0139] The execution device inputs the training samples from the faulty card into the first training card among the k×s first-team training card numbers indicated by the first training card number, which undertakes the training task. During one model iteration, after completing the gradient calculation, the first training card indicated by the k×s first-team training card number is synchronized with the training cards in the target pipeline parallel unit.

[0140] The above method embodiments include many possible implementation schemes, which are described below in conjunction with... Figure 7A Some of the implementation schemes will be illustrated with examples. It should be noted that... Figure 7A Any related concepts, operations, or logical relationships not explained can be described in the corresponding descriptions in the above embodiments.

[0141] In this application, Figure 7A The illustrated embodiment can be considered as a standalone embodiment. Figure 7A The embodiments shown can all be implemented without relying on the technical solutions of the embodiments described above; Figure 7A Some of the steps in the illustrated embodiments can also be used as separate embodiments.

[0142] In this embodiment, the aforementioned abnormal information may include a timeout abnormal signal (including the type of timeout abnormality, such as a communication abnormality) and the training card number that generated the timeout abnormal signal in the entire training cluster. The aforementioned first reassignment grouping information is a parallel grouping strategy. The aforementioned k×s group of first training card numbers are k×s communication groups. Each of the aforementioned n teams of first training card numbers corresponds to a team of training card numbers in a communication group. The aforementioned target first training card is the pipeline parallel unit that undertakes the training task of the pipeline parallel unit where the original faulty card is located. The aforementioned reference training card number is the training card number required for the reassignment training task.

[0143] exist Figure 7B In the corresponding example, n equals 3, m equals 4, and s equals 1. Figure 7C In the corresponding example, n equals 3, m equals 4, and s equals 2.

[0144] Next, combined Figure 2 , Figure 7B and Figure 7C This section details one embodiment corresponding to the above-described embodiment, and describes the model training process as follows: Figure 7A As shown, the specific process is as follows:

[0145] S701: Start training tasks on all training cards.

[0146] At the start of training, the execution device initiates the training task according to a hybrid parallel training strategy. The training cluster is divided into n parallel data parallel units, performing data parallelism. Within each data parallel unit, the training card is divided into m pipelined parallel units. Pipeline parallel training is performed between pipelined parallel units within the same data parallel unit. For example, as shown... Figure 2 As shown, the training cluster can be divided into three data parallel units, numbered dp1, dp2, and dp3 respectively. Each data parallel unit is further divided into four pipeline parallel units, numbered pp1, pp2, pp3, and pp4 respectively. Each pipeline parallel unit can contain only one training card, as shown below. Figure 7B As shown. Each pipeline parallel unit can also be divided into s tensor parallel units, for example, as shown. Figure 7C As shown, pp1 in dp1 is divided into two tensor parallel units, and the tensor parallel numbers of these two tensor parallel units are tp1 and tp2, respectively. Tensor parallelism is performed between tensor parallel units in the same pipeline parallel unit. Typically, each tensor parallel unit contains only one training card.

[0147] Each data parallel unit includes a complete set of model weights and training-related parameters. The difference between different data parallel units lies in the different input sample data. During training, each data parallel unit calculates the gradient using the exact same gradient calculation method. Different data parallel units share the same pipeline parallel unit number and tensor parallel unit number on the training card. Gradient averaging is performed after gradient calculation is complete.

[0148] S702: A training card has malfunctioned.

[0149] During training, if a training card in the training cluster malfunctions (i.e., a faulty card appears), preventing the training task from continuing or significantly slowing down the task execution speed, all training cards in the training cluster will pause their training tasks. The execution device will capture the timeout exception signal of the training cards in the training cluster and report the S7.100 timeout exception signal (including the type of timeout exception, such as communication exception) and the number of the training card that generated the timeout exception signal in the entire training cluster to the control device. For example, Figure 7B If pp1 in dp2 fails, then all pps except the failed pp1 can send a timeout error signal (carrying their own training card number) to the control terminal device, that is, send the error information of this application embodiment.

[0150] S703: Receives timeout error signal, confirms fault card number, and calculates parallel grouping information.

[0151] The control device receives anomaly information (including a timeout anomaly signal and the training card number that generated the timeout anomaly signal within the entire training cluster) reported by the execution device. It confirms the faulty card's number within the training cluster. For example, analysis of the received anomaly information reveals that only pp1 in dp2 did not emit a timeout anomaly signal; therefore, the faulty card is identified as pp1 in dp2, and its number is determined to be the number corresponding to pp1 in dp2. Based on this number, it calculates parallel grouping information, including the S7.200 parallel grouping strategy and the reference training card number required for repartitioning the training task. Before confirming the parallel grouping strategy, it is necessary to identify the pipeline parallel unit that will undertake the training task of the original faulty card's pipeline parallel unit. Specifically, among the data parallel units where the faulty card resides, at least one pipeline parallel unit (e.g., the one where the faulty card resides) must be selected. Figure 2 (at least one of pp2, pp3, and pp4 in dp2), the parallel grouping strategy includes communication groups for training cards that need to establish new communication links when the training cluster continues training. Each communication group includes multiple training card numbers, which means that a new communication link needs to be established between these multiple training cards.

[0152] For example, in Figure 7B In this context, the parallel grouping strategy calculated by the control terminal device based on the fault card number includes two communication groups (taking k=2 and s=1 as an example, then k×s=2). One communication group contains dp1pp1, dp2pp2, and dp3pp1, and the other communication group contains dp1pp1, dp2pp3, and dp3pp1. Therefore, the first redistribution group information can include two sets of communication first training card numbers. Each communication group can be divided into three teams of training card numbers (i.e., n=3). For example, in the communication group dp1pp1, dp2pp2, dp3pp1, dp1pp1 is one team of training card numbers, dp2pp2 is another team of training card numbers, and dp3pp1 is yet another team of training card numbers. Similarly, in the communication group dp1pp1, dp2pp3, dp3pp1, dp1pp1 is one team of training card numbers, dp2pp3 is another team of training card numbers, and dp3pp1 is yet another team of training card numbers.

[0153] This parallel grouping strategy is also used to instruct the execution-end device to control the training cluster to establish connections between the target first training card and the target pipeline parallel unit, including creating new connections such as... Figure 7BThe communication connections shown refer to establishing connections between the three pipelined parallel units pp1 of dp1, pp2 of dp2, and pp1 of dp3, and between the three pipelined parallel units pp1 of dp1, pp3 of dp2, and pp1 of dp3. In other words, the target first training card is the first training card indicated by the two first team training card numbers included in the two sets of training card numbers; in this embodiment, the two first team training card numbers are dp2pp2 and dp2pp3, which perform the same training task as the target pipelined parallel unit (i.e., dp1pp1 or dp3pp1).

[0154] For example, in Figure 7C Taking s = 2 as an example, each pipeline parallel unit includes 2 tensor parallel units (i.e., two tp). The parallel grouping strategy calculated by the control device based on the fault card number includes four communication groups (taking k = 2 and s = 2 as an example, then k × s = 4). One communication group contains dp1pp1tp1, dp2pp2tp1, and dp3pp1tp1; another group contains dp1pp1tp1, dp2pp3tp1, and dp3pp1tp1; another group contains dp1pp1tp2, dp2pp2tp2, and dp3pp1tp2; and yet another group contains dp1pp1tp2, dp2pp3tp2, and dp3pp1tp2. Therefore, the first reassignment group information can include four groups of first training card numbers for communication. Each communication group can be divided into three training card numbers (i.e., n equals 3). For example, in communication groups dp1pp1tp1, dp2pp2tp1, and dp3pp1tp1, dp1pp1tp1 is one training card number, dp2pp2tp1 is another training card number, and dp3pp1tp1 is yet another training card number; the three training card numbers for the remaining groups follow the same pattern.

[0155] This parallel grouping strategy is also used to instruct the execution end device to control the training cluster to establish connections between the target first training card and the target pipelined parallel unit (specifically, the tensor parallel unit within it), including creating new connections such as... Figure 7CThe communication connections shown are established between the TPs in the three pipelined parallel units dp1pp1tp1, dp2pp2tp1, and dp3pp1tp1; between the TPs in the three pipelined parallel units dp1pp1tp1, dp2pp3tp1, and dp3pp1tp1; between the TPs in the three pipelined parallel units dp1pp1tp2, dp2pp2tp2, and dp3pp1tp2; and between the TPs in the three pipelined parallel units dp1pp1tp2, dp2pp3tp2, and dp3pp1tp2. In other words, the target first training card is the first training card indicated by the four first team training card numbers included in the four sets of training card numbers; in this embodiment of the application, the four first team training card numbers are dp2pp2tp1, dp2pp3tp1, dp2pp2tp2 and dp2pp3tp2, which respectively undertake the same training task as the target pipeline parallel unit (i.e. dp1pp1tp1 or dp3pp1tp1, dp1pp1tp2 or dp3pp1tp2).

[0156] The reference training card number required for repartitioning the training task is the number of the training card that is in a different data parallel unit from the faulty card and is performing the same training task. For example, such as... Figure 7B As shown, if pp1 in dp2 fails, since the model weights and other parameters required for model training are exactly the same on the training cards of the same pipeline and unit number in different data parallel units (for example, the model weights and other parameters required for model training are exactly the same on the training card of pp1 in dp1 and the training card of pp1 in dp2), when pp1 in dp2 fails, the parameters required to execute the training task of pp1 in dp2 can be obtained from pp1 in dp1 or pp1 in dp3. After the control device decides which training card to select as the reference training card, it sends the pipeline parallel unit number (i.e., the reference training card number) of pp1 in dp1 or pp1 in dp3 to the execution device. Correspondingly, Figure 7C In one embodiment, after the control device decides which training card to select as the reference training card, it sends the number of tp1 in dp1 pp1 or tp1 in dp3 pp1 (i.e., the reference training card number) to the execution device, and sends the number of tp2 in dp1 pp1 or tp2 in dp3 pp1 (i.e., the reference training card number) to the execution device.

[0157] S704: Establish a new communication link.

[0158] The execution device establishes a new communication connection according to the parallel grouping strategy issued by the control device.

[0159] For example, as described in step S703 above, to Figure 7BFor example, the execution end device controls the creation of a training cluster. Figure 7B The communication connection shown refers to establishing a connection between the three pipeline parallel units pp1 of dp1, pp2 of dp2, and pp1 of dp3, and establishing a connection between the three pipeline parallel units pp1 of dp1, pp3 of dp2, and pp1 of dp3.

[0160] by Figure 7C For example, the execution end device controls the creation of a training cluster. Figure 7C The communication connections shown are established between the TPs in the three pipelined parallel units dp1pp1tp1, dp2pp2tp1, and dp3pp1tp1; between the TPs in the three pipelined parallel units dp1pp1tp1, dp2pp3tp1, and dp3pp1tp1; between the TPs in the three pipelined parallel units dp1pp1tp2, dp2pp2tp2, and dp3pp1tp2; and between the TPs in the three pipelined parallel units dp1pp1tp2, dp2pp3tp2, and dp3pp1tp2.

[0161] It should be noted that the communication connection can be established after receiving the parallel grouping strategy and then disconnected after the entire training cluster has finished training, that is, the connection is maintained throughout the entire training process. Alternatively, a short-term connection can be established when gradient transmission is required for model parameter transmission or gradient synchronization, and then disconnected after the corresponding data is transmitted. This application does not impose any restrictions on this.

[0162] S705: Redivide the training task.

[0163] The execution device sends model parameters to the target training card (which may include the first training card indicated by k×s first team training card numbers) according to the reference training card number required for the repartition training task issued by the control device. The target training card is the training card in the pipeline parallel unit that undertakes the training task of the pipeline parallel unit where the original faulty card is located.

[0164] by Figure 7B For example, assuming the reference training card number is dp1pp1, and the first training cards indicated by the two first-pair training card numbers are dp2pp2 and dp2pp3 respectively, then the execution device controls dp1pp1 to send its current model training parameters to dp2pp2 and dp2pp3. That is, dp2pp2 and dp2pp3 can continue training based on these current model training parameters during training tasks, without needing to return to a checkpoint.

[0165] by Figure 7CFor example, assuming the reference training card numbers include dp1pp1tp1 and dp1pp1tp2, and the four first-team training card numbers are dp2pp2tp1, dp2pp3tp1, dp2pp2tp2, and dp2pp3tp2 respectively, then the execution device controls dp1pp1tp1 to send its currently trained model parameters to dp2pp2tp1 and dp2pp3tp1; and controls dp1pp1tp1 to send its currently trained model parameters to dp2pp2tp2 and dp2pp3tp2. That is, dp2pp2tp1, dp2pp3tp1, dp2pp2tp2, and dp2pp3tp2 can continue training based on the currently trained model parameters during training tasks, without needing to return to a checkpoint.

[0166] When repartitioning the training task, if the training card in the selected pipelined parallel unit (i.e., at least one of the other pipelined parallel units sharing the same data parallel unit as the faulty card) has insufficient storage space to store the partitioned training task, the number of pipelined parallel units can be increased. This process is completed in step S703 when the control device calculates the parallel grouping strategy. The parameters allocated to the selected pipelined parallel units during repartitioning of the training task are transmitted using the newly established communication connection in S704. The parameters allocated to the selected pipelined parallel units during repartitioning of the training task can be partitioned on a layer-by-layer basis.

[0167] S706: The training cluster continues training.

[0168] After completing the task of establishing a new communication connection and re-dividing the training cluster, the execution device can control the training cluster to continue training.

[0169] When the training cluster continues training, each model training iteration mainly includes the following two steps:

[0170] S7061: Gradient calculation.

[0171] The execution device inputs the sample text used for model training into the training card in pp1 of each data parallel unit. For data parallel units with faulty cards, if pp1 fails as in the example in step S705, and pp2 and pp3 in dp2 are selected to take over the training task originally performed on the faulty card, then the sample data is input into the pipeline parallel unit in pp2 and pp3 in dp2 that takes over the training task originally performed on pp1 in dp2 (that is, the first training card in the first training card indicated by the k×s first training card number to take over the training task).

[0172] by Figure 7BFor example, suppose pp2 and pp3 in dp2 are chosen to undertake the training task originally performed on the fault card (dp2pp1). Taking the original training task of dp2pp1 as two layers of the model, and assuming the training of these two layers has a specific order, after regrouping, dp2pp2 trains one of these two layers first, and dp2pp3 trains the other layer later. That is, the output of dp2pp2 is used as the input of dp2pp3. Then, the execution device can control the input of the training samples from the fault card to dp2pp2 (i.e., the first training card to undertake the training task), and dp2pp2 trains the training task of dp1pp1. Taking the original training task of dp2pp1, which involved training 5 layers of the model, as an example, and noting that the training of these 5 layers had a specific order, after regrouping, dp2pp2 first trains the first 2 layers, and dp2pp3 then trains the last 3 layers. That is, the output of dp2pp2 after training the first 2 layers serves as the input to dp2pp3, which then trains the last 3 layers. The execution device can then control the input of the training samples from the faulty card to dp2pp2 (the first training card to undertake the training task), thereby training dp1pp1's training task through dp2pp2.

[0173] Again Figure 7C For example, suppose pp2tp1 and pp3tp1 in dp2 are chosen to undertake the training tasks originally performed on fault card dp2pp1tp1, and pp2tp2 and pp3tp2 are chosen to undertake the training tasks originally performed on fault card dp2pp1tp2. Taking the training task originally performed on fault card dp2pp1tp1 as 5 layers of the model, and the training of these 5 layers has a specific order, after regrouping, dp2pp2tp1 trains the first 2 layers of these 5 layers first (corresponding to the training task of training pp1tp1(a) in the figure), and dp2pp3tp1 trains the last 3 layers of these 5 layers afterward (corresponding to the training task of training pp1tp1(b) in the figure). That is, the output of dp2pp2tp1 after training the first 2 layers is used as the input of dp2pp3tp1, and dp2pp3tp1 then trains the last 3 layers. The execution device can then control the input of training samples from the faulty card to dp2pp2tp1 (i.e., the first training card to undertake the training task), and train dp1pp1tp1 through dp2pp2tp1. The principle is similar for pp2tp2 and pp3tp2 to undertake the training tasks originally performed on the faulty card dp2pp1tp2, and will not be elaborated here.

[0174] Other pipelined parallel units (such as the pipelined parallel unit where the faulty card resides and the selected pipelined parallel unit) within the data parallel unit where the faulty card resides. Figure 7BThe gradient calculation operation in dp2 (pp4) remains unchanged. The selected pipeline parallel unit, in addition to completing the gradient calculation that should have been completed before the faulty card appeared, also needs to undertake the gradient calculation for the training task originally assigned to the faulty card. Other data parallel units (such as...) Figure 7B The gradient calculation operations for dp1 and dp3 remain unchanged.

[0175] by Figure 7B For example, suppose pp2 and pp3 in dp2 are chosen to undertake the training tasks originally performed on the fault card (dp2pp1). After completing the training tasks on the fault card, pp2 and pp3 in dp2 continue training the original training tasks. For instance, following the original training order, the training output of pp1 becomes the training input of pp2, and the training output of pp2 becomes the training input of pp3. Then, following the example above, the output of dp2pp3 after completing the training tasks on the original fault card (i.e., training the training tasks of dp1pp1) becomes the input of dp2pp2. At this point, dp2pp2 is used for training (training its own original training tasks, i.e., training the training tasks of dp2pp2). The training output of dp2pp2 becomes the training input of dp2pp3, and then dp2pp3 is used for training (training its own original training tasks, i.e., training the training tasks of dp2pp3).

[0176] by Figure 7C For example, suppose pp2tp1 and pp3tp1 in dp2 are chosen to perform the training tasks originally performed on fault card dp2pp1tp1, and pp2tp2 and pp3tp2 are chosen to perform the training tasks originally performed on fault card dp2pp1tp2. Then, after pp2tp1 and pp3tp1 in dp2 have completed the training tasks originally performed on fault card pp1tp1, they continue training the original training tasks. For example, following the original training order, the training output of dp2pp1tp1 becomes the training input of dp2pp2tp1, and the training output of dp2pp2tp1 becomes the training input of dp2pp3tp1. Following the example above, the output of dp2pp1tp1 after training the original fault card's training task (i.e., training the training task of pp1tp1(a)) is used as the input of dp2pp3tp1. The output of dp2pp3tp1 after training the original fault card's training task (i.e., training the training task of pp1tp1(b)) is used as the input of dp2pp2tp1. Then, dp2pp2tp1 is used for training (training the task it was originally supposed to train, i.e., training the training task of pp2tp1). The training output of dp2pp2tp1 is used as the training input of dp2pp3tp1. Then, dp2pp3tp1 is used for training (training the task it was originally supposed to train, i.e., training the training task of pp3tp1).

[0177] Understandably, when the data parallel unit containing the faulty card resumes training, it can appropriately reduce the number of sample data to reduce the workload of the selected pipeline parallel unit, so as not to hinder the gradient synchronization time of other data parallel units.

[0178] S7062: Gradient synchronization.

[0179] Through the newly established communication link, the pipeline parallel unit (i.e., the target pipeline parallel unit) that is not in the same data parallel unit as the faulty card and has the same pipeline parallel number as the faulty card performs gradient synchronization (gradient averaging) with the selected pipeline parallel unit; the gradient synchronization operation of the training cards in the remaining pipeline parallel units remains unchanged. For example, the communication link can be established only when gradient synchronization is required, or it can be established in advance in step S704, and this application does not make specific limitations.

[0180] Because the parallel units in the pipeline have a sequential relationship—that is, during one model iteration, pp1 completes its training task and produces output, pp2 trains based on the output of pp1, pp3 trains based on the output of pp2, and so on—after the training tasks are redistributed, as follows: Figure 7B As shown, pp2 and pp3 in dp2 take over the training task of pp1 in dp2. At the beginning of training, in dp2, pp2 and pp3 complete the training task of pp1 and generate output. The output is input to pp2 to complete the training task of pp2 and generate output. Then the output of pp2 is input to pp3 for subsequent training. After completing the forward and backward calculations, the gradients of each pipeline parallel unit are obtained. The gradient data is transmitted through the newly established communication connection and the communication connection before the fault, and gradient synchronization is performed.

[0181] Based on the previous embodiment, when a training card in a pipelined parallel unit fails, all training tasks in the data parallel unit where the failed card is located can be redistributed (e.g., evenly distributed) to all pipelined parallel units in that data parallel unit except for the pipelined parallel unit where the failed card is located. The embodiments of this application are described below.

[0182] In this embodiment, each of the n data parallel units is divided into m pipelined parallel units, and each pipelined parallel unit includes s tensor parallel units; where m is an integer greater than 1, and s is an integer greater than or equal to 1.

[0183] The control device confirms the faulty card number based on the anomaly information and confirms the parallel grouping information based on the faulty card number. The parallel grouping information includes the second reassignment grouping information of the training cards in the training cluster, which includes the second training card numbers of group i×s. Each group of second training card numbers includes n teams of second training card numbers. The second training cards indicated by the second training card numbers of different teams are located in different data parallel units. The second training card indicated by the second training card number of each team corresponds to a pipeline parallel unit in its data parallel unit. The second training cards indicated by the second training card numbers of each team in the same group of training card numbers undertake the same training task. i is the number of groups after regrouping the number of layers of the training model based on the number of layers of the training model and the training cards in the pipeline parallel unit without faults in the data parallel unit where the faulty card is located; i is greater than or equal to m. The parallel grouping information also includes the reference training card number required for repartitioning the training task. The training card (one or more cards) indicated by the reference training card number includes training cards that are in different data parallel units from the faulty card and undertake the same training task.

[0184] The execution device establishes connections between the second training cards indicated by the team numbers in each group of second training card numbers based on the parallel grouping information sent by the control device. Each group of second training card numbers includes the second team training card numbers. The second training cards indicated by the i×s second team training card numbers in the i×s groups are in the same data parallel unit as the faulty card, but not in the pipeline parallel unit where the faulty card is located. The m groups of second training card numbers include the target group of second training card numbers. The second training cards indicated by the second team training card numbers in the target group of second training card numbers perform the same training task as the target pipeline parallel unit. The target pipeline parallel unit includes pipeline parallel units that are in different data parallel units than the faulty card but perform the same training task.

[0185] The execution end device controls the training card corresponding to the reference training card number to send the model parameters of the current model training to the second training card corresponding to the second training card number in the same group.

[0186] The execution device inputs the training samples from the faulty card into the first training card in the second training card group indicated by the second training card number (including the second training card number in the target group's second training card number) that undertakes the training task. During one model iteration, the second training card indicated by the second training card number (including the second training card number in the target group's second training card number) is gradient synchronized with the training cards in the target pipeline parallel unit.

[0187] The above method embodiments include many possible implementation schemes, which are described below in conjunction with... Figure 8A Some of the implementation schemes will be illustrated with examples. It should be noted that... Figure 8AAny related concepts, operations, or logical relationships not explained can be described in the corresponding descriptions in the above embodiments.

[0188] In this application, Figure 8A The illustrated embodiment can be considered as a standalone embodiment. Figure 8A The embodiments shown can all be implemented without relying on the technical solutions of the embodiments described above; Figure 8A Some of the steps in the illustrated embodiments can also be used as separate embodiments.

[0189] In this embodiment, the above-mentioned abnormal information is a timeout abnormal signal (including the type of timeout abnormality, such as communication abnormality) and the number of the training card that generated the timeout abnormal signal in the entire training cluster. The above-mentioned second reassignment grouping information is a parallel grouping strategy. The above-mentioned i×s group first training card number corresponds to i×s communication groups. Each of the above-mentioned n teams of second training card numbers corresponds to a team of training card numbers in a communication group. The above-mentioned reference training card number is the training card number required for reassigning the training task.

[0190] exist Figure 8B In the corresponding example, n equals 3, m equals 4, and s equals 1.

[0191] Next, combined Figure 2 , Figure 8B and Figure 8C This section provides a detailed description of one embodiment corresponding to the above-described embodiment. Figure 2 The training cluster shown is divided into three data parallel units, numbered dp1, dp2, and dp3. Each data parallel unit is further divided into four pipelined parallel units, numbered pp1, pp2, pp3, and pp4. Each pipelined unit can also be divided into multiple tensor parallel units. If it is divided into two tensor parallel units, then these two tensor parallel units are numbered tp1 and tp2 (e.g., ...). Figure 8C For ease of description, in the embodiments of this application, the unit number (such as the data parallel unit) is used to refer to the corresponding unit. For example, dp1pp1tp1 refers to the tensor parallel unit numbered tp1 in the pipeline parallel unit numbered pp1 within the data parallel unit numbered dp1, and dp1pp1 refers to the pipeline parallel unit numbered pp1 within the data parallel unit. Figure 8A As shown, the specific process is as follows:

[0192] S801: Start training tasks on all training cards.

[0193] This step can be referred to as step S701, and will not be repeated here.

[0194] S802: A training card has malfunctioned.

[0195] This step can be referred to as step S702, and will not be repeated here.

[0196] S803: Receives timeout error signal, confirms fault card number, and calculates parallel grouping information.

[0197] This step is similar to step S703. The control device receives the abnormal information reported by the execution device (including a timeout abnormal signal and the training card number that generated the timeout abnormal signal in the entire training cluster), confirms the faulty card number, and calculates the parallel grouping information based on the number. The parallel grouping information includes the 8.200 parallel grouping strategy and the training card number required to repartition the training task. The confirmation of the training card number required to repartition the training task is confirmed as in step S703 and will not be elaborated here.

[0198] The parallel grouping strategy in this training method is described in detail below.

[0199] Different from Figure 7A In the corresponding training method, in this embodiment of the application, before the control device calculates the parallel grouping strategy, it needs to confirm the repartitioning result of the training tasks and redistribute (e.g., evenly distribute) all training tasks in the data parallel unit where the faulty card is located to the pipeline parallel units other than the pipeline parallel unit where the faulty card is located. The order of the training tasks is the same as the order of the training tasks in the training cluster before the fault. For example, as shown... Figure 8B As shown, if pp1 in dp2 fails, all training tasks in dp2 are re-divided and evenly distributed to the pipelined parallel units (i.e., dp2pp2, dp2pp3, and dp2pp4) in dp2, excluding pp1. The order of the training tasks is the same as the order of the training tasks in the training cluster before the failure. After the control device confirms the re-division result of the training tasks, it calculates the parallel grouping information. The parallel grouping information is the communication group of the training cards that need to establish new communication links when the training cluster continues training. Each communication group includes multiple training card numbers, indicating that a new communication link needs to be established between these multiple training cards. That is, the training card numbers in the pipelined parallel units undertaking the same training task are placed in one communication group.

[0200] For example, in Figure 8BIn the training cluster shown, the control device can regroup the training model's layers based on the number of layers in the training model and the number of training cards (i.e., dp2pp2, dp2pp3, dp2pp4) in the pipeline parallel unit where the faulty card resides within the data parallel unit. For example, assuming the training model has 10 layers, it was originally divided into 4 groups (corresponding to the original pp1, pp2, pp3, pp4) for training. pp1 trains the first 2 layers, pp2 trains the next 3 layers, pp3 trains the next 3 layers, and pp4 trains the last 2 layers. Based on the training cards (dp2pp2, dp2pp3, dp2pp4) in the pipeline parallel unit where the faulty card is located, the training model is re-divided into 6 groups according to the number of layers (taking i = 6, s = 1, then i × s = 6 as an example). The first group is the first 2 layers of the model (corresponding to the training task of training pp1 in the figure), the second group is the next 1 layer (i.e. the 3rd layer of the model, corresponding to the training task of training pp2(a) in the figure), the third group is the 4th and 5th layers of the model (corresponding to the training task of training pp2(b) in the figure), the fourth group is the 6th and 7th layers of the model (corresponding to the training task of training pp3(a) in the figure), the fifth group is the 8th layer of the model (corresponding to the training task of training pp3(b) in the figure), and the sixth group is the last 2 layers of the model (corresponding to the training task of training pp4 in the figure). The dp2pp2 training model is configured with the first 3 layers (corresponding to the 1st and 2nd groups above), the dp2pp3 training model with the middle 4 layers (corresponding to the 3rd and 4th groups above), and the dp2pp4 training model with the last 3 layers (corresponding to the 5th and 6th groups above). The corresponding calculated or generated second reassignment grouping information includes the second training card numbers of the i×s groups (6 groups), specifically: the second training card numbers of the first group include dp1pp1, dp2pp2, and dp3pp1; the second training card numbers of the second group include dp1pp2, dp2pp2, and dp3pp2; the second training card numbers of the third group include dp1pp2, dp2pp3, and dp3pp2; the second training card numbers of the fourth group include dp1pp3, dp2pp3, and dp3pp3; the second training card numbers of the fifth group include dp1pp3, dp2pp4, and dp3pp3; and the second training card numbers of the sixth group include dp1pp4, dp2pp4, and dp3pp4.

[0201] Each communication group can be divided into three training card numbers (i.e., n equals 3). For example, in communication groups dp1pp1, dp2pp2, and dp3pp1, dp1pp1 is one training card number, dp2pp2 is another, and dp3pp1 is the third. Similarly, in communication groups dp1pp2, dp2pp2, and dp3pp2, dp1pp2 is one training card number, dp2pp2 is another, and dp3pp2 is the third. The training card numbers for the remaining groups follow the same pattern.

[0202] This parallel grouping strategy is also used to instruct the execution end device to control the training cluster to establish connections between the second training cards of each team indicated by the second training card number in each group; including creating new connections such as... Figure 8B The communication connections shown refer to establishing connections between the three pipelined parallel units pp1 of dp1, pp2 of dp2, and pp1 of dp3; between the three pipelined parallel units pp2 of dp1, pp3 of dp2, and pp2 of dp3; and between the three pipelined parallel units pp3 of dp1, pp4 of dp2, and pp3 of dp3. In other words, in this embodiment, the second training card indicated by the second training card number of the target group is dp2pp2 (which completes the training task of the original faulty card), and it undertakes the same training task as the target pipelined parallel unit (i.e., dp1pp1 or dp3pp1).

[0203] For example, in Figure 8C Taking s=4 as an example, each pipeline parallel unit includes 4 tensor parallel units (i.e., four tp). The parallel grouping strategy calculated by the control device based on the fault card number includes 24 communication packets (taking i=6 and s=4 as an example, then i×s=24). Figure 8C The 6 groups corresponding to each tp can be referenced. Figure 8B Six communication packets as an example:

[0204] That is, targeting Figure 8CThe TP1 corresponding to DP2 can be divided into 6 groups: Group 1, the second training card numbers include DP1PP1TP1, DP2PP2TP1, DP3PP1TP1; Group 2, the second training card numbers include DP1PP2TP1, DP2PP2TP1, DP3PP2TP1; Group 3, the second training card numbers include DP1PP2TP1, DP2PP3TP1, DP3PP2TP1; Group 4, the second training card numbers include DP1PP3TP1, DP2PP3TP1, DP3PP3TP1; Group 5, the second training card numbers include DP1PP3TP1, DP2PP4TP1, DP3PP3TP1; and Group 6, the second training card numbers include DP1PP4TP1, DP2PP4TP1, DP3PP4TP1.

[0205] against Figure 8C The tp2 corresponding to dp2 can be divided into 6 groups: Group 1, the second training card numbers include dp1pp1tp2, dp2pp2tp2, dp3pp1tp2; Group 2, the second training card numbers include dp1pp2tp2, dp2pp2tp2, dp3pp2tp2; Group 3, the second training card numbers include dp1pp2tp2, dp2pp3tp2, dp3pp2tp2; Group 4, the second training card numbers include dp1pp3tp2, dp2pp3tp2, dp3pp3tp2; Group 5, the second training card numbers include dp1pp3tp2, dp2pp4tp2, dp3pp3tp2; and Group 6, the second training card numbers include dp1pp4tp2, dp2pp4tp2, dp3pp4tp2.

[0206] against Figure 8C The groupings corresponding to tp3 and tp4 in dp2 can be deduced in the same way, and will not be elaborated here.

[0207] Each communication group can be divided into three training card numbers (i.e., n equals 3). For example, in communication groups dp1pp1tp1, dp2pp2tp1, and dp3pp1tp1, dp1pp1tp1 is one training card number, dp2pp2tp1 is another, and dp3pp1tp1 is yet another. Similarly, in communication groups dp1pp2tp1, dp2pp2tp1, and dp3pp2tp1, dp1pp2tp1 is one training card number, dp2pp2tp1 is another, and dp3pp2tp1 is yet another. The training card numbers for the remaining groups follow the same pattern.

[0208] This parallel grouping strategy is also used to instruct the execution end device to control the training cluster to establish connections between the second training cards of each team indicated by the second training card number in each group; including creating new connections such as... Figure 8C The communication connections shown are established between dp1pp1tp1, dp2pp2tp1, and dp3pp1tp1; between dp1pp2tp1, dp2pp3tp1, and dp3pp2tp1; and between dp1pp3tp1, dp2pp4tp1, and dp3pp3tp1. Connections to other groups are similar and will not be described further here.

[0209] In other words, in the embodiments of this application, the second training card number of the target group includes the second training card number indicating the second training card as dp2pp2tp1, dp2pp2tp2, dp2pp2tp3 and dp2pp2tp4 (the training task of the original fault card is completed through this training card), which undertake the same training task as the target pipeline parallel unit (i.e. dp1pp1tp1, dp1pp1tp2, dp1pp1tp3 and dp1pp1tp4; or dp3pp1tp1, dp3pp1tp2, dp3pp1tp3 and dp3pp1tp4).

[0210] The reference training card number required for repartitioning the training task is the number of the training card that is in a different data parallel unit from the faulty card and is performing the same training task. For example, such as... Figure 8B As shown, if pp1 in dp2 fails, since the model weights and other parameters required for model training are exactly the same on the training cards of the same pipeline and unit number in different data parallel units (for example, the model weights and other parameters required for model training are exactly the same on the training card of pp1 in dp1 and the training card of pp1 in dp2), when pp1 in dp2 fails, the parameters required to execute the training task of pp1 in dp2 can be obtained from pp1 in dp1 or pp1 in dp3. After the control device decides which training card to select as the reference training card, it sends the pipeline parallel unit number (i.e., the reference training card number) of pp1 in dp1 or pp1 in dp3 to the execution device. Correspondingly, Figure 8C In one embodiment, after the control device decides which training card to select as the reference training card, it sends the number of tp1 in dp1 pp1 or tp1 in dp3 pp1 (i.e., the reference training card number) to the execution device, and sends the number of tp2 in dp1 pp1 or tp2 in dp3 pp1 (i.e., the reference training card number) to the execution device.

[0211] S804: After confirming the results of the repartitioning task, establish a new communication link and repartition the training task.

[0212] During the process of establishing a new communication link, the execution device confirms the result of re-dividing the training task based on the parallel grouping strategy issued by the control device, and then establishes a new communication connection based on the parallel grouping strategy.

[0213] For example, as described in step S803 above, to Figure 8B For example, the execution end device controls the creation of a training cluster. Figure 8B The communication connections shown refer to establishing connections between the three pipelined parallel units pp1 of dp1, pp2 of dp2, and pp1 of dp3; between the three pipelined parallel units pp2 of dp1, pp3 of dp2, and pp2 of dp3; and between the three pipelined parallel units pp3 of dp1, pp4 of dp2, and pp3 of dp3. In other words, in this embodiment, the second training card indicated by the second training card number of the target group is dp2pp2 (which completes the training task of the original faulty card), and it undertakes the same training task as the target pipelined parallel unit (i.e., dp1pp1 or dp3pp1).

[0214] by Figure 8C For example, the execution end device controls the creation of a training cluster. Figure 8C The communication connections shown include establishing connections between dp1pp1tp1, dp2pp2tp1, and dp3pp1tp1; between dp1pp2tp1, dp2pp3tp1, and dp3pp2tp1; and between dp1pp3tp1, dp2pp4tp1, and dp3pp3tp1. Establishing connections for other groups is similar and will not be described further here.

[0215] During the process of re-dividing the training tasks, the execution device, according to the reference training card number required for the re-divided training tasks issued by the control device, controls the training card corresponding to the reference training card number to send model parameters to the second training card corresponding to the second training card number in the same group. The second training card corresponding to the second training card number in the same group is the training card in the pipeline parallel unit that undertakes the training task of the pipeline parallel unit where the original faulty card was located. The execution device can also control other training cards in the re-divided training tasks to obtain the model parameters required for training; for example, it can control pp3 in dp2 to obtain the model parameters required for training within its own dp2.

[0216] by Figure 8B For example, assuming the reference training card number is dp1pp1, then the second training cards in the same group are dp2pp2 and dp3pp1, with the second training card number being dp2pp2. The execution device controls dp1pp1 to send its current model training parameters to dp2pp2. That is, when dp2pp2 trains the training task of dp1pp1 (i.e., the original training task of dp2pp1), it can continue training based on the currently trained model parameters without needing to return to a checkpoint.

[0217] by Figure 8C For example, let's take one of the training cards (tp) as an example: Assume the reference training card numbers include dp1pp1tp1, dp1pp1tp2, dp1pp1tp3, and dp1pp1tp4, and the training card numbers for the four second teams in the same group are dp2pp2tp1, dp2pp2tp2, dp2pp2tp3, and dp2pp2tp4, respectively. Then, the execution device controls dp1pp1tp1, dp1pp1tp1, dp1pp1tp2, dp1pp1tp3, and dp1pp1tp4 to send the model parameters of their respective current model training to dp2pp2tp1, dp2pp2tp2, dp2pp2tp3, and dp2pp2tp4, respectively. In other words, when training dp2pp2tp1, dp2pp2tp2, dp2pp2tp3, and dp2pp2tp4, they can continue training based on the model parameters trained on the current model without having to return to the checkpoint.

[0218] S805: The training cluster continues training.

[0219] After completing the task of establishing a new communication connection and re-dividing the training cluster, the execution device can control the training cluster to continue training.

[0220] When the training cluster continues training, each model training iteration mainly includes the following two steps:

[0221] S8051: Gradient calculation.

[0222] The execution device inputs the sample data used for model training into each data parallel unit, specifically into the first pipelined parallel unit (the pipelined parallel unit that undertakes the foremost training task) within each data parallel unit. If the first pipelined parallel unit fails, the sample data is input into the pipelined parallel unit that originally handled the training task for the failed card; that is, the training samples from the failed card are input into the first training card in the second training card group indicated by the second training card number of the second training card group in the target group. In this embodiment, each training card corresponding to the second training card number of the target group is used to undertake the training task originally handled by the failed card, i.e., as shown... Figure 8B As shown in the figure, assuming that pp1 in dp1 is faulty, the second training card number of the target group includes dp1pp1, dp2pp2, and dp3pp1. The training card number of the second group is dp2pp2, which means that the training samples originally input to the faulty card number are input to dp2pp2 as the first training card to undertake the training task.

[0223] by Figure 8B For example, the second reassignment grouping information includes the second training card numbers of the i×s group (6 groups), specifically: the second training card numbers of the first group include dp1pp1, dp2pp2, and dp3pp1; the second training card numbers of the second group include dp1pp2, dp2pp2, and dp3pp2; the second training card numbers of the third group include dp1pp2, dp2pp3, and dp3pp2; the second training card numbers of the fourth group include dp1pp3, dp2pp3, and dp3pp3; the second training card numbers of the fifth group include dp1pp3, dp2pp4, and dp3pp3; and the second training card numbers of the sixth group include dp1pp4, dp2pp4, and dp3pp4. The execution device can then control the input of the training samples from the faulty card to dp2pp2 (i.e., the first training card to undertake the training task), and dp2pp2 trains the training task of dp1pp1. Then, it outputs the samples back to its own input, dp2pp2, and dp2pp2 trains its own training task (i.e., the training task of pp2tp1(a) in the diagram). Then, it outputs the samples to dp2pp3, which first trains the remaining training task of dp2pp2 (i.e., the training task of pp2tp1(b) in the diagram), and then outputs the samples back to its own input, dp2pp3, and then trains its own training task (i.e., the training task of pp3tp1(a) in the diagram). Then, it outputs the samples to dp2pp4, which first trains the remaining training task of dp2pp3 (i.e., the training task of pp3tp1(b) in the diagram), and finally outputs the samples back to dp2pp4, which then trains its own training task.

[0224] Again Figure 8C For example, the training process of one of the TPs will be described (as described above). Figure 8B(Similar to): The second reassignment grouping information includes the second training card number of the i×s group (24 groups), specifically including the 6 groups corresponding to tp1 of dp2: the second training card number of the first group includes dp1pp1tp1, dp2pp2tp1, dp3pp1tp1, the second training card number of the second group includes dp1pp2tp1, dp2pp2tp1, dp3pp2tp1, and the second training card number of the third group includes dp1p The training cards for the fourth group include p2tp1, dp2pp3tp1, and dp3pp2tp1. The training cards for the fifth group include dp1pp3tp1, dp2pp4tp1, and dp3pp3tp1. The training cards for the sixth group include dp1pp4tp1, dp2pp4tp1, and dp3pp4tp1. The execution device can then control the input of the training samples from the faulty card to dp2pp2tp1 (i.e., the first training card to undertake the training task), and train dp1pp1tp1's training task through dp2pp2tp1; then output it back to the input of its own dp2pp2tp1, and dp2pp2tp1 trains its own training task (i.e., training pp2tp1(a) in the diagram); then output it to dp2pp3tp1 (i.e., training pp2tp1(b) in the diagram), and dp2pp3... tp1 first trains the remaining training tasks of dp2pp2tp1, then outputs it back to the input of dp2pp3tp1, and then dp2pp3tp1 trains its own training tasks (i.e., training the training tasks of pp3tp1(a) in the figure); then it outputs to dp2pp4tp1, and dp2pp4tp1 first trains the remaining training tasks of dp2pp3 (i.e., training the training tasks of pp3tp1(b) in the figure), and finally outputs it back to dp2pp4tp1, and dp2pp4tp1 trains its own training tasks.

[0225] The training card in the pipelined parallel unit of the data parallel unit where the faulty card resides performs the corresponding gradient calculation operation according to the result of the re-divided training task. Other data parallel units (such as...) Figure 8B The operations for calculating gradients in dp1 and dp3 remain unchanged.

[0226] Specifically, by controlling the training card corresponding to the reference training card number required for the re-division of the training task to send the model parameters of the current model training to the second training card corresponding to the second training card number in the same group, the training card that originally undertook the training task of the faulty card can continue training from the current training without having to go back to the most recent checkpoint.

[0227] For example, such as Figure 8BAs shown, if pp1 in dp2 fails, since the model weights and other parameters required for model training are exactly the same on the training cards of the same pipeline and unit number in different data parallel units (for example, the model weights and other parameters required for model training are exactly the same on the training card of pp1 in dp1 and the training card of pp1 in dp2), when pp1 in dp2 fails, the parameters required to execute the training task of pp1 in dp2 can be obtained from pp1 in dp1 or pp1 in dp3. After the control device decides which training card to select as the reference training card, it sends the pipeline parallel unit number (i.e., the reference training card number) of pp1 in dp1 or pp1 in dp3 to the execution device. Correspondingly, Figure 8C In one embodiment, after the control device decides which training card to select as the reference training card, it sends the number of tp1 in dp1 pp1 or tp1 in dp3 pp1 (i.e., the reference training card number) to the execution device, and sends the number of tp2 in dp1 pp1 or tp2 in dp3 pp1 (i.e., the reference training card number) to the execution device.

[0228] S8052: Gradient synchronization.

[0229] A new communication connection is established between the pipeline parallel unit in the data parallel unit where the faulty card is located and the pipeline parallel unit in other data parallel units that carry the same training task. For details on how to establish a new communication link, refer to step S804. Gradient synchronization (gradient averaging) is performed through the new communication connection and the existing communication connection.

[0230] In this embodiment of the application, when the fault card calculates the gradient, the existing gradient calculation optimization method can be used. However, when synchronizing the gradient, more complex communication group planning is required, that is, the parallel group information in step S803 is changed. This will not be described in detail here.

[0231] In other embodiments, each pipelined parallel unit in the training cluster can be further divided into s tensor parallel units, each tensor parallel unit typically containing a training card, where s is an integer greater than 1. A faulty training card corresponds to at least one tensor parallel unit in a pipelined parallel unit.

[0232] The control device confirms the faulty card number based on the anomaly information and then confirms the parallel grouping information based on the faulty card number. The parallel grouping information includes the third reassignment grouping information of the training cards in the training cluster. This third reassignment grouping information includes q groups of third training card numbers, each group containing n teams of third training card numbers. The third training card numbers in different teams indicate different data parallel units where the third training card resides. Here, q is an integer greater than or equal to 1 and less than s. Each group of third training card numbers includes the third team of training card numbers. The target third training card is the third training card indicated by the q third team training card numbers included in the q groups of training card numbers. The target third training card and the faulty card are in the same pipeline parallel unit and perform the same training task as the target tensor parallel unit. The target tensor parallel unit includes tensor parallel units that perform the same training task as the faulty card but are in different data parallel units. The parallel grouping information also includes the reference training card number required for repartitioning the training task. The training card (one or more) indicated by the reference training card number is in a different data parallel unit than the faulty card but performs the same training task.

[0233] Based on the parallel grouping information sent by the control device, the execution device establishes a connection between the target third training card and the target tensor parallel unit.

[0234] The execution end device control reference training card number corresponds to the training card that sends the model parameters of the current model training to the third training card indicated by the q third training card numbers.

[0235] The above method embodiments include many possible implementation schemes, which are described below in conjunction with... Figure 9 The training cluster shown illustrates some implementation schemes. It should be noted that... Figure 9 Any related concepts, operations, or logical relationships not explained in the corresponding embodiments can be described in the corresponding descriptions in the above embodiments.

[0236] In this application, Figure 9 The corresponding embodiment can be considered as a separate embodiment. Figure 9 The corresponding embodiments can all be implemented without relying on the technical solutions of the above embodiments; Figure 9 Some steps in the corresponding embodiments can also be used as separate embodiments.

[0237] In this embodiment, the above-mentioned abnormal information is a timeout abnormal signal (including the type of timeout abnormality, such as communication abnormality) and the number of the training card that generated the timeout abnormal signal in the entire training cluster. The third reassignment grouping information of the training cards in the training cluster is a parallel grouping strategy. The third training card number of the above-mentioned s groups is s communication groups. The third training card number of the above-mentioned q teams is the training card number of the q teams. The reference training card number required for repartitioning the training task is the training card number required for repartitioning the training task.

[0238] exist Figure 9 In the corresponding example, n equals 3, m equals 3, and s equals 4.

[0239] Next, using the model's training cluster as an example... Figure 9 Taking the above embodiment as an example, a specific embodiment corresponding to the above embodiment will be described in detail. The training cluster is divided into three data parallel units, numbered dp1, dp2, and dp3. Each data parallel unit is further divided into four pipelined parallel units, numbered pp1, pp2, pp3, and pp4. Each pipelined unit is also divided into four tensor parallel units, numbered tp1, tp2, tp3, and tp4. For ease of description, in this embodiment, the unit number (e.g., data parallel unit) is used to refer to the corresponding unit. For example, dp1pp1tp1 refers to the tensor parallel unit numbered tp1 within the pipelined parallel unit numbered pp1 in the data parallel unit numbered dp1, and dp1pp1 refers to the pipelined parallel unit numbered pp1 within the data parallel unit numbered pp1.

[0240] In a data parallel unit, if a tensor parallel unit fails, at least one tensor parallel unit in the same pipelined parallel unit as the failed tensor parallel unit will take over the tasks of the failed tensor parallel unit. Figure 9 Taking the training card on dp2pp1tp1 as an example of a training cluster failure before the failure, this application's embodiment is described. The flowchart of this embodiment can be found in the provided text. Figure 7A The specific process is as follows:

[0241] S701: Start training tasks on all training cards.

[0242] At the start of training, the execution device initiates the training task according to a hybrid parallel training strategy. The training cluster is divided into n parallel data parallel units, performing data parallelism. Within each data parallel unit, the training card is divided into m pipelined parallel units, and pipelined parallel training is performed between pipelined parallel units within the same data parallel unit. Within each pipelined parallel unit, the training card is divided into s tensor parallel units, and tensor parallel computation is performed between tensor parallel units within the same pipelined parallel unit. For example, as shown... Figure 9As shown, the training cluster can be divided into 3 data parallel units, numbered dp1, dp2, and dp3 respectively. Each data parallel unit is further divided into 4 pipeline parallel units, numbered pp1, pp2, pp3, and pp4 respectively. The 4 pipeline parallel units within each data parallel unit are trained in the following order: pp1, pp2, pp3, pp4. Each pipeline parallel unit is also divided into 4 tensor parallel units, numbered tp1, tp2, tp3, and tp4 respectively. Typically, each tensor parallel unit contains only one training card.

[0243] The communication connections between the tensor parallel units in the training cluster are as follows: Figure 9 As shown, this communication connection can be a hardware connection or established via wireless transmission. This connection can be established at the start of model training, or it can be established when communication is needed between the various tensor parallel units. For example, when each data parallel unit calculates its own gradient using sample data, only a hardware connection needs to be established. Figure 9 The communication connection indicated by the bidirectional arrows indicates that communication occurs within each data parallel unit. When performing gradient synchronization after calculating its own gradient, communication between data parallel units is required. In this case, only communication as shown by the bidirectional arrows is needed. Figure 9 The communication connection shown by the solid line is used for gradient synchronization. The method and establishment time of the communication connection are not limited in this embodiment.

[0244] Each data parallel unit includes a complete set of model weights and training-related parameters. The difference between different data parallel units lies in the input sample data. During training, each data parallel unit calculates the gradient using the exact same gradient calculation method, and gradient averaging is performed between different data parallel units.

[0245] S702: A training card has malfunctioned.

[0246] During training, if a training card in the training cluster malfunctions (i.e., a faulty card appears), preventing the training task from continuing or significantly slowing down the task execution speed, all training cards in the training cluster will suspend their training tasks, as described in step S3031. The execution device captures the timeout exception signal of the training card in the training cluster and reports the S7.100 timeout exception signal (including the type of timeout exception, such as a communication exception) and the number of the training card that generated the timeout exception signal in the entire training cluster to the control device. For example, as shown... Figure 9If pp1tp1 in dp2 fails, then all pps except the failed pp1tp1 can send a timeout error signal (carrying their own training card number) to the control terminal device, that is, send the error information of this application embodiment.

[0247] S703: Receives timeout error signal, confirms fault card number, and calculates parallel grouping information.

[0248] The control device receives anomaly information (including a timeout anomaly signal and the training card number that generated the timeout anomaly signal within the entire training cluster) reported by the execution device. It confirms the faulty card's number within the training cluster. For example, analysis of the received anomaly information reveals that only pp1tp1 in dp2 did not emit a timeout anomaly signal; therefore, the faulty card is identified as pp1tp1 in dp2, and its number corresponds to tp1 of pp1 in dp2. Based on this number, it calculates parallel grouping information, which may also include the S7.200 parallel grouping strategy and the reference training card number required for repartitioning the training task. Before confirming the parallel grouping strategy, it is necessary to identify the tensor parallel unit that will undertake the training task of the tensor parallel unit where the original faulty card resides. That is, at least one tensor parallel unit other than the tensor parallel unit where the faulty card resides must be selected from the pipeline parallel units. For example,... Figure 9 As shown, if dp2pp1tp1 fails, dp2pp1tp2 and dp2pp1tp3 can be selected to take over the training task originally performed on dp2pp1tp1. In other embodiments, multiple tensor parallel units can be selected to take over the training task on the tensor parallel unit where the faulty card was located. The selected tensor parallel units are in the same pipelined parallel unit as the tensor parallel unit where the faulty card was located.

[0249] The parallel grouping strategy includes communication groups (i.e., the third training card number of group q) of training cards that need to establish new communication links when the training cluster continues training. Each communication group includes multiple training card numbers (equivalent to each group of third training card numbers including n third training card numbers), which means that a new communication link needs to be established between these multiple training cards.

[0250] For example, such as Figure 9As shown, when the control device confirms that the fault card number in the training cluster is dp2pp1tp1, it can calculate a parallel grouping strategy based on the fault card number. The parallel grouping strategy includes two communication groups (i.e., q equals 2). One communication group contains dp1pp1tp1, dp2pp1tp2, and dp3pp1tp1, and the other communication group contains dp1pp1tp1, dp2pp1tp3, and dp3pp1tp1. Therefore, the third allocation group information can include two groups of first training card numbers for communication. Each communication group can be divided into three training card numbers (i.e., n equals 3). For example, in the communication group dp1pp1tp1, dp2pp1tp2, dp3pp1tp1, dp1pp1tp1 is one training card number, dp2pp1tp2 is another training card number, and dp3pp1tp1 is the third training card number. Similarly, in the communication group dp1pp1tp1, dp2pp1tp3, dp3pp1tp1, dp1pp1tp1 is one training card number, dp2pp1tp3 is another training card number, and dp3pp1tp1 is the third training card number.

[0251] This parallel grouping strategy is also used to instruct the execution-end device to control the training cluster to establish connections between the target third training card and the target tensor parallel unit, including creating new connections such as... Figure 9 The communication connections shown refer to establishing connections between the three tensor parallel units pp1tp1 of dp1, pp1tp2 of dp2, and pp1tp1 of dp3, and between the three tensor parallel units pp1tp1 of dp1, pp1tp3 of dp2, and pp1tp1 of dp3. In other words, the target third training card is the first training card indicated by the two third-team training card numbers included in the two sets of training card numbers; in this embodiment, the two third-team training card numbers are dp2pp1tp2 and dp2pp1tp3, which perform the same training task as the target tensor parallel unit (i.e., dp1pp1tp1 or dp3pp1tp1).

[0252] The reference training card number required for repartitioning the training task is the number of the training card that is in a different data parallel unit from the faulty card and performs the same training task. For example, such as... Figure 9As shown, if dp1pp1tp1 fails, the model weights and other parameters required for model training are identical on training cards with the same pipeline unit number and tensor parallel number across different data parallel units. For example, the model weights and other parameters required for model training are exactly the same on the training cards in dp1pp1tp1 and dp2pp1tp1. Therefore, when dp2pp1tp1 fails, the parameters required to execute the training task of dp2pp1tp1 can be obtained from dp1pp1tp1 or dp3pp1tp1. After the control device decides which training card to select as the reference training card, it sends the training card number (i.e., the reference training card number) from dp1pp1tp1 or dp3pp1tp1 to the execution device.

[0253] S704: Establish a new communication link.

[0254] The execution device establishes a new communication connection according to the parallel grouping strategy issued by the control device.

[0255] For example, with Figure 9 For example, the execution end device controls the creation of a training cluster. Figure 9 The communication connection shown refers to establishing a connection between the three tensor parallel units pp1tp1 of dp1, pp1tp2 of dp2, and pp1tp1 of dp3, and establishing a connection between the three tensor parallel units pp1tp1 of dp1, pp1tp3 of dp2, and pp1tp1 of dp3.

[0256] It should be noted that the communication connection can be established after receiving the parallel grouping strategy and then disconnected after the entire training cluster has finished training, that is, the connection is maintained throughout the entire training process. Alternatively, a short-term connection can be established when gradient synchronization is required for gradient transmission and then disconnected after the corresponding data is transmitted. This application does not limit this.

[0257] S705: Redivide the training task.

[0258] The execution device sends model parameters to the target training card (which may include the third training card indicated by q third team training card numbers) according to the reference training card number required for the repartition training task issued by the control device. The target training card is the training card in the pipeline parallel unit that undertakes the training task of the pipeline parallel unit where the original fault card is located.

[0259] by Figure 9For example, assuming the reference training card number is dp1pp1tp1, the q third training card numbers indicate two third training cards, namely dp2pp1tp2 and dp2pp1tp3. The execution device controls dp1pp1tp1 to send its currently trained model parameters to dp2pp2tp2 and dp2pp3tp3. That is, dp2pp2tp2 and dp2pp3tp3 can continue training based on these currently trained model parameters during training tasks, without needing to return to a checkpoint.

[0260] When repartitioning the training task, if the training card in the selected tensor parallel unit (i.e., at least one tensor parallel unit in the same pipeline parallel unit as the faulty card) has insufficient storage space to store the partitioned training task, the number of selected tensor parallel units can be increased. This process is completed in step S703 when the control device calculates the parallel grouping strategy. The parameters allocated to the selected tensor parallel units during the repartitioning of the training task are transmitted using the newly established communication connection in S704.

[0261] S706: The training cluster continues training.

[0262] After completing the task of establishing a new communication connection and re-dividing the training cluster, the execution device can control the training cluster to continue training.

[0263] When the training cluster continues training, each model training iteration mainly includes the following two steps:

[0264] S7061: Gradient calculation.

[0265] The execution device inputs the sample text used for model training into pp1tp of each data parallel unit. For data parallel units with faulty cards, if dp2pp1tp1 fails, ... Figure 9 For example, suppose tp2 and tp3 of pp1 in dp2 are chosen to undertake the training task originally performed on the fault card (dp2pp1tp1). Then dp2pp1tp2 and dp2pp1tp3 will jointly undertake the training content originally performed on the fault card (dp2pp1tp1) in parallel.

[0266] Other pipelined parallel units (such as the pipelined parallel unit where the faulty card resides and the selected tensor parallel unit) within the data parallel unit where the faulty card resides. Figure 9 The gradient calculation operations of pp2, pp3, pp4, etc. in dp2 remain unchanged. The selected tensor parallel units, in addition to completing the gradient calculations that should have been completed before the faulty card appeared, also need to undertake the gradient calculations for the training tasks originally assigned to the faulty card. Other data parallel units (such as...) Figure 9 The gradient calculation operations for dp1 and dp3 remain unchanged.

[0267] by Figure 9 For example, suppose TP2 and TP3 of PP1 in DP2 are chosen to undertake the training tasks originally performed on the fault card (DP2PP1TP1). This means that DP2PP1TP2 and DP2PP1TP3 not only train the original training tasks on the fault card but also train the original training tasks in parallel. DP2PP1TP4 also trains its original training tasks in parallel.

[0268] Understandably, when the data parallel unit where the faulty card is located resumes training, it can appropriately reduce the number of sample data to reduce the workload of the selected tensor parallel unit (i.e., at least one tensor parallel unit among other tensor parallel units in the same data parallel unit and pipeline parallel unit as the faulty card), so as not to drag down the gradient synchronization time of other data parallel units.

[0269] S7062: Gradient synchronization.

[0270] During gradient synchronization, through a newly established communication link, the target tensor parallel unit (i.e., the tensor parallel unit) located in a different data parallel unit from the faulty card and with the same pipeline parallel numbering and tensor parallel numbering as the faulty card performs gradient synchronization (gradient averaging) with the selected tensor parallel unit; the gradient synchronization operation of the training cards in the remaining tensor parallel units remains unchanged. For example, ... Figure 9 As shown, the gradient synchronization operation of the training card in the tensor parallel unit, except for dp1pp1tp1, dp2pp1tp2, and dp3pp1tp1, remains unchanged. Gradient synchronization between dp2pp1tp2 and dp1pp1tp1, and dp3pp1tp1 is performed through a newly established communication connection. Gradient synchronization between dp2pp1tp3 and dp1pp1tp1, and dp3pp1tp1 is also performed through a newly established communication connection.

[0271] Based on the previous embodiment, when a fault occurs in a tensor parallel unit, the training tasks in that tensor parallel unit can be distributed according to preset rules (e.g., evenly distributed) to all tensor parallel units in the pipeline parallel unit containing that tensor parallel unit, excluding the faulty tensor parallel unit. The embodiments of this application are described below.

[0272] In this embodiment, each of the n data parallel units is divided into m pipelined parallel units, and each of the m pipelined parallel units is divided into s tensor parallel units, where m is an integer greater than 1 and s is an integer greater than 1. A faulty training card corresponds to at least one tensor parallel unit within a pipelined parallel unit. When multiple faulty training cards are simultaneously in multiple different pipelined parallel units, the processing behavior of each faulty pipelined parallel unit can be the same, which will not be elaborated upon in this application.

[0273] The control device confirms the faulty card number based on the anomaly information and then confirms the parallel grouping information based on the faulty card number. The parallel grouping information includes the fourth reassignment grouping information of the training cards in the training cluster. This fourth reassignment grouping information includes j groups of fourth training card numbers, each group comprising n teams of fourth training card numbers. The fourth training cards in different teams are located in different data parallel units, and the fourth training cards in each team within the same group undertake the same training task. j is the number of groups after regrouping the training task undertaken in the target tensor parallel unit based on the absence of faulty training cards in the pipeline parallel unit where the faulty card is located; j is greater than or equal to s. The parallel grouping information also includes the reference training card number required for repartitioning the training task. The training card (one or more) indicated by the reference training card number is in a different data parallel unit than the faulty card and undertakes the same training task.

[0274] The execution device, based on the parallel grouping information received from the control device, controls the establishment of connections between the fourth training cards indicated by the fourth training card numbers in each group of fourth training card numbers. Each group of fourth training card numbers includes the fourth training card number of the fourth team. The fourth training cards indicated by the j fourth training card numbers in the j groups of fourth training card numbers are in the same pipeline parallel unit as the faulty card. The j groups of fourth training card numbers include the target group of fourth training card numbers. The fourth training cards indicated by the fourth training card numbers in the target group of fourth training card numbers perform the same training task as the target tensor parallel unit. The target tensor parallel unit includes tensor parallel units that are in different data parallel units than the faulty card but perform the same training task.

[0275] The execution end device control reference training card number corresponds to the training card number and sends the model parameters of the current model training to the third training card indicated by the fourth training card number in the same group.

[0276] The above method embodiments include many possible implementation schemes, which are described below in conjunction with... Figure 10 The training cluster shown illustrates some implementation schemes. It should be noted that... Figure 10Any related concepts, operations, or logical relationships not explained in the corresponding embodiments can be described in the corresponding descriptions in the above embodiments.

[0277] In this application, Figure 10 The corresponding embodiment can be considered as a separate embodiment. Figure 10 The corresponding embodiments can all be implemented without relying on the technical solutions of the above embodiments; Figure 10 Some steps in the corresponding embodiments can also be used as separate embodiments.

[0278] In this embodiment, the above-mentioned abnormal information is a timeout abnormal signal (including the type of timeout abnormality, such as communication abnormality) and the number of the training card that generated the timeout abnormal signal in the entire training cluster. The fourth reassignment grouping information of the training cards in the training cluster is a parallel grouping strategy. The number of the fourth training card in the j-group is j communication groups. The number of the fourth training card in the n-team is the number of the training card in the n-team. The reference training card number required for reassigning the training task is the training card number required for reassigning the training task.

[0279] exist Figure 10 In the corresponding example, n equals 3, m equals 4, and s equals 4.

[0280] Next, using the model's training cluster as an example... Figure 10 Taking the above embodiment as an example, we will describe in detail one embodiment corresponding to the above embodiment. Figure 10 The training cluster shown is divided into three data parallel units, numbered dp1, dp2, and dp3. Each data parallel unit is further divided into four pipelined parallel units, numbered pp1, pp2, pp3, and pp4. Each pipelined unit is also divided into four tensor parallel units, numbered tp1, tp2, tp3, and tp4. For ease of description, in this embodiment, the unit number (e.g., data parallel unit) is used to refer to the corresponding unit. For example, dp1pp1tp1 refers to the tensor parallel unit numbered tp1 within the pipelined parallel unit numbered pp1 in the data parallel unit numbered dp1, and dp1pp1 refers to the pipelined parallel unit numbered pp1 within the data parallel unit numbered dp1.

[0281] It should be noted that, in Figure 10 The communication connection method of the training cluster shown before the failure can be referenced. Figure 9 As shown. The model training process can be found in the following figure. Figure 8A As shown, the specific process is as follows:

[0282] S801: Start training tasks on all training cards.

[0283] At the start of training, the execution device initiates the training task according to a hybrid parallel training strategy. The training cluster is divided into n parallel data parallel units, performing data parallelism. Within each data parallel unit, the training card is divided into m pipelined parallel units, and pipelined parallel training is performed between pipelined parallel units within the same data parallel unit. Within each pipelined parallel unit, the training card is divided into s tensor parallel units, and tensor parallel computation is performed between tensor parallel units within the same pipelined parallel unit. For example, as shown... Figure 10 As shown, the training cluster can be divided into 3 data parallel units, numbered dp1, dp2, and dp3 respectively. Each data parallel unit is further divided into 4 pipeline parallel units, numbered pp1, pp2, pp3, and pp4 respectively. There is a sequential order among the 4 pipeline parallel units in each data parallel unit, for example, training is performed in the order of pp1, pp2, pp3, and pp4. Each pipeline parallel unit is also divided into 4 tensor parallel units, numbered tp1, tp2, tp3, and tp4 respectively. Typically, each tensor parallel unit contains only one training card.

[0284] The communication connections between the tensor parallel units in the training cluster can be found in [reference]. Figure 9 As shown, it will not be elaborated upon here.

[0285] Each data parallel unit includes a complete set of model weights and training-related parameters. The difference between different data parallel units lies in the input sample data. During training, each data parallel unit calculates the gradient using the exact same gradient calculation method, and gradient averaging is performed between different data parallel units.

[0286] S802: A training card has malfunctioned.

[0287] During training, if a training card in the training cluster malfunctions (i.e., a faulty card appears), preventing the training task from continuing or significantly slowing down the task execution speed, all training cards in the training cluster will suspend their training tasks, as described in step S3031. The execution device captures the timeout exception signal of the training card in the training cluster and reports the S8.100 timeout exception signal (including the type of timeout exception, such as communication exception) and the number of the training card that generated the timeout exception signal in the entire training cluster to the control device.

[0288] S803: Receives timeout error signal, confirms fault card number, and calculates parallel grouping information.

[0289] The control device receives a timeout error signal reported by the execution device, along with the training card's ID within the entire training cluster. It confirms the faulty card's ID within the cluster and calculates parallel grouping information based on this ID. This parallel grouping information includes the S8.200 parallel grouping strategy and the training card IDs required to repartition the training tasks. The parallel grouping information can be used to instruct the reallocation (e.g., equal distribution) of all training tasks on the pipeline parallel unit containing the faulty card to the remaining tensor parallel units within that unit. The remaining tensor parallel units are all tensor parallel units in the pipeline parallel unit containing the faulty card, excluding the unit itself. For example, as shown... Figure 10 As shown, if dp2pp1tp1 fails, the training tasks originally on dp2pp1tp1 can be redistributed equally to dp2pp1tp2, dp2pp1tp3, and dp2pp1tp4.

[0290] The parallel grouping strategy involves creating communication groups for training cards that require new communication links during training cluster continuation. Each communication group includes multiple training card numbers, indicating that new communication links need to be established between these multiple training cards. For example,... Figure 10 As shown, when the control device confirms that the faulty card number in the training cluster is dp2pp1tp1, it can regroup the training tasks undertaken in the target tensor parallel unit based on the absence of faulty training cards in the pipeline parallel unit where the faulty card resides. For example, after regrouping, the number of groups is 6. The corresponding calculated or generated fourth reassignment grouping information includes 6 groups of second training card numbers, specifically:

[0291] The second training cards for Group 1 are numbered dp1pp1tp1, dp2pp1tp2, and dp3pp1tp1; the second training cards for Group 2 are numbered dp1pp1tp2, dp2pp1tp2, and dp3pp1tp2; the second training cards for Group 3 are numbered dp1pp1tp2, dp2pp1tp3, and dp3pp1tp2; the second training cards for Group 4 are numbered dp1pp1tp3, dp2pp1tp3, and dp3pp1tp3; the second training cards for Group 5 are numbered dp1pp1tp3, dp2pp1tp4, and dp3pp1tp3; and the second training cards for Group 6 are numbered dp1pp1tp4, dp2pp1tp4, and dp3pp1tp4.

[0292] Each communication group can be divided into three training card numbers (i.e., n equals 3). For example, in communication groups dp1pp1tp1, dp2pp1tp2, and dp3pp1tp1, dp1pp1tp1 is one training card number, dp2pp2tp2 is another, and dp3pp1tp1 is the third. Similarly, in communication groups dp1pp1tp2, dp2pp1tp2, and dp3pp1tp2, dp1pp1tp2 is one training card number, dp2pp1tp2 is another, and dp3pp1tp2 is the third. The training card numbers for the remaining groups follow the same pattern.

[0293] This parallel grouping strategy is also used to instruct the execution end device to control the training cluster to establish connections between the fourth training cards of each team indicated by the fourth training card number in each group; including creating new connections such as... Figure 10 The communication connections shown are established between dp1pp1tp1, dp2pp1tp2, and dp3pp1tp1; between dp1pp1tp2, dp2pp1tp3, and dp3pp1tp2 (not shown in the diagram); and between dp1pp1tp3, dp2pp1tp4, and dp3pp1tp3 (not shown in the diagram). Connections to other groups are similar and will not be described further.

[0294] In other words, in the embodiments of this application, the fourth training card number of the target group includes the fourth training card number indicating the fourth training card, which is dp2pp2tp2 (the training task of the original fault card is completed through this training card), and it undertakes the same training task as the target pipeline parallel unit (i.e., dp1pp1tp1, dp3pp1tp1).

[0295] The reference training card number required for repartitioning the training task is the number of the training card that is in a different data parallel unit from the faulty card and is performing the same training task. For example, such as... Figure 10 As shown, if pp1tp1 in dp2 fails, since the model weights and other parameters required for model training are exactly the same on the training cards of different data parallel units within the same pipeline and with the same unit number, for example, the model weights and other parameters required for model training are exactly the same on the training cards of pp1tp1 in dp1 and pp1tp1 in dp2. Therefore, when pp1tp1 in dp2 fails, the parameters required to execute the training task of pp1tp1 in dp2 can be obtained from pp1tp1 in dp1 or pp1tp1 in dp3. After the control device decides which training card to select as the reference training card, it sends the pipeline parallel unit number (i.e., the reference training card number) of pp1tp1 in dp1 or pp1tp1 in dp3 to the execution device. Correspondingly, Figure 10In one embodiment, after the control device decides which training card to select as the reference training card, it sends the number of tp1 in dp1 pp1 or tp1 in dp3 pp1 (i.e., the reference training card number) to the execution device.

[0296] S804: After confirming the results of the repartitioning task, establish a new communication link and repartition the training task.

[0297] The execution device establishes new communication connections according to the parallel grouping strategy issued by the control device. This includes establishing connections between the fourth training cards of each team indicated by the fourth training card number in each group, including creating connections such as... Figure 10 The communication connections shown are established between dp1pp1tp1, dp2pp1tp2, and dp3pp1tp1; between dp1pp1tp2, dp2pp1tp3, and dp3pp1tp2 (not shown in the diagram); and between dp1pp1tp3, dp2pp1tp4, and dp3pp1tp3 (not shown in the diagram). Connections to other groups are similar and will not be described further.

[0298] It should be noted that the communication connection can be established after receiving the parallel grouping strategy and then disconnected after the entire training cluster has finished training, that is, the connection is maintained throughout the entire training process. Alternatively, a short-term connection can be established when gradient synchronization is required for gradient transmission and then disconnected after the corresponding data is transmitted. This application does not limit this.

[0299] During the process of re-dividing the training tasks, the execution device, according to the reference training card number required for the re-dividing training tasks issued by the control device, controls the training card corresponding to the reference training card number required for the re-dividing training tasks to send model parameters to the second training card corresponding to the fourth training card number in the same group. The second training card corresponding to the fourth training card number in the same group is the training card in the pipeline parallel unit that undertakes the training task of the pipeline parallel unit where the original faulty card was located. The execution device can also control other training cards in the re-divided training tasks to obtain the model parameters required for training. For example, it can control pp3tp3 in dp2 to obtain the model parameters required for training pp1tp2(b) in its own dp2.

[0300] S805: The training cluster continues training.

[0301] After completing the task of establishing a new communication connection and re-dividing the training cluster, the execution device can control the training cluster to continue training.

[0302] When the training cluster continues training, each model training iteration mainly includes the following two steps:

[0303] S8051: Gradient calculation.

[0304] During gradient calculation, the gradient calculation operations of the tensor parallel units (i.e., the tensor parallel units in the pipeline parallel units that are not experiencing faults) remain unchanged, except for the tensor parallel units in the pipeline parallel unit where the faulty card is located. The remaining tensor parallel units in the pipeline parallel unit where the faulty card is located complete the re-partitioning training task. For example, as shown... Figure 10 As shown, dp2pp1tp2 performs all training tasks on the original dp2pp1tp1 and the original dp2pp1tp2 (i.e., the training task of training pp1tp2(a) in the figure) in parallel. dp2pp1tp3 performs the training tasks on the original dp2pp1tp2 (i.e., the training task of training pp1tp2(b) in the figure) and the original dp2pp1tp3 (i.e., the training task of training pp1tp3(a) in the figure) in parallel. dp2pp1tp4 performs the training tasks on the original dp2pp1tp3 (i.e., the training task of training pp1tp3(b) in the figure) and all training tasks on the original dp2pp1tp3 in parallel. The operations on other tensor parallel units remain unchanged.

[0305] It is understandable that when the data parallel unit where the faulty card is located continues training, it can appropriately reduce the number of sample data to reduce the workload of the tensor parallel unit in the pipeline parallel unit where the faulty card is located, so as not to delay the gradient synchronization time of other data parallel units.

[0306] S8052: Gradient synchronization.

[0307] During gradient synchronization, gradient synchronization (gradient averaging) is performed between the tensor parallel unit in the pipeline parallel unit where the faulty card is located and the tensor parallel unit that performs the same training task, through the newly established communication link; the gradient synchronization operation of the training cards in the remaining tensor parallel units remains unchanged.

[0308] The aforementioned model training method, when a faulty card appears in the training cluster, does not replenish redundant cards, avoiding the time waste caused by the inability to replenish them. This application replaces the existing method of deleting the entire data parallel unit by deleting only the smallest unit (pipeline parallel unit or tensor parallel unit) where the fault occurred, thus reducing the waste of training card resources. This application also uses model parameters from training cards that perform the same task in different data parallel units than the faulty card to continue training the training card that originally took over the task of the faulty card. This eliminates the need to roll back to the last checkpoint node saved before the fault and uses the model parameters in the checkpoint file for continued training, avoiding the waste of training resources caused by rolling back to the last checkpoint node saved before the fault.

[0309] It is understood that, in order to achieve the aforementioned functions, the device includes corresponding hardware structures and / or software modules for performing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0310] This application embodiment can divide the model training device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0311] See Figure 11 , Figure 11 This is a schematic diagram of a model training device provided in an embodiment of this application. The model training device 11 can be applied to the above-described... Figures 7A to 10 In any of the embodiments shown in the method, such as Figure 11 As shown, the model training device 11 includes a processing module 110 and a transceiver module 111. The processing module 110 may be one or more processors, and the transceiver module 111 may be a transceiver or a communication interface. This model training device can be used to implement the steps involved in the control or execution of any of the above method embodiments. Optionally, the model training device 11 may further include a storage module 112 for storing the program code and data of the model training device 11.

[0312] In one example, the model training device acts as a control terminal device or is a chip applied in a control terminal device, and executes the steps performed by the control terminal device in the above method embodiments. The transceiver module 111 is used for specific execution. Figures 7A to 10 The sending and / or receiving actions performed by the control device in any of the embodiments may be, for example, other processes performed by the control device using the techniques described herein. The processing module 110 may be used to support the model training device 11 in performing the processing actions in the above method embodiments, for example, to support the control device in performing other processes using the techniques described herein.

[0313] For example, transceiver module 111 is used to receive exception information from the execution device and to send parallel packet information to the execution device.

[0314] The processing module 110 is used to confirm the fault card number based on the abnormal information; confirm the parallel grouping information based on the fault card number; the parallel grouping information is used to establish a new communication link for the training cards in the training cluster, and to reallocate the training tasks of the training cards other than the fault card in the data parallel unit where the fault card is located.

[0315] In one possible implementation, the parallel grouping information includes reference training card numbers required for repartitioning the training task. The training cards indicated by the reference training card numbers are in different data parallel units and undertake the same training task as the faulty cards. The training cards indicated by the reference training card numbers provide model parameters for the subsequent training cluster to continue training.

[0316] In one possible implementation, each of the n data parallel units is divided into m pipelined parallel units; each pipelined parallel unit includes s tensor parallel units; the parallel grouping information includes the first reassignment grouping information of the training cards in the training cluster, the first reassignment grouping information includes k×s group first training card numbers, each group first training card number includes n teams first training card numbers, the first training card number in each team of the same group of first training card numbers indicates that the first training card undertakes the same training task, and the first training card number in different teams indicates that the first training card is in a different data parallel unit; m is an integer greater than 1; k is an integer greater than or equal to 1 and less than m; s is an integer greater than or equal to 1.

[0317] Wherein, each group of first training card numbers includes the first team training card number, and the first training card indicated by the k×s first team training card numbers in the k×s group of first training card numbers undertakes the same training task as the target pipeline parallel unit. The target pipeline parallel unit includes a pipeline parallel unit that is in a different data parallel unit than the faulty card and undertakes the same training task.

[0318] In one possible implementation, each of the plurality of data parallel units is divided into m pipelined parallel units; each pipelined parallel unit includes s tensor parallel units; the parallel grouping information includes the second reassignment grouping information of the training cards in the training cluster, the second reassignment grouping information including i×s group second training card numbers; each group second training card number includes n teams of second training card numbers, the second training card numbers of different teams indicate that the second training card is located in a different data parallel unit; the second training card numbers of each team in the same group of second training card numbers indicate that the second training card undertakes the same training task; m is an integer greater than 1; i is the number of groups after regrouping the number of layers of the training model based on the number of layers of the training model and the training cards in the pipelined parallel units without faults in the data parallel unit where the faulty card is located; i is greater than or equal to m; s is an integer greater than or equal to 1.

[0319] Each group of second training card numbers includes a second team training card number. The second training card indicated by the i×s second team training card numbers in the i×s group of second training card numbers is in the same data parallel unit as the faulty card, but not in the pipeline parallel unit where the faulty card is located. The i×s group of second training card numbers includes a target group of second training card numbers. The second training card indicated by the second team training card numbers in the target group of second training card numbers undertakes the same training task as the target pipeline parallel unit. The target pipeline parallel unit includes pipeline parallel units that are in different data parallel units from the faulty card but undertake the same training task.

[0320] In one possible implementation, each of the n data parallel units is divided into m pipelined parallel units, and each pipelined parallel unit includes s tensor parallel units; the faulty training card corresponds to at least one tensor parallel unit in a pipelined parallel unit; the parallel grouping information includes the third reassignment grouping information of the training cards in the training cluster, the third reassignment grouping information includes q groups of third training card numbers, each group of third training card numbers includes n teams of third training card numbers, and the third training card numbers of different teams indicate that the third training card is located in a different data parallel unit; m is an integer greater than 1; q is an integer greater than or equal to 1 and less than s; s is an integer greater than 1.

[0321] Each group of third training card numbers includes the third team training card number. The third training cards indicated by the q third team training card numbers in the q groups of third training card numbers are in the same pipeline parallel unit as the faulty card. The third training cards indicated by the q third team training card numbers undertake the same training task as the target tensor parallel unit. The target tensor parallel unit includes tensor parallel units that are in different data parallel units than the faulty card and undertake the same training task.

[0322] In one possible implementation, each of the n data parallel units is divided into m pipelined parallel units, and each pipelined parallel unit includes s tensor parallel units; the faulty training card corresponds to at least one tensor parallel unit in a pipelined parallel unit; the parallel grouping information includes the fourth reassignment grouping information of the training cards in the training cluster, the fourth reassignment grouping information includes j groups of fourth training card numbers, each group of fourth training card numbers includes n teams of fourth training card numbers, and the fourth training card numbers of different teams indicate that the fourth training card is located in a different data parallel unit; the fourth training card numbers of each team in the same group of fourth training card numbers indicate that the fourth training card undertakes the same training task; m is an integer greater than 1; s is an integer greater than 1; j is the number of groups after regrouping the training tasks undertaken in the target tensor parallel unit based on the training cards in the pipelined parallel unit where the faulty card is located without faults; j is greater than or equal to s.

[0323] Each group of fourth training card numbers includes the fourth team training card number. The fourth training card indicated by the j fourth team training card numbers in the j groups of fourth training card numbers is in the same pipeline parallel unit as the faulty card. The j groups of fourth training card numbers include the target group fourth training card number. The fourth training card indicated by the fourth team training card number in the target group fourth training card number undertakes the same training task as the target tensor parallel unit. The target tensor parallel unit includes a tensor parallel unit that is in a different data parallel unit than the faulty card and undertakes the same training task.

[0324] In one example, the model training device acts as an execution device or is a chip applied in an execution device, and performs the steps executed by the execution device in the above method embodiments. The transceiver module 111 is used for specific execution. Figures 7A to 10 The sending and / or receiving actions performed by the execution end device in any of the embodiments may be, for example, other processes performed by the execution end device using the techniques described herein. The processing module 110 may be used to support the model training device 11 in performing the processing actions in the above method embodiments, for example, to support the execution end device in performing other processes using the techniques described herein.

[0325] For example, transceiver module 111 is used to send exception information to the control terminal device and receive parallel packet information from the control terminal device.

[0326] The processing module 110 is used to control the training cluster to continue training the model based on the parallel grouping information.

[0327] In one possible implementation, each of the n data parallel units is divided into m pipelined parallel units; each pipelined parallel unit includes s tensor parallel units; the parallel grouping information includes the first reassignment grouping information of the training cards in the training cluster, the first reassignment grouping information includes k×s group first training card numbers, each group first training card number includes n teams first training card numbers, the first training card number in each team of the same group of first training card numbers indicates that the first training card undertakes the same training task, and the first training card number in different teams indicates that the first training card is in a different data parallel unit; m is an integer greater than 1; k is an integer greater than or equal to 1 and less than m; s is an integer greater than or equal to 1.

[0328] The processing module 110 controls the training cluster to continue training the model based on the parallel grouping information, including: establishing a connection between the target first training card and the target pipeline parallel unit;

[0329] Wherein, each group of first training card numbers includes the first team training card number, and the target first training card is the first training card indicated by the k×s first team training card numbers included in the k×s group of training card numbers; the target first training card and the target pipeline parallel unit undertake the same training task, and the target pipeline parallel unit includes a pipeline parallel unit that is in a different data parallel unit than the fault card and undertakes the same training task.

[0330] In one possible implementation, the parallel grouping information further includes a reference training card number required for repartitioning the training task, wherein the training card indicated by the reference training card number is in a different data parallel unit and undertakes the same training task as the faulty card; the processing module 110 is further configured to include:

[0331] The model parameters of the current model training in the training card corresponding to the reference training card number are sent to the first training card indicated by the k×s first team training card numbers.

[0332] In one possible implementation, the processing module 110 further includes controlling the training cluster to continue training the model based on the parallel grouping information:

[0333] The training sample of the fault card is input into the first training card that undertakes the training task among the k×s first training card numbers indicated by the first training card number of the first team;

[0334] After completing the gradient calculation, the first training card indicated by the k×s first team training card numbers is synchronized with the training card in the target pipeline parallel unit.

[0335] In one possible implementation, each of the plurality of data parallel units is divided into m pipelined parallel units; each pipelined parallel unit includes s tensor parallel units; the parallel grouping information includes the second reassignment grouping information of the training cards in the training cluster, the second reassignment grouping information including i×s groups of second training card numbers; each group of second training card numbers includes n teams of second training card numbers, the second training card numbers of different teams indicate that the second training card is located in a different data parallel unit; the second training card numbers of each team in the same group of training card numbers indicate that the second training card undertakes the same training task; m is an integer greater than 1; i is the number of groups after regrouping the number of layers of the training model based on the number of layers of the training model and the training cards in the pipelined parallel units without faults in the data parallel unit where the faulty card is located; i is greater than or equal to m; s is an integer greater than or equal to 1.

[0336] The processing module 110 controls the training cluster to continue training the model based on the parallel grouping information, including: establishing a connection between the second training cards of each team indicated by the second training card number in each group;

[0337] Each group of second training card numbers includes a second team training card number. The second training card indicated by the i×s second team training card numbers in the i×s group of second training card numbers is in the same data parallel unit as the faulty card, but not in the pipeline parallel unit where the faulty card is located. The i×s group of second training card numbers includes a target group of second training card numbers. The second training card indicated by the second team training card numbers in the target group of second training card numbers undertakes the same training task as the target pipeline parallel unit. The target pipeline parallel unit includes pipeline parallel units that are in different data parallel units from the faulty card but undertake the same training task.

[0338] In one possible implementation, the parallel grouping information further includes a reference training card number required for repartitioning the training task, the reference training card number indicating a training card that is in a different data parallel unit than the faulty card and performs the same training task; the processing module 110 is further configured to include:

[0339] The model parameters of the current model training in the training card corresponding to the reference training card number are sent to the second training card corresponding to the second training card number in the same group.

[0340] In one possible implementation, the processing module 110 further includes controlling the training cluster to continue training the model based on the parallel grouping information:

[0341] The training sample of the fault card is input into the first training card in the second training card of the second training card number, which is included in the second training card number of the target group;

[0342] After completing the gradient calculation, the second training card indicated by the second training card number included in the second training card number of the target group is gradient synchronized with the training card in the target pipeline parallel unit.

[0343] In one possible implementation, each of the n data parallel units is divided into m pipelined parallel units, and each pipelined parallel unit includes s tensor parallel units; the faulty training card corresponds to at least one tensor parallel unit in a pipelined parallel unit; the parallel grouping information includes the third reassignment grouping information of the training cards in the training cluster, the third reassignment grouping information includes q groups of third training card numbers, each group of third training card numbers includes n teams of third training card numbers, and the third training card numbers of different teams indicate that the third training card is located in a different data parallel unit; m is an integer greater than 1; q is an integer greater than or equal to 1 and less than s; s is an integer greater than 1.

[0344] The processing module 110 controls the training cluster to continue training the model based on the parallel grouping information, including: establishing a connection between the target third training card and the target tensor parallel unit;

[0345] Wherein, each group of third training card numbers includes the third team training card number, and the target third training card is the third training card indicated by the q third team training card numbers included in the q group training card numbers; the target third training card and the fault card are in the same pipeline parallel unit, and undertake the same training task as the target tensor parallel unit, and the target tensor parallel unit includes a tensor parallel unit that is in a different data parallel unit than the fault card and undertakes the same training task.

[0346] In one possible implementation, the parallel grouping information further includes a reference training card number required for repartitioning the training task, wherein the training card indicated by the reference training card number is in a different data parallel unit and undertakes the same training task as the faulty card; the processing module 110 is further configured to include:

[0347] The model parameters of the current model training in the training card corresponding to the reference training card number are sent to the third training card indicated by the q third training card numbers.

[0348] In one possible implementation, each of the n data parallel units is divided into m pipelined parallel units, and each pipelined parallel unit includes s tensor parallel units; the faulty training card corresponds to at least one tensor parallel unit in a pipelined parallel unit; the parallel grouping information includes the fourth reassignment grouping information of the training cards in the training cluster, the fourth reassignment grouping information includes j groups of fourth training card numbers, each group of fourth training card numbers includes n teams of fourth training card numbers, and the fourth training card numbers of different teams indicate that the fourth training card is located in a different data parallel unit; the fourth training card numbers of each team in the same group of fourth training card numbers indicate that the fourth training card undertakes the same training task; m is an integer greater than 1; s is an integer greater than 1; j is the number of groups after regrouping the training tasks undertaken in the target tensor parallel unit based on the training cards in the pipelined parallel unit where the faulty card is located without faults; j is greater than or equal to s.

[0349] The processing module 110 controls the training cluster to continue training the model based on the parallel grouping information, including: establishing a connection between the fourth training cards of each team indicated by the fourth training card number in each group;

[0350] Each group of fourth training card numbers includes the fourth team training card number. The fourth training card indicated by the j fourth team training card numbers in the j groups of fourth training card numbers is in the same pipeline parallel unit as the faulty card. The j groups of fourth training card numbers include the target group fourth training card number. The fourth training card indicated by the fourth team training card number in the target group fourth training card number undertakes the same training task as the target tensor parallel unit. The target tensor parallel unit includes a tensor parallel unit that is in a different data parallel unit than the faulty card and undertakes the same training task.

[0351] In one possible implementation, the parallel grouping information further includes a reference training card number required for repartitioning the training task, wherein the training card indicated by the reference training card number is in a different data parallel unit and undertakes the same training task as the faulty card; the processing module 110 is further configured to include:

[0352] The model parameters of the current model training in the training card corresponding to the reference training card number are sent to the third training card indicated by the training card number of the fourth team in the same group.

[0353] In one possible implementation, when the model training device 11 is a chip of a control or execution device, the transceiver module 111 can be a communication interface, pins, or circuits. The communication interface can be used to input data to be processed to the processor and can output the processor's processing results. Specifically, the communication interface can be a general purpose input / output (GPIO) interface, which can connect to multiple peripheral devices (such as LCD displays, cameras, radio frequency (RF) modules, antennas, etc.). The communication interface is connected to the processor via a bus.

[0354] Processing module 110 may be a processor, which can execute computer execution instructions stored in the storage module to cause the chip to perform... Figures 3 to 6 The method involved in any of the embodiments shown is further described below. The processor may include a controller, an arithmetic logic unit (ALU), and registers. For example, the controller is primarily responsible for instruction decoding and issuing control signals for the operations corresponding to the instructions. The ALU is primarily responsible for performing fixed-point or floating-point arithmetic operations, shift operations, and logical operations, and can also perform address operations and translations. The registers are primarily responsible for storing register operands and intermediate operation results temporarily stored during instruction execution. In specific implementations, the processor's hardware architecture can be an ASIC architecture, a microprocessor without interlocked piped stages architecture (MIPS), an advanced reduced instruction set machine (RISC) machine (ARM) architecture, or a network processor (NP) architecture, etc. The processor can be single-core or multi-core. The storage module can be an in-chip storage module, such as a register or cache. The storage module can also be an external storage module, such as ROM or other types of static storage devices that can store static information and instructions, RAM, etc.

[0355] It should be noted that the functions of the processor and interface can be implemented through hardware design, software design, or a combination of both; no restrictions are imposed here.

[0356] The embodiments of this application can be applied to providing cloud computing. When users rent servers for model training, they can directly adopt the training method in the embodiments of this application. As long as a complete set of information required for model training is available, training resources can be dynamically allocated during the training process, not limited to a fixed cluster, thereby improving the efficiency of computing power allocation.

[0357] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0358] In summary, the above description is merely an embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the disclosure of this application should be included within the scope of protection of this application.

Claims

1. A model training method, characterized in that, include: The control device confirms the fault card number, which indicates the training card that has failed in the training cluster. The training cluster is divided into n data parallel units for data parallel training; where n is an integer greater than 1. The control terminal device confirms the parallel grouping information based on the fault card number. The parallel grouping information is used to establish new communication links for training cards in the training cluster, and to redistribute training tasks for training cards other than the fault card in the data parallel unit where the fault card is located. The control device sends the parallel packet information to the execution device.

2. The method according to claim 1, characterized in that, The parallel grouping information includes the reference training card number required for repartitioning the training task. The training card and the faulty card indicated by the reference training card number are in different data parallel units and undertake the same training task. The training card indicated by the reference training card number provides model parameters for the subsequent training cluster to continue training.

3. The method according to claim 1 or 2, characterized in that, Each of the n data parallel units is divided into m pipelined parallel units; each pipelined parallel unit includes s tensor parallel units; the parallel grouping information includes the first reassignment grouping information of the training cards in the training cluster, the first reassignment grouping information includes k×s group first training card numbers, each group first training card number includes n team first training card numbers, the first training card number in each team of the same group of first training card numbers indicates that the first training card undertakes the same training task, and the first training card number in different teams indicates that the first training card is in a different data parallel unit; m is an integer greater than 1; k is an integer greater than or equal to 1 and less than m; s is an integer greater than or equal to 1. Wherein, each group of first training card numbers includes the first team training card number, and the first training card indicated by the k×s first team training card numbers in the k×s group of first training card numbers undertakes the same training task as the target pipeline parallel unit. The target pipeline parallel unit includes a pipeline parallel unit that is in a different data parallel unit than the faulty card and undertakes the same training task.

4. The method according to claim 1 or 2, characterized in that, Each of the multiple data parallel units is divided into m pipeline parallel units; each pipeline parallel unit includes s tensor parallel units; the parallel grouping information includes the second reassignment grouping information of the training cards in the training cluster, the second reassignment grouping information includes i×s group second training card numbers; each group second training card number includes n teams of second training card numbers, the second training card numbers of different teams indicate that the second training card belongs to a different data parallel unit; the second training card numbers of each team in the same group of second training card numbers indicate that the second training card undertakes the same training task; m is an integer greater than 1; i is the number of groups after regrouping the number of layers of the training model based on the number of layers of the training model and the training cards in the pipeline parallel units without faults in the data parallel unit where the faulty card belongs; i is greater than or equal to m; s is an integer greater than or equal to 1. Each group of second training card numbers includes a second team training card number. The second training card indicated by the i×s second team training card numbers in the i×s group of second training card numbers is in the same data parallel unit as the faulty card, but not in the pipeline parallel unit where the faulty card is located. The i×s group of second training card numbers includes a target group of second training card numbers. The second training card indicated by the second team training card numbers in the target group of second training card numbers undertakes the same training task as the target pipeline parallel unit. The target pipeline parallel unit includes pipeline parallel units that are in different data parallel units from the faulty card but undertake the same training task.

5. The method according to claim 1 or 2, characterized in that, Each of the n data parallel units is divided into m pipelined parallel units, and each pipelined parallel unit includes s tensor parallel units; the faulty training card corresponds to at least one tensor parallel unit in a pipelined parallel unit; the parallel grouping information includes the third-level allocation grouping information of the training cards in the training cluster, the third-level allocation grouping information includes q groups of third training card numbers, each group of third training card numbers includes n teams of third training card numbers, and the third training card numbers of different teams indicate that the third training card belongs to a different data parallel unit; m is an integer greater than 1; q is an integer greater than or equal to 1 and less than s; s is an integer greater than 1. Each group of third training card numbers includes the third team training card number. The third training cards indicated by the q third team training card numbers in the q groups of third training card numbers are in the same pipeline parallel unit as the faulty card. The third training cards indicated by the q third team training card numbers undertake the same training task as the target tensor parallel unit. The target tensor parallel unit includes tensor parallel units that are in different data parallel units than the faulty card and undertake the same training task.

6. The method according to claim 1 or 2, characterized in that, Each of the n data parallel units is divided into m pipelined parallel units, and each pipelined parallel unit includes s tensor parallel units; the faulty training card corresponds to at least one tensor parallel unit in a pipelined parallel unit; the parallel grouping information includes the fourth reassignment grouping information of the training cards in the training cluster, the fourth reassignment grouping information includes j groups of fourth training card numbers, each group of fourth training card numbers includes n teams of fourth training card numbers, and the fourth training card numbers of different teams indicate that the fourth training card is located in a different data parallel unit; the fourth training card numbers of each team in the same group of fourth training card numbers indicate that the fourth training card undertakes the same training task; m is an integer greater than 1; s is an integer greater than 1; j is the number of groups after regrouping the training tasks undertaken in the target tensor parallel unit based on the training cards in the pipelined parallel unit where the faulty card is located without faults; j is greater than or equal to s. Each group of fourth training card numbers includes the fourth team training card number. The fourth training card indicated by the j fourth team training card numbers in the j groups of fourth training card numbers is in the same pipeline parallel unit as the faulty card. The j groups of fourth training card numbers include the target group fourth training card number. The fourth training card indicated by the fourth team training card number in the target group fourth training card number undertakes the same training task as the target tensor parallel unit. The target tensor parallel unit includes a tensor parallel unit that is in a different data parallel unit than the faulty card and undertakes the same training task.

7. A model training method, characterized in that, include: The execution device sends an abnormal message to the control device; The execution device receives parallel packet information sent by the control device. The parallel packet information is used to establish new communication links for training cards in the training cluster, and to reallocate training tasks for training cards other than the faulty card in the data parallel unit where the faulty card is located. The training cluster performs data parallel training through n data parallel units, where n is an integer greater than 1. The execution device controls the training cluster to continue training the model based on the parallel grouping information.

8. The method according to claim 7, characterized in that, Each of the n data parallel units is divided into m pipelined parallel units; each pipelined parallel unit includes s tensor parallel units; the parallel grouping information includes the first reassignment grouping information of the training cards in the training cluster, the first reassignment grouping information includes k×s group first training card numbers, each group first training card number includes n team first training card numbers, the first training card number in each team of the same group of first training card numbers indicates that the first training card undertakes the same training task, and the first training card number in different teams indicates that the first training card is in a different data parallel unit; m is an integer greater than 1; k is an integer greater than or equal to 1 and less than m. s is an integer greater than or equal to 1; The step of controlling the training cluster to continue training the model based on the parallel grouping information includes: establishing a connection between the target first training card and the target pipeline parallel unit; Wherein, each group of first training card numbers includes the first team training card number, and the target first training card is the first training card indicated by the k×s first team training card numbers included in the k×s group of training card numbers; the target first training card and the target pipeline parallel unit undertake the same training task, and the target pipeline parallel unit includes a pipeline parallel unit that is in a different data parallel unit than the fault card and undertakes the same training task.

9. The method according to claim 8, characterized in that, The parallel grouping information also includes a reference training card number required for repartitioning the training task, wherein the training card indicated by the reference training card number is in a different data parallel unit and undertakes the same training task as the faulty card; the method further includes: The model parameters of the current model training in the training card corresponding to the reference training card number are sent to the first training card indicated by the k×s first team training card numbers.

10. The method according to any one of claims 8-9, characterized in that, The step of controlling the training cluster to continue training the model based on the parallel grouping information also includes: The training sample of the fault card is input into the first training card that undertakes the training task among the k×s first training card numbers indicated by the first training card number of the first team; After completing the gradient calculation, the first training card indicated by the k×s first team training card numbers is synchronized with the training card in the target pipeline parallel unit.

11. The method according to claim 7, characterized in that, Each of the multiple data parallel units is divided into m pipeline parallel units; each pipeline parallel unit includes s tensor parallel units; the parallel grouping information includes the second reassignment grouping information of the training cards in the training cluster, the second reassignment grouping information includes i×s group second training card numbers; each group second training card number includes n teams of second training card numbers, the second training card numbers of different teams indicate that the second training card is in a different data parallel unit; the second training card numbers of each team in the same group of training card numbers indicate that the second training card undertakes the same training task; m is an integer greater than 1; i is the number of groups after regrouping the number of layers of the training model based on the number of layers of the training model and the training cards in the pipeline parallel units without faults in the data parallel unit where the faulty card is located. The i is greater than or equal to m; s is an integer greater than or equal to 1; The step of controlling the training cluster to continue training the model based on the parallel grouping information includes: establishing a connection between the second training cards of each team indicated by the second training card number in each group of second training card numbers; Each group of second training card numbers includes a second team training card number. The second training card indicated by the i×s second team training card numbers in the i×s group of second training card numbers is in the same data parallel unit as the faulty card, but not in the pipeline parallel unit where the faulty card is located. The i×s group of second training card numbers includes a target group of second training card numbers. The second training card indicated by the second team training card numbers in the target group of second training card numbers undertakes the same training task as the target pipeline parallel unit. The target pipeline parallel unit includes pipeline parallel units that are in different data parallel units from the faulty card but undertake the same training task.

12. The method according to claim 11, characterized in that, The parallel grouping information also includes reference training card numbers required for repartitioning the training task, wherein the reference training card numbers indicate training cards that are in different data parallel units than the faulty card but perform the same training task; the method further includes: The model parameters of the current model training in the training card corresponding to the reference training card number are sent to the second training card corresponding to the second training card number in the same group.

13. The method according to any one of claims 11-12, characterized in that, The step of controlling the training cluster to continue training the model based on the parallel grouping information also includes: The training sample of the fault card is input into the first training card in the second training card of the second training card number, which is included in the second training card number of the target group; After completing the gradient calculation, the second training card indicated by the second training card number included in the second training card number of the target group is gradient synchronized with the training card in the target pipeline parallel unit.

14. The method according to claim 7, characterized in that, Each of the n data parallel units is divided into m pipelined parallel units, and each pipelined parallel unit includes s tensor parallel units; the faulty training card corresponds to at least one tensor parallel unit in a pipelined parallel unit; the parallel grouping information includes the third-level allocation grouping information of the training cards in the training cluster, the third-level allocation grouping information includes q groups of third training card numbers, each group of third training card numbers includes n teams of third training card numbers, and the third training card numbers of different teams indicate that the third training card belongs to a different data parallel unit; m is an integer greater than 1; q is an integer greater than or equal to 1 and less than s; s is an integer greater than 1. The step of controlling the training cluster to continue training the model based on the parallel grouping information includes: establishing a connection between the target third training card and the target tensor parallel unit; Wherein, each group of third training card numbers includes the third team training card number, and the target third training card is the third training card indicated by the q third team training card numbers included in the q group training card numbers; the target third training card and the fault card are in the same pipeline parallel unit, and undertake the same training task as the target tensor parallel unit, and the target tensor parallel unit includes a tensor parallel unit that is in a different data parallel unit than the fault card and undertakes the same training task.

15. The method according to claim 14, characterized in that, The parallel grouping information also includes a reference training card number required for repartitioning the training task, wherein the training card indicated by the reference training card number is in a different data parallel unit and undertakes the same training task as the faulty card; the method further includes: The model parameters of the current model training in the training card corresponding to the reference training card number are sent to the third training card indicated by the q third training card numbers.

16. The method according to claim 7, characterized in that, Each of the n data parallel units is divided into m pipeline parallel units, and each pipeline parallel unit includes s tensor parallel units; the faulty training card corresponds to at least one tensor parallel unit in a pipeline parallel unit; the parallel grouping information includes the fourth reassignment grouping information of the training cards in the training cluster, the fourth reassignment grouping information includes j groups of fourth training card numbers, each group of fourth training card numbers includes n teams of fourth training card numbers, and the fourth training card numbers of different teams indicate that the fourth training card is located in a different data parallel unit; the fourth training card numbers of each team in the same group of fourth training card numbers indicate that the fourth training card undertakes the same training task; m is an integer greater than 1; s is an integer greater than 1; j is the number of groups after regrouping the training tasks undertaken in the target tensor parallel unit based on the training cards in the pipeline parallel unit where the faulty card is located that are not faulty. The j is greater than or equal to the s; The step of controlling the training cluster to continue training the model based on the parallel grouping information includes: establishing a connection between the fourth training cards of each team indicated by the fourth training card number in each group; Each group of fourth training card numbers includes the fourth team training card number. The fourth training card indicated by the j fourth team training card numbers in the j groups of fourth training card numbers is in the same pipeline parallel unit as the faulty card. The j groups of fourth training card numbers include the target group fourth training card number. The fourth training card indicated by the fourth team training card number in the target group fourth training card number undertakes the same training task as the target tensor parallel unit. The target tensor parallel unit includes a tensor parallel unit that is in a different data parallel unit than the faulty card and undertakes the same training task.

17. The method according to claim 16, characterized in that, The parallel grouping information also includes a reference training card number required for repartitioning the training task, wherein the training card indicated by the reference training card number is in a different data parallel unit and undertakes the same training task as the faulty card; the method further includes: The model parameters of the current model training in the training card corresponding to the reference training card number are sent to the third training card indicated by the training card number of the fourth team in the same group.

18. A model training device, characterized in that, Includes units or modules for implementing the method as described in any one of claims 1 to 17.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed, cause the computer to perform the method as described in any one of claims 1 to 17.

20. A computer program product, characterized in that, The computer program product includes: computer program code, which, when executed by a computer, causes the computer to perform the method as described in any one of claims 1 to 17.

21. A chip, characterized in that, The chip includes at least one processor and an interface, the processor being configured to read and execute instructions stored in a memory, which, when executed, cause the chip to perform the method as described in any one of claims 1 to 17.

Images