AI model training method, distributed training system and related equipment

Abstract

The invention discloses an AI model training method, a distributed training system and related equipment, and relates to the technical field of artificial intelligence. In the process of training the AI model in the Nth (N is a positive integer) round, the computing node stores state data to a first storage area in the memory, and the state data comprises parameter values after the Nth round of training. In the (N + 1) th round of training process, the calculation node gradually updates parameter values in the AI model, and the updated parameter values are stored in a second storage area in the memory. When a fault occurs in the process, the computing node starts a fault storage mechanism, and the fault storage mechanism comprises persistent storage of the state data in the first storage area. Therefore, in the model training process, the operation of copying the parameter value does not need to be executed in the memory, and the influence of the copying parameter value on the AI model training efficiency is avoided. After the fault of the computing node is recovered, the computing node can recover the AI model and continue to train by using the parameter value stored persistently, so that the overall efficiency of training the AI model can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to an AI model training method, a distributed training system, and related devices. Background Art

[0002] With the development of artificial intelligence (AI) technology, there are diverse ways to train AI models. Specifically, when the parameter scale of an AI model is small, a single computing node can be used to iteratively train the AI model. When the parameter scale of the AI model is large, an AI cluster including multiple computing nodes can be used to jointly train the AI model. For example, for an AI model with more than hundreds of billions of parameters, a dozen or dozens of computing nodes can be used to complete the model training.

[0003] During the process of training an AI model, computing nodes may fail. For example, as the scale of the AI cluster increases, the failure rate of the AI cluster also increases proportionally. Therefore, a checkpoint mechanism is usually adopted to train the AI model and resume the training of the AI model after a failure. As Figure 1 shown, during the process of training the AI model, the parameter values in the currently trained AI model are periodically backed up. In this way, if a failure occurs during the process of training the AI model, the AI model can be continuously trained based on the parameter values backed up most recently (such as Figure 1 the parameter values obtained after the y-th round of training), without having to retrain the AI model from scratch.

[0004] However, the AI model restored based on the backed-up parameter values is usually separated from the AI model at the time of failure by one or more rounds of iterative training processes. Since after the failure recovery, it is necessary to start training again from the AI model that has undergone the y-th round of training, that is, the AI model needs to re-execute the training process between the (y + 1)-th round and the z-th round, and the time consumed by this process is the loss of time in the entire model training process, hereinafter referred to as the training loss time. Therefore, the training loss time generated after the failure recovery will reduce the overall efficiency of model training. Among them, the greater the training loss time, the lower the overall efficiency of model training. Summary of the Invention

[0005] This application provides an AI model training method to reduce the training loss time after failure recovery and improve the overall efficiency of model training. In addition, this application also provides a distributed training system, a computing device, a computer-readable storage medium, and a computer program product.

[0006] In a first aspect, the present application provides an AI model training method, which can be executed by a computing node, hereinafter referred to as the first computing node. Taking the iterative training of the AI model for two rounds as an example, the AI model is hosted on the first computing node. During the Nth (N is a positive integer) round of training the AI model, the first computing node stores the first state data of the AI model in the first storage area in the memory, and the first state data includes the parameter values of the AI model after the Nth round of training. During the (N + 1)th round of training the AI model, the first computing node gradually updates the parameter values in the AI model. For example, if the AI model includes multiple network layers, the AI model can update the parameter values of the next network layer after updating the parameter values of one network layer. Moreover, for the parameter values that have been updated, the first computing node does not use the parameter values to overwrite the parameter values in the first storage area, but stores the updated parameter values in the second storage area in the memory to avoid adjusting the parameter values saved in the first storage area. And when a failure occurs during the (N + 1)th round of training the AI model, such as a software program participating in the AI model training fails, etc., the first computing node starts a failure saving mechanism, and the failure saving mechanism includes persistently storing the first state data in the first storage area in the memory, such as persistently storing the first state data to the local hard disk, or persistently storing it to cloud storage, etc.

[0007] Since the first computing node stores the parameter values obtained from training in different storage areas in the memory during the Nth round and the (N + 1)th round of training the AI model, this enables, when the (N + 1)th round of training the AI model, if a failure occurs during the process of updating the parameter values, the first computing node to use the parameter values saved in the first storage area to restore the AI model to the AI model after the Nth round of training is completed, which can effectively reduce the training loss duration after a failure occurs during the model training process (that is, the training loss duration is less than the duration required for one round of training), thereby improving the overall efficiency of AI model training. When the first computing node belongs to a distributed training system, this AI model training method can also effectively improve the availability of the distributed training system.

[0008] Moreover, the newly generated parameter values during adjacent rounds of training of the AI model can be saved to different storage areas in the memory, which enables the parameter values to be updated uniformly. That is, it can avoid the situation where some parameter values in the AI model are updated while the rest are not (such as not being updated due to a failure), thereby avoiding the impact on the training accuracy of the AI model caused by the non-uniform update of the parameter values in the AI model, and further ensuring that the training accuracy of the AI model can reach a relatively high level. In addition, during the iterative training of the AI model, the parameter values obtained through different rounds of training are stored in different storage areas in the memory, and there is no need to perform the operation of copying parameter values in the memory, which can avoid the impact of copying parameter values on the training efficiency of the first computing node for the AI model.

[0009] In a possible implementation manner, the fault preservation mechanism may further include, after the first computing node completes fault recovery, restoring the AI model after the Nth round of training according to the persistently stored first state data, and continuing to perform the (N + 1)th round of training on the restored AI model. In this way, after the first computing node completes fault recovery, it can continue to execute the training process of the AI model starting from the (N + 1)th round without having to train the AI model from scratch. At the same time, the training loss duration caused by the failure of the first computing node can be less than the duration required for one round of training, thereby effectively improving the training efficiency for the AI model.

[0010] In a possible implementation manner, the first computing node belongs to a distributed training system, and this distributed training system includes multiple computing nodes (it may also include at least one switching node, so that multiple computing nodes can communicate through this switching node). The multiple computing nodes in the distributed training system can be used for jointly training an AI large model. Among them, each computing node can carry a part of the model in the AI model, or each computing node can carry the complete AI large model. The above-mentioned first computing node is one of the computing nodes included in the distributed training system, and the AI model carried on the first computing node is part or all of the AI large model. In this way, during the process of jointly training the AI large model using multiple computing nodes, each computing node can store the parameter values generated during the training process of adjacent rounds of the model in different storage areas in the memory, which can effectively reduce the training loss duration after a failure occurs during the model training process, thereby improving the overall efficiency of AI model training and the availability of the distributed training system.

[0011] Optionally, the first computing node can also independently train the AI model without jointly training the AI model with other computing nodes, etc.

[0012] In a possible implementation, multiple computing nodes in a distributed training system include not only a first computing node but also a second computing node. The second computing node is configured with the same AI model as the first computing node. At this time, the first computing node and the second computing node can train the AI model in the same pace, that is, the second computing node and the first computing node can simultaneously store the same state data in memory. Then, when a failure occurs during the (M + 1)-th (M is a positive integer) round of training the AI model, the first computing node obtains second state data from the second computing node. The second state data includes the parameter values of the AI model after the M-th round of training. Thus, after the first computing node completes the fault recovery, it can restore the AI model after the M-th round of training according to the second state data obtained from the second computing node and continue to perform the (M + 1)-th round of training on the AI model. In this way, in a data parallel distributed training scenario, the first computing node can not only restore the AI model by persistently storing state data but also by obtaining state data from other computing nodes, thereby improving the reliability of restoring the AI model after the first computing node's fault recovery and ensuring the training efficiency of the AI model.

[0013] In a possible implementation, multiple computing nodes in a distributed training system include not only a first computing node but also a third computing node. At this time, the AI model carried on the first computing node is a partial model of the AI large model to be trained, and the AI model carried on the third node is another partial model of the AI large model. That is, the distributed training system can deploy different parts of the AI large model on different computing nodes in a model parallel manner, and each computing node is responsible for training a part of the AI large model, thereby reducing the computing power requirements for a single computing node.

[0014] In a possible implementation, the failure that occurs in the first computing node can specifically be a failure of the software program participating in the training of the AI model on the first computing node, such as a software program running error, etc., resulting in the interruption of the training of the AI model. Or, the failure that occurs in the first computing node can be a communication failure sent by the first computing node. For example, in a data parallel training scenario, when the first computing node updates the parameter values, it needs to interact with other computing nodes for gradients. Thus, when the communication function of the first computing node is abnormal, it will cause the interruption of the training of the AI model.

[0015] In a possible implementation, the first computing node stores the first state data in memory, which may include, in addition to parameter values, momentum, and the momentum is used to update the parameter values of the AI model during the (N + 1)-th round of training the AI model. In practical applications, when the memory space of the first computing node is sufficient, by storing other data in addition to the parameter values of the AI model, the subsequent recovery and training effects for the AI model can be improved.

[0016] In a possible implementation, the first computing node is an acceleration card, and the memory used by the first computing node to store the first state data is HBM (High Bandwidth Memory). In practical applications, the first computing node can also be a CPU or a server, and the memory used to store the first state data can be other types of storage media.

[0017] In a second aspect, the present application provides a distributed training system, which includes multiple computing nodes for jointly training an AI large model. Among them, the multiple computing nodes include a first computing node, and the AI model carried on the first computing node is part or all of the AI large model. The first computing node is configured to: during the N-th round of training the AI model, store the first state data of the AI model in the first storage area in memory, where the first state data includes the parameter values of the AI model after the N-th round of training, and the AI model is carried on the first computing node, and N is a positive integer; during the (N + 1)-th round of training the AI model, gradually update the parameter values in the AI model and store the updated parameter values in the second storage area in memory; when a failure occurs during the (N + 1)-th round of training the AI model, start a failure saving mechanism, and the failure saving mechanism includes persistently storing the first state data in the first storage area in memory.

[0018] In a possible implementation, the failure saving mechanism further includes: after the first computing node completes failure recovery, based on the persistently stored first state data, recover the AI model after the N-th round of training; continue to perform the (N + 1)-th round of training on the AI model.

[0019] In a possible implementation, the multiple computing nodes further include a second computing node, and the second computing node is configured with the same AI model as the first computing node. The first computing node is further configured to: when a failure occurs during the (M + 1)-th round of training the AI model, obtain second state data from the second computing node, where the second state data includes the parameter values of the AI model after the M-th round of training, and M is a positive integer; after the first computing node completes failure recovery, based on the second state data, recover the AI model after the M-th round of training; continue to perform the (M + 1)-th round of training on the AI model.

[0020] In a possible implementation, the multiple computing nodes further include a third computing node. The AI model carried on the first computing node is a partial model in the large AI model, and the AI model carried on the third computing node is another partial model in the large AI model.

[0021] In a possible implementation, the fault is a fault that occurs in the software program participating in the training of the AI model on the first computing node, or the fault is a communication fault that occurs in the first computing node.

[0022] In a possible implementation, the first state data further includes momentum, and the momentum is used to update the parameter values of the AI model during the (N + 1)-th round of training the AI model.

[0023] In a possible implementation, the first computing node is an acceleration card, and the memory is HBM (High Bandwidth Memory).

[0024] The distributed training system provided in the second aspect corresponds to the AI model training method provided in the first aspect. Therefore, for the technical effects of the second aspect and various implementation manners in the second aspect, reference may be made to the technical effects of the corresponding implementation manners in the first aspect, which will not be elaborated herein.

[0025] In a third aspect, the present application provides an acceleration card, which is used to execute the AI model training method described in the first aspect and any one of the implementation manners in the first aspect.

[0026] In a fourth aspect, the present application provides a computing device, which includes a processor and a memory; the memory is used to store instructions, and the processor executes the instructions stored in the memory so that the computing device executes the AI model training method described in the first aspect and any one of the implementation manners in the first aspect.

[0027] In a fifth aspect, the present application provides a computer-readable storage medium, in which instructions are stored. When it runs on a computing device, it causes the computing device to execute the operation steps of the AI model training method described in the first aspect and any one of the implementation manners in the first aspect.

[0028] In a sixth aspect, the present application provides a computer program product containing instructions. When it runs on a computing device, it causes the computing device to execute the operation steps of the AI model training method described in the first aspect and any one of the implementation manners in the first aspect.

[0029] Based on the implementation manners provided in the above aspects of the present application, further combinations can be made to provide more implementation manners. Description of the Drawings

[0030] Figure 1It is a schematic structural diagram of a distributed training system;

[0031] Figure 2 It is a schematic structural diagram of an exemplary distributed training system provided by this application;

[0032] Figure 3 It is a schematic structural diagram of another exemplary distributed training system provided by this application;

[0033] Figure 4 It is a schematic flowchart of an AI model training method provided by this application;

[0034] Figure 5 It is a schematic diagram for reducing the training loss duration;

[0035] Figure 6 It is a schematic flowchart of an AI model training method provided by this application;

[0036] Figure 7 It is a schematic diagram for restoring model training using the state data of persistent storage;

[0037] Figure 8 It is a schematic flowchart of another AI model training method provided by this application;

[0038] Figure 9 It is a schematic diagram for restoring model training using the state data saved in computing node 102;

[0039] Figure 10 It is a schematic hardware structure diagram of a computing device provided by this application. Specific embodiments

[0040] To make the above objects, features, and advantages of this application more obvious and understandable, the following will exemplarily illustrate various non-limiting embodiments in the embodiments of this application with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of this application, rather than all of them. All other embodiments obtained based on the embodiments in this application and based on the above content belong to the scope protected by this application.

[0041] See Figure 2 , it is a schematic structural diagram of an exemplary distributed training system 20. As Figure 2 shown, the distributed training system 20 may include multiple computing nodes. Further, the distributed training system may further include at least one switching node, and the multiple computing nodes may have a communication connection with at least one switching node. For ease of understanding, Figure 2 takes an example including 4 computing nodes (computing node 101 to computing node 104) and 3 switching nodes (switching node 201 to switching node 203) for illustration.

[0042] Among them, the computing node can be a node with model training capabilities. Exemplarily, the computing node can be an acceleration card, such as a deep-learning processing unit (DPU), a data processing unit (DPU), a graphics processing unit (GPU), a neural-network processing unit (NPU), or a tensor processing unit (TPU), etc., or it can be other types of acceleration cards. Alternatively, the computing node can be a general-purpose processor, such as a central processing unit (CPU), etc. Or, the computing node can also include a computing device with a CPU and an acceleration card. The present application does not limit the specific implementation manner of the computing node.

[0043] And each computing node is configured with a memory. For example, as Figure 2 shown, the computing node 101 can include a memory 1012, and can also include a computing unit 1011 for providing computing power and a communication unit 1013 for communicating with other computing nodes. The remaining computing nodes can have the same structure as the computing node 101, which will not be elaborated here. Exemplarily, the memory in the computing node can be, for example, a high bandwidth memory (HBM), or it can be other types of memories, and this is not limited.

[0044] The switching node can be a node with data forwarding capabilities, such as a switch, etc.

[0045] In the distributed training system 20, multiple computing nodes can be used to perform distributed training on the AI large model. Among them, the AI large model refers to a model with a relatively large parameter scale, such as a model with the number of parameters greater than a threshold. Among them, the threshold can be set according to the requirements of the actual application scenario. For example, the threshold can be 100,000, etc. In this embodiment, the parameter scale in the AI large model is not limited. In different application scenarios, based on different thresholds, it can be judged whether a model is an AI large model. Exemplarily, the AI model can specifically be a neural network model, such as a natural language processing model, a generative inference model, etc. In actual application, the AI model can also be other types of models.

[0046] In the first implementation, the number of parameters in the AI large model meets a preset condition. For example, the AI large model can include trillions of parameters. At this time, the distributed training system 20 can adopt the model parallelism method to jointly train the same AI large model using multiple computing nodes. Different computing nodes are used to train different parts of the AI large model. That is, the AI model carried on each computing node can be a part of the AI large model. Taking the AI large model as a neural network model as an example, the network layers included in the neural network model can be divided into multiple groups, each group includes at least one network layer, and each group can be scheduled to a different computing node. Correspondingly, each computing node can train some network layers in the AI large model. In this way, the model training task on each computing node can adapt to the computing power of the computing node, so as to parallelize the training of different parts of the AI large model through multiple computing nodes, thereby improving the training efficiency of the AI large model and reducing the computing power requirements for a single computing node.

[0047] In the second implementation, the number of parameters in the AI large model does not meet the preset condition. For example, the AI large model can include hundreds of thousands of parameters. At this time, the distributed training system 20 can adopt the data parallelism method to parallel train the same AI large model using multiple computing nodes. That is, the same AI large model and different data sets can be configured on each computing node, so that multiple computing nodes can train the same AI large model using different data sets. In each round of training the AI large model, each computing node can obtain the gradients obtained by the other computing nodes during training, aggregate the gradients obtained by the multiple computing nodes respectively, and update the model parameters according to the aggregation result, so as to complete one round of model training.

[0048] In actual application, multiple computing nodes in the distributed training system 20 can also adopt other methods to complete the distributed training of the AI large model. For example, the above first implementation and the second implementation can be combined, etc., which is not limited here.

[0049] Since the distributed training system 20 may fail during the iterative training of the AI large model. For example, the software program participating in the training of the AI large model on some computing nodes has an exception, or the communication connection between some computing nodes and the switching node is interrupted, etc. If the parameter values obtained by each computing node during training are periodically backed up in the distributed training system 20, not only will the periodic backup of parameter values reduce the training efficiency of the distributed training system 20 for the AI large model, but also when the distributed training system 20 fails, the training loss time for restoring the training of the AI large model based on the backed-up parameter values will also reduce the training efficiency of the distributed training system 20 for the AI large model.

[0050] Based on this, in Figure 2 the distributed training system 20 shown in Figure 2 , during the process of training the AI model, each computing node can store the parameter values obtained after different rounds of training in different storage areas in the memory. Among them, the AI model carried on each computing node can be part or all of the AI large model to be trained. Specifically, taking the computing node 101 as an example, during the Nth round of training of the AI model carried on the computing node 101, the parameter values in the AI model obtained by training can be stored in the storage area A in the memory 1012, where N is a positive integer. Then, the computing node 101 continues to perform the (N + 1)th round of training on the AI model. During this process, the computing node 101 can gradually update the parameter values in the AI model. And for the parameter values that have been updated, the computing node 101 does not store the updated parameter values in the storage area A (that is, does not replace the parameter values before the update with the updated parameter values in the storage area A), but stores the updated parameter values in the storage area B in the memory 1012. That is, the different values of the parameters in the AI model before and after the update are respectively saved in different storage areas in the memory 1012. In this way, when a failure occurs during the (N + 1)th round of training of the AI model and the parameter values are gradually updated, the computing node 101 can perform persistent storage on the parameter values saved in the storage area A, so as to use these parameter values to restore the AI model later, that is, restore the AI model to the state after the Nth round of training. In this way, the training loss time caused by failures during the model training process can be effectively reduced. Specifically, the training loss time can be controlled within a range less than the time required for one round of training the AI model, such as being controlled within a few seconds or dozens of seconds, so as to improve the overall efficiency of model training.

[0051] Moreover, reducing the training loss time can also effectively improve the availability of the distributed training system 20. Among them, the availability of the distributed training system 20 can be calculated through the following formula (1).

[0052] Availability = (MTBF - T 训练损失 ) / (MTBF + MTTR) Formula (1)

[0053] Among them, "Availability" refers to the availability of the distributed training system 20; "MTBF" refers to the mean time between failures (MTBF), that is, the average time interval between two adjacent failures in the distributed training system 20; "T 训练损失", refers to the average training loss duration, and its value can be one-half of the maximum value of the training loss duration; "MTTR" refers to the mean time to recovery (MTTR), which is the average time required to recover from a failure in the distributed training system 20. As shown in formula (1), the smaller the T training loss, the greater the Availability. Therefore, by reducing the training loss duration in the distributed training system 20, the availability of the distributed training system 20 can be effectively improved.

[0054] Furthermore, the computing node 101 can alternately store the updated parameter values generated during the iterative training process in storage area A and storage area B in the memory 1012. For example, during the (N + 2)-th round of training of the AI model, the computing node 101 can write the newly generated parameter values into storage area A. That is, after storage area B has successfully stored the parameter values obtained after the (N + 1)-th round of training, it will use the parameter values obtained after the (N + 2)-th round of training to overwrite the parameter values stored in storage area A after the N-th round of training. And so on, until the computing node 101 completes the training process for the AI model.

[0055] In addition, during each round of training of the AI model, the computing node 101 can uniformly update the parameter values in the AI model, that is, it can avoid the situation where in the AI model finally trained by the computing node 101 due to a failure, some parameter values are updated while the rest are not. In actual application scenarios, if in some rounds of the model training process, only some parameter values in the AI model are updated while the rest are not, since the updates of the parameter values in the AI model affect each other, this will cause the unupdated parameter values to have unexpected effects in the subsequent iterative training process, such as possibly causing the AI model to fail to converge or resulting in a low training accuracy of the AI model. Therefore, the computing node 101 recovering the AI model according to the parameter values stored in storage area A can ensure that the parameter values are uniformly updated during each round of training, thereby ensuring that the training accuracy of the AI model can reach a relatively high level.

[0056] Moreover, during the iterative training of the AI model by the computing node 101, it will store the parameter values obtained after different rounds of training in different storage areas in the memory 1012, without performing the operation of copying parameter values in the memory 1012, which can avoid the impact of copying parameter values on the efficiency of the computing node 101 in training the AI model.

[0057] In actual application, during the training process of large AI models with a large number of parameters (such as reaching trillions of parameters), the utilization rate of the memory 1012 in the computing node 101 is usually low, such as less than 50%. Therefore, the existing storage space in the memory 1012 can also support the computing node 101 to store the parameter values obtained from two (or more than two) rounds of training, and there is no additional hardware requirement for the distributed training system 20.

[0058] It can be understood that the above describes the process of training an AI model by the computing node 101. For other computing nodes in the distributed training system 20, a similar method can also be used to train and (recover after a failure) the AI model, which will not be elaborated here.

[0059] As an example, the above Figure 2 shown distributed training system 20 can be applied to a training scenario such as Figure 3 shown.

[0060] Such as Figure 3 shown, the computing nodes in the distributed training system 20 can specifically train the acceleration cards in the training server. For example, the computing node 101 can be the acceleration card 3111 in the training server 31, etc.; the switching node can specifically be a switch. For example, the switching node 201 can be the switch 401, etc.

[0061] Such as Figure 3 shown, in addition to the acceleration card, each training server can also include a CPU. For example, the training server 31 includes a CPU 301, etc. This CPU can manage the acceleration card, such as controlling the acceleration card to start or end the training of the AI model, etc. The CPU in the training server can be configured with memory and a network card. Such as Figure 3 shown, the CPU 301 can be configured with a network card 3012 and memory 3013.

[0062] Among them, the CPU in the training server can receive, through the network card, Figure 3 shown the dataset sent by the data plane network, and provide this dataset to the acceleration card. This dataset is used to train the AI model on the acceleration card, and different acceleration cards can obtain the same or different datasets. And the CPU can also receive, through the network card, Figure 3 shown the control commands sent by the management plane network. For example, it can receive the control commands sent by the scheduling server 33 in the management plane network, etc., so as to instruct the acceleration card to start the training process of the AI model according to this control command. Thus, each acceleration card can use the dataset forwarded by the CPU to perform iterative training on the AI model deployed on this acceleration card.

[0063] Further, if multiple acceleration cards perform parallel training on the same AI model, then during each round of iterative training, each acceleration card can obtain the parameter values obtained by other acceleration cards through the network card and Figure 3 the parameter plane network shown, aggregate the parameter values obtained by multiple acceleration cards, and then send the aggregation result to other acceleration cards through this parameter plane network, and perform gradient synchronization with other acceleration cards, etc., so as to realize the synchronous update of the parameter values in the same AI model by multiple acceleration cards. In actual application, the acceleration cards located within the same training server can communicate through the parameter plane network, or can communicate through the internal bus, and there is no limitation on this.

[0064] Alternatively, when multiple acceleration cards perform parallel training on the same AI model, during each round of iterative training, each acceleration card can send the parameter values obtained by training to the parameter plane network, so that the switch (such as switch 403, etc.) in the parameter plane network can aggregate the parameter values obtained by multiple acceleration cards, and send the aggregation result to each acceleration card again (i.e., in-network computing), so as to realize the synchronous update of the parameter values in the AI model on multiple acceleration cards.

[0065] In actual application, Figure 3 the management plane network, data plane network, and parameter plane network shown can share the same network, or can be implemented through independent networks respectively. Figure 3 In

[0066] it is described by taking the management plane network and data plane network sharing the same network and the parameter plane network using a separate network as an example, and there is no limitation on this. Figure 3 Note that in addition to the above Figure 2 training scenarios shown, Figure 2 the distributed training system 20 shown can also be applied to other training scenarios. For example, in other possible training scenarios,

[0067] the computing nodes in the distributed training system 20 shown can be the CPUs in the training server, so that multiple CPUs perform distributed training on this AI model. Figure 2 In addition, Figure 3 the above combination of

[0068] For ease of understanding, the embodiments of the AI model training method provided in this application will be described below with reference to the accompanying drawings.

[0069] See Figure 4 , Figure 4 which is a schematic flowchart of an AI model training method provided in an embodiment of this application. This method can be applied to Figure 2 or Figure 3 the distributed training system shown, and can also be applied to other applicable distributed training systems. For ease of description, in this embodiment, taking the distributed training system 20 shown in Figure 2 as an example, the process of the computing node 101 iteratively training the AI model is introduced.

[0070] As Figure 4 shown, the model training method may specifically include:

[0071] S401: During the Nth round of training the AI model by the computing node 101, the trained state data is stored in the storage area 1 in the memory 1012. The state data at least includes the parameter values of the AI model after the Nth round of training, where N is a positive integer.

[0072] In this embodiment, multiple computing nodes in the distributed training system 20 can jointly train the same large AI model. At this time, the AI model trained by the computing node 101 can be a complete large AI model (such as when the large AI model has a relatively small number of parameters, etc.). The distributed training system 20 can train the large AI model in a data parallel manner, that is, the same AI model can be deployed on the computing node 101 and the computing node 104. The AI model on each computing node is a complete large AI model, so that different computing nodes can parallelly train the AI model using different data sets. Correspondingly, during each round of training the AI model, each computing node can use the aggregation result to update the parameter values. The aggregation result can be obtained by aggregating the parameter values trained by each computing node, and can be aggregated by the computing node or by the switch in the parameter plane network (i.e., in-network computing). Exemplarily, the parameter plane network can be a remote direct memory access over converged ethernet (RoCE) network on the aggregation ethernet, or an InfiniBand network, etc.

[0073] Alternatively, the AI model trained by computing node 101 can be a partial model in a large AI model (such as the AI model has a large number of parameters, etc.), such as it can be partial network layers in the large AI model. At this time, the distributed training system 20 can adopt the model parallelism method to train the AI model. For example, computing node 101 is used to train a partial model in the large AI model, and computing node 104 is used to train another partial model in the large AI model. Different computing nodes are used to train different parts of the AI model.

[0074] As an implementation example, during the Nth round of training the AI model, computing node 101 can input the input data in training sample 1 into the AI model. The AI model makes inferences based on the input data to obtain corresponding inference results. Then, computing node 101 can compare the difference between the inference result and the label (usually the true result) in training sample 1. Thus, computing node 101 can calculate the gradient used to update the parameter values in the AI model according to the difference between the inference result and the label, and further update the parameter values according to the gradient, thereby completing one round of the training process for the AI model. Among them, when the AI model trained by computing node 101 is a complete model, when computing node 101 updates the parameter values each time, it can first aggregate the gradients calculated by multiple computing nodes respectively to obtain a gradient aggregation result, and then use the gradient aggregation result to update the parameter values.

[0075] Generally, during the process of training the AI model by computing node 101, the generated data will be saved in memory 1012. The generated data can include activation values, parameter values, gradients, momentum, etc. Among them, the activation value refers to the output value generated by the network layer in the AI model based on the input. The parameter value refers to the value of the model parameters during the process of training the AI model, such as weights, etc. During each round of training, the values of the model parameters will be updated; each parameter value can be 16-bit binary floating-point precision (FP16), or can be 32-bit binary floating-point precision (FP32), etc. The gradient refers to the gradient used to update the values of the model parameters, which can indicate the change amplitude of the values of the model parameters in a certain direction. The momentum refers to the value used to adjust the gradient change, which can adjust the gradient value used to update the parameter values currently based on the gradient values used to update the parameter values previously, so as to improve the stability of model training; exemplarily, the momentum in the AI model can include first-order momentum, second-order momentum, etc., which is not limited herein.

[0076] In a possible implementation, the memory 1012 may include three storage areas, namely storage area 1, storage area 2, and storage area 3. When the computing node 101 trains the AI model in each round, it can store the state data in the generated data through storage area 1 and store other data through storage area 2. Among them, the state data refers to the data that can indicate the state of the AI model, or the data that can be used to restore the AI model.

[0077] Exemplarily, the state data may be parameter values, specifically the parameter values in the AI model after completing the Nth round of training. Correspondingly, the other data stored in storage area 2 may include momentum (such as first-order momentum and second-order momentum, etc.), activation values and gradients generated in the Nth round of training, and other data. In other examples, the state data, in addition to including parameter values, may also include other data that can be used to restore the state of the AI model. For example, the state data may include parameter values and momentum. At this time, the data stored in storage area 2 includes activation values and gradients.

[0078] In actual application, when the storage space in the memory 1012 is large enough (such as greater than a threshold), the state data stored by the computing node 101 in storage area 1 may include other data (such as momentum) in addition to parameter values; when the storage space in the memory 1012 is small (such as less than the threshold), the state data stored by the computing node 101 in storage area 1 may only include parameter values. Or, when the storage space in the memory 1012 is large enough, during the model training process, the computing node 101 can generate high-precision parameter values, such as the parameter values with 32-bit floating-point precision. Correspondingly, the computing node 101 can store the high-precision parameter values in storage area 1; when the storage space in the memory 1012 is small, the computing node 101 can generate lower-precision parameter values based on the high-precision parameter values, such as parameter values with 16-bit floating-point precision, etc., and store the lower-precision parameter values in storage area 1 to reduce the memory resources occupied by storing parameter values.

[0079] S402: During the process of the computing node 101 training the AI model in the (N + 1)th round, gradually update the parameter values in the AI model.

[0080] Generally, after the computing node 101 completes the Nth round of model training, it can determine whether the AI model meets the model training termination condition, such as determining whether the number of iterative training times of the AI model reaches a preset number, or determining whether the AI model converges, etc. When the termination condition of the model training is not met, the computing node 101 can continue to perform the next round of training on the AI model, that is, the (N + 1)th round of training.

[0081] Similar to the Nth round of training the AI model, during the (N + 1)th round of training, the computing node 101 can input the input data in the training sample 2 into the AI model. The AI model makes inferences based on the input data, obtains the corresponding inference results, and compares the differences between the inference results and the labels in the training sample 2. Thus, the computing node 101 can calculate the gradients used to update the parameter values in the AI model according to the differences between the inference results and the labels, and further update the parameter values accordingly based on the gradients. In this embodiment, it is assumed that the update of some parameter values has been completed, and the updated model parameters are the first model parameters described in step S402.

[0082] Among them, for multiple parameters in the AI model, the computing node 101 can gradually update the values of the multiple parameters. For example, when the AI model is specifically a neural network model, the AI model can include K network layers, where K is a positive integer greater than 1. Among them, each network layer can include multiple parameters. Then, when the computing node 101 updates the parameter values in the AI model, it can first update the parameter values in the Kth network layer. After updating all the parameter values in the Kth network layer, it then updates the parameter values in the (K - 1)th network layer, and then updates the parameter values in the (K - 2)th layer, and so on, until the parameter values in the first network layer are updated.

[0083] S403: The computing node 101 stores the updated parameter values in the storage area 3 in the memory 1012.

[0084] In this embodiment, during the process of the computing node 101 updating the parameter values in the AI model, instead of replacing the parameter values before the update stored in the storage area 1 with the updated parameter values, the updated parameter values are stored in a new storage area (i.e., the storage area 3) included in the memory 1012, so as to avoid the parameter values obtained through the Nth round of model training being overwritten. In this way, even if a failure occurs during the process of updating the model parameter values in the (N + 1)th round, the parameter values stored in the storage area 1 in the memory 1012 are the values of all the parameters after the AI model has undergone the Nth round of training, and there will not be a situation where some parameter values are the values after the Nth round of model training, while some other parameter values are the updated values generated in the (N + 1)th round of model training.

[0085] For data such as activation values and gradients, during the (N + 1)th round of training, the computing node 101 can save the newly generated activation values, gradients, etc. through the storage area 2, specifically by replacing the old activation values and gradient data in the storage area 2 with the newly generated activation values and gradient data.

[0086] S404: When a failure occurs during the (N + 1)-th round of training the AI model, the computing node 101 activates the failure saving mechanism, which includes persisting the state data in storage area 1 of the persistent storage memory 1012.

[0087] In practical applications, during the process of training the AI model, it is inevitable that failures may occur. For example, the training of the AI model may be interrupted due to a program running error in the computing node 101, or the training of the AI model may be interrupted due to an abnormal communication function of the computing node 101 (in the data parallel scenario), etc. If the computing node 101 fails when continuing to update the remaining part of the parameter values after completing the update of some parameter values during the process of updating the parameter values of the AI model, then the (N + 1)-th round of model training fails. At this time, since the complete model parameter values (which may also include other data such as momentum) generated by the AI model after the N-th round of training are stored in storage area 1, the computing node 101 can activate the failure saving mechanism to persist the state data in storage area 1 of the persistent storage memory 1012, so that the currently trained AI model can be restored to the AI model obtained after the N-th round of training by using the persistently stored state data. In practical applications, after activating the failure saving mechanism, the computing node 101 can also persist other data, such as the activation values, gradients, etc. in storage area 2, or can persist the files of the AI model, etc., which is not limited here.

[0088] In this way, after the computing node 101 completes the failure recovery, it can re-execute the (N + 1)-th round of training on the AI model according to the persistently stored state data. Thus, no matter in which round of model training a failure occurs, the computing node 101 can continue to train the AI model from this round, which enables the training loss duration caused by the failure to be controlled within a range less than the duration required for one round of training the AI model, as Figure 5 shown. In practical applications, the training loss duration can be reduced to within dozens of seconds or even within a few seconds, thereby effectively reducing the training loss duration to improve the overall efficiency of training the AI model and the availability of the distributed training system 20.

[0089] Exemplarily, when a failure occurs, the computing node 101 can persist the state data in storage area 1 to the local hard disk of the computing node 101, or persist the state data in the cloud storage mounted to the computing node 101, etc. Correspondingly, after the computing node 101 recovers from the failure, it can obtain the state data from the local hard disk or the cloud storage, and restore the AI model to the AI model obtained after completing the N-th round of training according to the state data. Thus, the computing node 101 can continue to perform the (N + 1)-th round of training on the AI model.

[0090] Further, when the AI model trained by computing node 101 is the complete model in the large AI model required to be trained by distributed training system 20, since multiple computing nodes (including computing node 101) will train the same AI model in parallel and save the same state data in memory during each round of training the AI model. Therefore, after computing node 101 recovers from a failure, in addition to being able to restore the AI model from the state data persistently stored on the local hard disk or cloud storage, it can also obtain state data by accessing the memory of other computing nodes, and restore the AI model on computing node 101 to the AI model obtained after completing the Nth round of training according to the state data in the memory of other computing nodes, so that computing node 101 can continue to perform the (N + 1)th round of training on this AI model. And, since multiple computing nodes perform synchronous training on the AI model, other computing nodes except computing node 101 can also restore the AI model to the AI model obtained after completing the Nth round of training according to the state data saved in their respective memories, so that multiple computing nodes can continue to perform the (N + 1)th round of training process on the AI model.

[0091] Among them, multiple computing nodes in distributed training system 20 can execute the process of training the AI model under the control of a scheduling server (such as Figure 3 the scheduling server 33 shown, etc.). Specifically, before computing node 101 fails, the scheduling server can send control commands to each computing node respectively to control each computing node to execute the model training process. And, the scheduling server can sense that the AI model fails. For example, computing node 101 can send a failure indication message to the scheduler when it fails, or the scheduling server does not receive a heartbeat message sent by computing node 101 for a long time, etc. Then, after computing node 101 completes the failure recovery, the scheduling server can send a new control command to computing node 101 (and other computing nodes) to instruct computing node 101 (and other computing nodes) to resume the training of the AI model.

[0092] In this embodiment, the two-round training process of the AI model is taken as an example for illustration. During the training process before the Nth round of the AI model and during the training process after the (N + 1)th round, the state data generated in the previous round of training and the model state data generated in the current round can also be saved in different storage areas in memory 1012 in a similar manner as described above, so that when computing node 101 fails, the AI model can be restored to the AI model after the previous round of training is completed.

[0093] Exemplarily, since the storage space of the memory 1012 is usually limited, it is difficult to support the computing node 101 to store the state data generated during each round of training of the AI model in a separate storage area. Therefore, the computing node 101 can alternately store the state data generated during adjacent two rounds of model training through the storage area 1 and the storage area 3. For example, during the Nth round of training, the computing node 101 can store the state data through the storage area 1; during the (N + 1)th round of training, the computing node 101 can store the new state data through the storage area 3; during the (N + 2)th round of training, the computing node 101 uses the storage area 1 to store the new state data again, that is, uses the state data generated during the (N + 2)th round of training to overwrite the state data generated during the Nth round of training.

[0094] Moreover, the timing when the computing node 101 fails is not limited to the above process of updating the second parameter value. For example, when the computing node 101 fails before updating the first parameter value, the computing node 101 can also resume the training of the AI model through the state data stored in the storage area 1.

[0095] The above Figure 4 In the embodiments shown above, the AI model trained by the computing node 101 is a part of a complete large AI model, such as some network layers, etc. Or, the AI model trained by the computing node 101 can be a complete large AI model, that is, different computing nodes use different training samples to train the same large AI model. Next, two training scenarios will be exemplarily described with reference to the accompanying drawings.

[0096] See Figure 6 , which shows a schematic flowchart of a method for training an AI model. In Figure 6 the embodiments shown, the large AI model to be trained in the distributed training system 20 includes 4 AI models, namely AI model 1, AI model 2, AI model 3, and AI model 4, and these 4 AI models are respectively trained by the computing nodes 101 to 104, and each computing node is used to train one AI model, and different AI models can be different parts of the large AI model.

[0097] Taking the computing node 101 training the AI model 1 as an example, as Figure 6 shown, this method can specifically include:

[0098] S601: During the Nth round of training the AI model 1 by the computing node 101, store the trained state data, which includes parameter values, into the storage area 1 in the memory 1012, and store the activation values, gradients, and momenta generated during the model training process into the storage area 2 in the memory 1012.

[0099] S602: During the (N + 1)-th round of training the AI model 1, the computing node 101 stores the newly generated activation values, gradients, and momentum in the storage area 2.

[0100] S603: The computing node 101 gradually updates the parameter values in the AI model 1.

[0101] S604: The computing node 101 stores the updated parameter values in the storage area 3.

[0102] Among them, for the specific implementation processes of steps S601 to S604, reference can be made to the relevant descriptions of steps S401 to S403 in the above Figure 4 illustrated embodiments, which will not be elaborated here. Moreover, during the (N + 1)-th round of training the AI model 1, the computing node 101 can perform the operations of storing gradients in the storage area 2 and storing the updated parameter values in the storage area 3 in parallel.

[0103] S605: When a failure occurs during the (N + 1)-th round of training the AI model 1, the computing node 101 persists the state data in the storage area 1.

[0104] In this embodiment, when a failure occurs during the process of gradually updating the parameter values in the AI model, the computing node 101 can initiate a failure saving mechanism, which can include persisting the state data in the storage area 1, and can also include, after the computing node 101 completes failure recovery, continuing to perform iterative training on the AI model that has undergone the N-th round of training.

[0105] Exemplarily, the failure that occurs to the computing node 101 can be, for example, a failure of the program code running on the computing node 101 for training the AI model, etc.

[0106] As Figure 7 shown, the computing node 101 can back up the state data in the storage area 1 to a persistent storage medium, and this state data can be used as a checkpoint when resuming model training.

[0107] S606: After the computing node 101 completes failure recovery, the computing node 101 can restore the AI model 1 to the AI model 1 after completing the N-th round of training according to the persistently stored state data.

[0108] Exemplarily, the computing node 101 can complete fault recovery by executing a preset fault handling strategy. For example, the computing node 101 can complete fault recovery by restarting the computing node 101. Or, the computing node 101 can also complete fault recovery under the operation and maintenance of the operation and maintenance personnel. For example, the operation and maintenance personnel can perform software upgrade on the computing node 101, so that the computing node 101 can restore the normal model training function by running the upgraded software.

[0109] As Figure 7 shown, the computing node 101 can read the state data saved in the persistent storage medium into the memory 1012. For example, it can be to read the state data into the storage area 1, so that the computing node 101 can use the state data in the storage area 1 to restore the AI model 1 to the AI model 1 obtained when completing the Nth round of training. And when continuing to perform the (N + 1)th round of training on the AI model 1, when updating the parameters in the AI model 1, the momentum in the state data can be used to calculate the gradient for updating the parameter, so as to update the parameter value according to the calculated gradient.

[0110] Furthermore, when continuing to perform the (N + 1)th round of training on the AI model 1, the newly generated state data (including the newly generated parameter values and new momentum) can be saved to the storage area 3 in the memory 1012.

[0111] S607: The computing node 101 continues to train the AI model 1 until the training termination condition is met.

[0112] Exemplarily, the training termination condition can be, for example, that the number of iterative training times of the AI model 1 meets the preset number, or the AI model 1 is in a converged state, etc., and this is not limited.

[0113] Similarly, for the process of the computing nodes 102 to 104 training the AI model respectively, reference can be made to the relevant descriptions of the computing node 101 training the AI model 1 above, and details are not described here.

[0114] See Figure 8 , which shows a schematic flowchart of a method for training an AI model. In the Figure 8 shown embodiment, the computing nodes 101 to 104 use different training samples to train the same AI model (that is, the large AI model required to be trained by the distributed training system 20). And in each round of model training process, the computing node 101 will aggregate the parameter values obtained by multiple computing nodes during training, and feedback the aggregated result to the remaining computing nodes, so that each computing node can use the aggregated set to update the parameters of the AI model.

[0115] Taking the computing node 101 training the AI model as an example, asFigure 8 As shown in the figure, the method may specifically include:

[0116] S801: During the M-th round of training the AI model by computing node 101, the trained state data is stored in storage area 1 of memory 1012. The state data includes the parameter values in the AI model, and the activation values, gradients, and momenta generated during the model training process are stored in storage area 2 of memory 1012, where M is a positive integer.

[0117] S802: During the (M + 1)-th round of training the AI model by computing node 101, the newly generated activation values, gradients, and momenta are stored in storage area 2.

[0118] S803: Computing node 101 gradually updates the parameter values in the AI model.

[0119] S804: Computing node 101 stores the updated parameter values in storage area 3.

[0120] Among them, since multiple computing nodes 101 train the same AI model in parallel, when computing node 101 updates the first parameter value each time, it will first complete gradient synchronization with other computing nodes, that is, synchronize the gradient used to update the first parameter value to each of the remaining computing nodes. After completing the gradient synchronization, each computing node (including computing node 101) uses this gradient to update the first parameter value, as Figure 9 shown.

[0121] S805: When a failure occurs during the (M + 1)-th round of training the AI model, computing node 101 stops training the AI model.

[0122] Normally, since the training pace of the remaining computing nodes for the AI model is the same as that of computing node 101 for the AI model, when computing node 101 fails and stops training the AI model, each of the remaining computing nodes can also pause the training process for the AI model, so that the model training process of distributed training system 20 pauses, as Figure 9 shown.

[0123] S806: Computing node 101 determines whether a hardware failure has occurred. If so, it continues to execute step S807; if not, it continues to execute step S809.

[0124] S807: After computing node 101 completes the fault recovery, it obtains the state data of the AI model from the memory of computing node 102, and the state data is the same as the state data written by computing node 101 to storage area 1 when it completed the M-th round of training.

[0125] Among them, a hardware failure can be, for example, a hardware failure on computing node 101, such as physical damage to the processor, memory, or network card in computing node 101. At this time, the operation and maintenance personnel can achieve fault recovery of computing node 101 by restarting the faulty hardware on computing node 101 or replacing the faulty hardware. Since multiple computing nodes train the AI model in unison, the state data stored in the memory of multiple computing nodes is usually the same. This enables computing node 101 to obtain the state data from the memory of other computing nodes even if the state data in storage area 1 is lost due to a failure of computing node 101.

[0126] S808: Computing node 101 restores the AI model to the AI model after completing the M-th round of training according to the state data. Then, computing node 101 continues to execute step S811.

[0127] Among them, since the parameter values are updated based on the gradient synchronization method between different computing nodes, when computing node 101 restores the AI model to the AI model after completing the M-th round of training, the remaining computing nodes 101 can also restore the AI models they trained to the AI models after completing the M-th round of training according to the state data stored in the memory, as Figure 9 shown, so that each computing node can continue to train the AI model in parallel based on the same pace.

[0128] S809: Computing node 101 persistently stores the state data stored in storage area 1.

[0129] S810: After computing node 101 completes fault recovery, computing node 101 can restore the AI model to the AI model after completing the M-th round of training according to the persistently stored state data.

[0130] S811: Computing node 101 continues to train the AI model until the training termination condition is met.

[0131] In this embodiment, it is described by taking the case of continuing to train the AI model on computing node 101 after computing node 101 fails and after completing fault recovery. In other embodiments, when a hardware failure occurs in computing node 101, the computing node can also be replaced. For example, the operation and maintenance personnel can deploy a new computing node in the distributed training system 20 and resume the training of the AI model on the replaced computing node.

[0132] It should be noted that other reasonable combinations of steps that can be conceived by those skilled in the art based on the above description also fall within the protection scope of this application. Secondly, those skilled in the art should also be familiar that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential for this application.

[0133] The above combination with Figures 1 to 9 introduces the distributed training system and the AI model training method provided by the embodiments of this application. Next, the structure of the computing device for implementing the computing node 101 provided by the embodiments of this application will be introduced with reference to the accompanying drawings.

[0134] See Figure 10 , which shows a schematic hardware structure diagram of a computing device 1000. The computing device 1000 can, for example, implement the computing node 101 and the like in the above Figure 2 shown embodiments.

[0135] As Figure 10 shown, the computing device 1000 includes a processor 1001, a memory 1002, and a communication interface 1003. Among them, the processor 1001, the memory 1002, and the communication interface 1003 communicate through a bus 1004, and can also communicate through other means such as wireless transmission. The memory 1002 is used to store instructions, and the processor 1001 is used to execute the instructions stored in the memory 1002. Further, the computing device 1000 may further include a memory unit 1005. The memory unit 1005 can be used to cache program codes, or can be used to cache data, such as the above state data. The memory unit 1005 can be connected to the processor 1001, the storage medium 1002, and the communication interface 1003 through the bus 1004. Among them, the memory 1002 stores program codes, and the processor 1001 can call the program codes stored in the memory 1002 to perform the following operations:

[0136] During the Nth round of training the AI model, the first computing node stores the first state data of the AI model in the first storage area in the memory. The first state data includes the parameter values of the AI model after the Nth round of training. The AI model is carried on the first computing node, and N is a positive integer;

[0137] During the (N + 1)th round of training the AI model, the first computing node gradually updates the parameter values in the AI model and stores the updated parameter values in the second storage area in the memory;

[0138] When a failure occurs during the training of the AI model in the (N + 1)-th round, the first computing node activates a failure preservation mechanism, which includes persistently storing the first state data in the first storage area of the memory.

[0139] It should be understood that in this embodiment, the processor 1001 may be a CPU, and the processor 1001 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0140] The memory 1002 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1001. The memory 1002 may also include a non-volatile random access memory.

[0141] The memory 1002 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM).

[0142] The communication interface 1003 is used to communicate with other devices connected to the computing device 1000. In addition to the data bus, the bus 1004 may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, all kinds of buses are labeled as bus 1004 in the figure.

[0143] It should be understood that the computing device 1000 according to the embodiment of the present application may correspond to the method executed by the computing node 101 in the method shown in Figure 4 , Figure 6 or Figure 8 The above and other operations and / or functions implemented by the computing device 1000 are respectively for implementing the processes of the corresponding methods in Figure 4 , Figure 6 or Figure 8 . For the sake of brevity, they will not be described in detail here.

[0144] The present application provides an acceleration card, which includes a power supply circuit and a processing circuit. The power supply circuit is used to supply power to the processing circuit, and the processing circuit is used to execute Figure 4 , Figure 6 or Figure 8 the method executed by the computing node 101 in the method shown.

[0145] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state drive), etc. The computer-readable storage medium includes instructions, and the instructions instruct the computing device to execute the above AI model training method.

[0146] The embodiment of the present application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on the computing device, the processes or functions according to the embodiments of the present application are fully or partially generated.

[0147] The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.).

[0148] The computer program product may be a software installation package. In the case of any method that requires the use of the foregoing model training method, the computer program product can be downloaded and executed on a computing device.

[0149] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that contains one or more collections of available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0150] The terms used in the above embodiments are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and claims of the present application, the singular forms "a", "an", "the", "above", "said", "this", and "one" are also intended to include the form "one or more", unless the context clearly indicates otherwise. It should also be understood that in the embodiments of the present application, "one or more" means one, two, or more than two; the character " / " generally indicates that the related objects before and after are in an "or" relationship. In the embodiments of the present application, "simultaneously" means within the same time period, including the case of being at the same moment. The terms "first", "second", etc. in the specification, claims, and drawings of the present application are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances, which is only a way of distinguishing objects with the same attributes when describing the embodiments of the present application.

[0151] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but rather mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized.

[0152] As described above, the above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for training an artificial intelligence (AI) model, characterized in that, The method includes: During the Nth round of training the AI model, the first computing node stores the first state data of the AI model in the first storage area in the memory. The first state data includes the parameter values of the AI model after the Nth round of training. The AI model is hosted on the first computing node, and N is a positive integer. During the (N + 1)th round of training the AI model, the first computing node gradually updates the parameter values in the AI model and stores the updated parameter values in the second storage area in the memory. When a failure occurs during the (N + 1)th round of training the AI model, the first computing node activates a failure preservation mechanism. The failure preservation mechanism includes persistently storing the first state data in the first storage area in the memory.

2. The method according to claim 1, wherein The failure preservation mechanism further includes: After the first computing node completes failure recovery, based on the persistently stored first state data, the AI model after the Nth round of training is restored. Continue to perform the (N + 1)th round of training on the AI model.

3. The method according to claim 1 or 2, characterized in that, The first computing node belongs to a distributed training system. The distributed training system includes multiple computing nodes for jointly training an AI large model. The multiple computing nodes include the first computing node. Among them, the AI model is part or all of the AI large model.

4. The method according to claim 3, wherein The multiple computing nodes further include a second computing node, and the second computing node is configured with the same AI model as the first computing node. The method further includes: When a failure occurs during the (M + 1)th round of training the AI model, the first computing node obtains second state data from the second computing node. The second state data includes the parameter values of the AI model after the Mth round of training, and M is a positive integer. After the first computing node completes failure recovery, the first computing node restores the AI model after the Mth round of training according to the second state data. The first computing node continues to perform the (M + 1)th round of training on the AI model.

5. The method according to claim 3, characterized in that The multiple computing nodes further include a third computing node. The AI model hosted on the first computing node is part of the AI large model, and the AI model hosted on the third computing node is another part of the AI large model.

6. The method according to any one of claims 3 to 5, characterized in that The failure is a failure of the software program participating in training the AI model on the first computing node, or the failure is a communication failure of the first computing node.

7. The method according to any one of claims 1 to 6, characterized in that The first state data further includes momentum, and the momentum is used to update the parameter values of the AI model during the (N + 1)th round of training the AI model.

8. The method according to any one of claims 1 to 6, characterized in that, The first computing node is an acceleration card, and the memory is a high-bandwidth memory HBM.

9. A distributed training system, characterized in that, The distributed training system includes multiple computing nodes for jointly training an AI large model. The multiple computing nodes include a first computing node, and the AI model hosted on the first computing node is part or all of the AI large model. The first computing node is configured to: During the Nth round of training the AI model, store the first state data of the AI model in a first storage area in the memory, where the first state data includes the parameter values of the AI model after the Nth round of training, the AI model is hosted on the first computing node, and N is a positive integer; During the (N + 1)th round of training the AI model, gradually update the parameter values in the AI model and store the updated parameter values in a second storage area in the memory; When a failure occurs during the (N + 1)th round of training the AI model, start a failure saving mechanism, where the failure saving mechanism includes persistently storing the first state data in the first storage area in the memory.

10. The distributed training system according to claim 9, wherein The failure saving mechanism further includes: After the first computing node completes failure recovery, restore the AI model after the Nth round of training according to the persistently stored first state data; Continue to perform the (N + 1)th round of training on the AI model.

11. The distributed training system according to claim 9, wherein The multiple computing nodes further include a second computing node, and the second computing node is configured with the same AI model as the first computing node; The first computing node is further configured to: When a failure occurs during the (M + 1)th round of training the AI model, obtain second state data from the second computing node, where the second state data includes the parameter values of the AI model after the Mth round of training, and M is a positive integer; After the first computing node completes failure recovery, restore the AI model after the Mth round of training according to the second state data; Continue to perform the (M + 1)th round of training on the AI model.

12. The distributed training system according to claim 9, wherein The multiple computing nodes further include a third computing node, the AI model hosted on the first computing node is a partial model of the AI large model, and the AI model hosted on the third computing node is another partial model of the AI large model.

13. The distributed training system according to any one of claims 9 to 12, characterized in that, The failure is a failure of the software program participating in training the AI model on the first computing node, or the failure is a communication failure of the first computing node.

14. The distributed training system according to any one of claims 9 to 13, characterized in that, The first state data further includes momentum, and the momentum is used to update the parameter values of the AI model during the (N + 1)th round of training the AI model.

15. The distributed training system according to any one of claims 9 to 14, characterized in that, The first computing node is an acceleration card, and the memory is a high-bandwidth memory HBM.

16. An acceleration card, characterized in that, The acceleration card is used to execute the method according to any one of claims 1 to 8.

17. A computing device, characterized in that, The computing device includes a processor and a memory; The memory is used to store instructions, and the processor executes the instructions stored in the memory so that the computing device executes the method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, Including instructions, when running on a computing device, cause the computing device to execute the method according to any one of claims 1 to 8.

19. A computer program product comprising instructions, characterized in that, When running on a computing device, cause the computing device to execute the method according to any one of claims 1 to 8.

Images