Model training method and apparatus, system, and related device

Abstract

A model training method and apparatus, a system, and a related device, relating to the technical field of artificial intelligence. A distributed training system comprises multiple computing nodes used for performing distributed training on an AI (artificial intelligence) model. In a forward computation process, data is transmitted among the multiple computing nodes by means of executing a forward communication operator in the AI model. The computing nodes in the distributed training system acquire a re-computation layer of the AI model in a reverse computation process, and acquire a forward communication operator in the re-computation layer; then, the computing nodes remove the forward communication operator from the re-computation layer, and use the re-computation layer with the forward communication operator removed to perform reverse computation. Thus, in a reverse computation stage, a computing node performs re-computation with respect to a computation layer with a forward communication operator removed, such that activation values required for updating parameters in the computation layer can be obtained without having to execute the forward communication operator, thereby improving the overall efficiency of distributed training performed on the AI model.

Description

Model training method, device, system and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on February 28, 2024, with application number 202410223899.7 and application name “Model training methods, devices, systems and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, system and related equipment. Background Art

[0003] With the development of artificial intelligence (AI) technology, the parameter scale of AI models is getting larger and larger. For this reason, multiple computing nodes can usually be used to perform distributed training on the AI ​​model to improve the efficiency of the AI ​​model and reduce the computing power requirements for a single computing node. Among them, a round of model training process usually includes a forward calculation phase and a reverse calculation phase. The forward calculation phase refers to the process of performing calculations in sequence from the first computing layer to the last computing layer in the AI ​​model (also known as the inference phase); the reverse calculation phase refers to the process of updating model parameters in sequence from the last computing layer to the first computing layer.

[0004] In actual applications, the storage resources of a single computing node are limited, and it may be difficult to store all the activation values ​​generated in the forward calculation phase during model training. Typically, in the forward calculation phase, the computing node may not save the activation values ​​generated by the computing layer in the AI ​​model, or only save the activation values ​​generated by part of the computing layer to reduce the use of storage resources; accordingly, in the reverse calculation phase, the computing node can recalculate (i.e., recalculate) the computing layer to obtain the activation value generated by the computing layer in the forward phase, so as to use the activation value to update the parameters in the computing layer.

[0005] However, this method of training AI models often has the problem of low model training efficiency.

[0006] Summary of the Invention

[0007] This application provides a model training method to improve the overall efficiency of distributed training for AI models. In addition, this application also provides a model training device, system, computing device, computer-readable storage medium, and computer program product.

[0008] In a first aspect, the present application provides a model training method, which is applied to a distributed training system, wherein the distributed training system includes multiple computing nodes, and the multiple computing nodes are used to perform distributed training on an AI (artificial intelligence) model. The AI ​​model includes a forward communication operator, and in the forward calculation process, data is transmitted between the multiple computing nodes by executing the forward communication operator. The transmitted data may be data obtained based on activation values ​​calculated by a network layer in each computing node, etc. Specifically, the computing nodes in the distributed training system obtain the recalculation layer of the AI ​​model in the reverse calculation process. The recalculation layer may be, for example, a computing layer that does not save activation values ​​in the forward calculation process and needs to perform parameter updates based on activation values ​​in the reverse calculation process, and the computing nodes obtain the forward communication operator in the recalculation layer; then, the computing nodes remove the forward communication operator from the recalculation layer, and use the recalculation layer without the forward communication operator to perform reverse calculation.

[0009] In this way, in the reverse calculation phase, the computing node recalculates the computing layer without the forward communication operator, which can not only obtain the activation value required to update the parameters in the computing layer, but also does not need to execute the forward communication operator in the computing layer (that is, the recalculation layer). This can avoid the communication delay caused by the computing node executing unnecessary forward communication operators and affect the efficiency of updating the model parameters, thereby improving the overall efficiency of distributed training for AI models.

[0010] In one possible implementation, the forward communication operator removed is the terminal forward communication operator in the recalculation layer. The terminal forward communication operator refers to the terminal forward communication operator in the recalculation layer. Thus, after the forward communication operator is removed from the recalculation layer, when the recalculation layer is executed, not only can the required activation value be obtained, but unnecessary forward communication operators can also be avoided, thereby improving the training efficiency of the AI ​​model.

[0011] In one possible implementation, the forward communication operators removed are all forward communication operators in the recalculation layer. That is, when the recalculation layer includes multiple forward communication operators, all forward communication operators in the recalculation layer can be removed. In this way, during the reverse calculation process, not only is it unnecessary to execute the terminal forward communication operator, but other forward communication operators located at non-terminal positions are also unnecessary, thereby further improving the training efficiency of the AI ​​model.

[0012] In one possible embodiment, the recalculation layer includes multiple forward communication operators. In this case, when the computing node removes the forward communication operators in the recalculation layer, it can be specifically divided into multiple sub-recalculation layers, wherein each of the multiple sub-recalculation layers includes a forward communication operator. Generally, in the divided sub-recalculation layers, the forward communication operator is located at the end of the sub-recalculation layer; then, the computing node removes the forward communication operator included in each of the multiple sub-recalculation layers; then, when the computing node uses the recalculation layer without the forward communication operator to perform reverse calculation, it can be specifically performed using each of the multiple sub-recalculation layers without the forward communication operator to perform reverse calculation. In this way, by dividing the recalculation layer including multiple forward communication operators into multiple sub-recalculation layers and removing the forward communication operators in each sub-recalculation layer one by one, it is possible to remove all forward communication operators in the recalculation layer, so that no forward communication operator in the recalculation layer needs to be executed during the reverse calculation process, thereby effectively improving the training efficiency of the AI ​​model.

[0013] In one possible implementation, when the recalculation layer is divided into multiple sub-recalculation layers during the reverse calculation phase, during the forward calculation phase, the computing node saves the input data of each of the multiple sub-recalculation layers, and the saved input data is used for reverse calculation of the sub-recalculation layer. In this way, during the reverse calculation phase, the computing node can use this saved input data to recalculate each sub-recalculation layer to obtain the activation value generated by the sub-recalculation layer, thereby avoiding recalculation failures that may lead to interruption of model training.

[0014] In one possible implementation, for multiple computing nodes in a distributed training system, taking the first computing node performing reverse computing using a recomputation layer that removes forward communication operators as an example, before dividing the computing layer into multiple sub-recomputation layers, the first computing node can also estimate whether the number of storage resources of the first computing node meets the recomputation layer division condition. The recomputation layer division condition can be, for example, whether the number of storage resources of the first computing node can be greater than a threshold, etc. Thus, when the first computing node divides the recomputation layer into multiple sub-recomputation layers, it can specifically be when the number of storage resources of the first computing node meets the recomputation layer division condition that the recomputation layer is divided into multiple sub-recomputation layers. In this way, the first computing node can determine whether to divide the recomputation layer including multiple forward communication operators based on the number of its own storage resources, so as to avoid the number of storage resources exceeding the number of storage resources of the first computing node after dividing the recomputation layer, thereby affecting the first computing node's training of the AI ​​model.

[0015] In one possible implementation, before performing reverse calculation using the recalculation layer with the forward communication operator removed, the computing node can first determine the reverse communication operator corresponding to the forward communication operator based on the mapping relationship and the removed forward communication operator. Thus, during the reverse calculation process, the computing node will execute the recalculation layer with the forward communication operator removed to obtain an activation value, and will execute the reverse communication operator to obtain a gradient for updating the parameters in the recalculation layer. Thus, the computing node can update the parameters in the recalculation layer based on the activation value and the gradient, that is, update the parameters in the calculation layer corresponding to the recalculation layer. In this way, after removing the forward communication operator, the reverse communication operator can be inserted to ensure that the computing node can normally obtain the gradient, so as to successfully update the parameters in the calculation layer.

[0016] In one possible implementation, the computing node executes the reverse communication operator while executing the recalculation layer that removes the forward communication operator. In this way, the process of executing the reverse communication operator can be hidden in the process of the computing node executing the recalculation layer. This allows the computing node to directly obtain the gradient obtained based on the activation value and the execution of the reverse communication operator after recalculating the activation value, and calculate the gradient used to update the parameters in the recalculation layer without waiting for the computing node to execute the reverse communication operator. This can improve the training efficiency of the AI ​​model.

[0017] In one possible implementation, the determined recalculation layer includes a calculation module, a forward communication operator, and a post-processing module, wherein the execution order of the forward communication operator in the recalculation layer is before the execution order of the post-processing module and after the execution order of the calculation module. Then, in the reverse calculation process, after removing the forward communication operator of the recalculation layer, the computing node can also fill the output data of the calculation module according to the second matrix dimension when the first matrix dimension of the output data of the calculation module does not match the second matrix dimension of the input data of the post-processing module. In this way, by filling the output data of the calculation module, it is possible to avoid the mismatch between the matrix dimension of the activation value output by the calculation module and the matrix dimension of the input data of the post-processing module, which leads to the interruption of model training, thereby ensuring that the training of the AI ​​model can be executed normally after the forward communication operator is removed.

[0018] In the second aspect, the present application provides a model training device, which is applied to a distributed training system. The distributed training system includes multiple computing nodes, and the multiple computing nodes are used to perform distributed training on an artificial intelligence AI model. The AI ​​model includes a forward communication operator. During the forward calculation process, data is transmitted between multiple computing nodes by executing the forward communication operator; the model training device includes: an acquisition unit, used to obtain the recalculation layer of the AI ​​model during the reverse calculation process, and obtain the forward communication operator in the recalculation layer; a removal unit, used to remove the forward communication operator from the recalculation layer; and a reverse calculation unit, used to perform reverse calculation using the recalculation layer without the forward communication operator.

[0019] In a possible implementation, the forward communication operator is a terminal forward communication operator in the recomputation layer.

[0020] In a possible implementation, the forward communication operators are all forward communication operators in the recomputation layer.

[0021] In one possible embodiment, the recalculation layer includes multiple forward communication operators; a removal unit is specifically used to divide the recalculation layer into multiple sub-recalculation layers, each of the multiple sub-recalculation layers includes a forward communication operator; the forward communication operator included in each sub-recalculation layer in the multiple sub-recalculation layers is removed; and a reverse calculation unit is specifically used to perform reverse calculation using each sub-recalculation layer in the multiple sub-recalculation layers with the forward communication operator removed.

[0022] In a possible embodiment, the model training device also includes: a storage unit, which is used to save the input data of each sub-computation layer in the multiple sub-computation layers during the forward calculation process, and the input data is used for the sub-computation layer to perform reverse calculation.

[0023] In one possible embodiment, the model training device is applied to a first computing node among multiple computing nodes, and the first computing node performs reverse calculation using a recomputation layer that removes a forward communication operator; then, the model training device also includes: an estimation unit, which is used to estimate whether the number of storage resources of the first computing node meets the recomputation layer division condition before dividing the recomputation layer into multiple sub-recomputation layers; when the removal unit divides the recomputation layer into multiple sub-recomputation layers, it is specifically used to divide the recomputation layer into multiple sub-recomputation layers when the number of storage resources of the first computing node meets the recomputation layer division condition.

[0024] In one possible embodiment, the model training device also includes: a determination unit, used to determine the reverse communication operator corresponding to the forward communication operator based on the mapping relationship and the forward communication operator; a reverse calculation unit, specifically used to: execute the recalculation layer that removes the forward communication operator to obtain an activation value; execute the reverse communication operator to obtain a gradient for updating the parameters in the recalculation layer; and update the parameters in the recalculation layer based on the activation value and the gradient.

[0025] In a possible implementation, the reverse calculation unit is specifically configured to execute the reverse communication operator during a process of executing a recalculation layer that removes the forward communication operator.

[0026] In one possible embodiment, the recalculation layer includes a calculation module, a forward communication operator and a post-processing module, and the execution order of the forward communication operator in the recalculation layer is before the execution order of the post-processing module and after the execution order of the calculation module; the model training device also includes: a filling unit, which is used to fill the output data of the calculation module according to the second matrix dimension when the first matrix dimension of the output data of the calculation module does not match the second matrix dimension of the input data of the post-processing module.

[0027] In a third aspect, the present application provides a distributed training system, which includes multiple computing nodes, and the multiple computing nodes are used to perform distributed training on an artificial intelligence AI model. The AI ​​model includes a forward communication operator. During the forward computing process, data is transmitted between multiple computing nodes by executing the forward communication operator. Each of the multiple computing nodes is used to execute the model training method in the first aspect or any implementation of the first aspect.

[0028] In a fourth aspect, the present application provides a computing device comprising a processor and a memory. The processor and the memory communicate with each other. The processor is used to execute instructions stored in the memory so that the computing device performs the model training method as in the first aspect or any one of the implementations of the first aspect. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The computing device may further include a bus. The processor is connected to the memory via the bus. The memory may include a readable memory and a random access memory.

[0029] In a fifth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computing device, the computing device executes the operating steps of the model training method described in the first aspect or any implementation of the first aspect.

[0030] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device, enables the computing device to execute the operational steps of the model training method described in the first aspect or any one of the implementations of the first aspect.

[0031] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG1 is a schematic diagram of the structure of an exemplary distributed training system provided by the present application;

[0033] FIG2 is a flow chart of a model training method provided in this application;

[0034] Figure 3 is a schematic diagram showing that the forward communication operator is located between the calculation module and the post-processing module;

[0035] FIG4 is a schematic diagram showing that the computing layer includes multiple computing modules and multiple forward communication operators;

[0036] FIG5 is a flow chart of another model training method provided in this application;

[0037] FIG6 is a schematic diagram of dividing a computing layer into multiple sub-computing layers provided by the present application;

[0038] FIG7 is a flow chart of another model training method provided by the present application;

[0039] FIG8 is a schematic diagram of the computing node 101 provided in the present application successively performing recalculation and reverse communication operators;

[0040] FIG9 is a schematic diagram of the computing node 101 provided in the present application executing the recalculation process and the reverse communication operator in parallel;

[0041] FIG10 is a schematic structural diagram of a model training device provided by the present application;

[0042] FIG11 is a schematic diagram of the hardware structure of a computing device provided in this application. DETAILED DESCRIPTION

[0043] The terms "first," "second," and so on, in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate and are merely used to describe the manner in which objects with the same attributes are distinguished in the embodiments of this application.

[0044] The technical solution in this application will be described below in conjunction with the drawings provided in this application.

[0045] FIG1 is a schematic diagram illustrating the structure of an exemplary distributed training system 10. As shown in FIG1 , distributed training system 10 may include multiple computing nodes and at least one switching node, through which the multiple computing nodes may communicate. For ease of understanding, FIG1 illustrates an example system comprising four computing nodes (computing nodes 101 through 104) and three switching nodes (switching nodes 201 through 203).

[0046] Among them, the computing node can be a node with model training capability. Exemplarily, the computing node can be an accelerator card, which can be, for example, a deep learning processor (deep-learning processing unit, DPU), a data processing unit (Data processing unit, DPU), a graphics processing unit (graphics processing unit, GPU), a neural network processor (neural-network processing unit, NPU), or a tensor processing unit (tensor processing unit, TPU), etc., or other types of accelerator cards. Alternatively, the computing node can be a general-purpose processor, such as a central processing unit (CPU), etc. Alternatively, the computing node can also include a computing device of a CPU and an accelerator card. This application does not limit the specific implementation of the computing node.

[0047] Furthermore, each computing node is configured with a memory, which may be, for example, a high bandwidth memory (HBM) or a dynamic random access memory (DRAM), or may be other types of memory, etc., which are not limited to this.

[0048] A switching node may be a node with data forwarding capability, such as a switch, a router, etc.

[0049] In the distributed training system 10 shown in Figure 1, computing nodes 101 and computing nodes 104 can be used to perform distributed training on the AI ​​model. Exemplarily, the AI ​​model can be, for example, a large language model Meta AI (LLaMA) model, a bidirectional encoder representations from transformers (BERT) model, or a generative pre-trained Transform (GPT-3) model, or other types of models, such as GPT-4, etc., which are not limited to this. Generally, the AI ​​model can include multiple computing layers, taking N computing layers as an example, namely computing layer 1 to computing layer N, where N is a positive integer greater than 1. The output data of the previous computing layer can be used as the input data of the next computing layer, and the output data of the last computing layer can be the output data of the AI ​​model, such as the output data of computing layer 1 is the input data of computing layer 2, and the output data of computing layer N is the output data (inference result) of the AI ​​model.

[0050] In actual application, the distributed training system 10 can perform distributed training on the AI ​​model based on tensor parallelism or sequence parallelism.

[0051] Among them, tensor parallelism refers to dividing the matrix corresponding to the tensor in the computing layer into multiple different sub-matrices, and deploying different sub-matrices on different computing nodes for calculation; and during the model training process, multiple computing nodes can communicate collectively to obtain the calculation results of each computing node for the sub-matrix and merge them, so as to continue the calculation of the next computing layer based on the merged calculation results.

[0052] Sequence parallelism refers to dividing the sequence input to the AI ​​model into multiple subsequences, and different subsequences are calculated using AI models on different computing nodes; and during the model training process, multiple computing nodes can communicate collectively to obtain the calculation results of each computing node based on the subsequence and merge them, so as to continue the calculation of the next computing layer based on the merged calculation results.

[0053] In the process of training the AI ​​model in one round, each computing node can first execute the forward calculation process of the AI ​​model. Specifically, taking computing node 101 as an example, computing node 101 can input the training sample as input data into the AI ​​model, and computing layer 1 (i.e., the first computing layer) in the AI ​​model can perform calculations based on the input data, specifically using the activation function in the computing layer 1 to calculate the input data to obtain an activation value. The generated activation value is used for further calculation to obtain the intermediate data of the computing layer 1, or the activation value can be used as the intermediate data of the computing layer 1. Exemplarily, the activation function can be, for example, a Sigmoid function, a rectified linear unit (ReLU), etc., or can be other types of activation functions. Then, computing node 101 executes the communication operator in computing layer 1 (hereinafter referred to as the forward communication operator), and exchanges the intermediate data calculated by each of them (the intermediate data of the first computing layer) with computing node 102 to computing node 104 through the switching node, so that computing node 101 can process the intermediate data generated by the four computing nodes respectively (such as data aggregation or splicing, etc.), and use the obtained processing results as the output data of computing layer 1.

[0054] Next, computing node 101 can use the output data of computing layer 1 as the input data of computing layer 2 (the second computing layer), and use the activation function in computing layer 2 to calculate the input data to obtain the activation value generated by computing layer 2. The activation value is used for further calculation to obtain the intermediate data of computing layer 2, or the activation value can be used as the intermediate data of computing layer 2. Then, computing node 101 can execute the forward communication operator in computing layer 2, and exchange the intermediate data calculated by each of computing nodes 102 to computing node 104 (the intermediate data generated by the second computing layer) respectively, so that computing node 101 can process the intermediate data generated by the four computing nodes respectively, and use the obtained processing results as the output data of computing layer 2.

[0055] And so on, until the calculation process of calculation layer N (the last calculation layer) in the AI ​​model is completed. At this point, the forward calculation phase ends, and the output data of calculation layer N is the inference result obtained by executing the forward calculation process. Usually, the storage resources of computing node 101 are limited, such as the memory space of computing node 101 is small, so in the forward calculation phase, computing node 101 can only save (in memory) the output data of each calculation layer, or computing node 101 saves the output data of each calculation layer and the activation values ​​generated by some calculation layers.

[0056] After completing the forward calculation process, the computing node 101 (and other computing nodes) can continue to perform the reverse calculation process. Specifically, the computing node 101 can compare the difference between the inference result obtained by executing the forward calculation process and the label (actual result) in the training sample. For the computing layer N in the AI ​​model, if the activation value generated by the computing layer N is not saved in the forward calculation stage, the computing node 101 will re-use the activation function in the computing layer N to calculate the output data based on the output data of the previous computing layer (i.e., computing layer N-1) to obtain the activation value (i.e., obtain the activation value by recalculation), so that the computing node 101 can update the parameters in the computing layer N (specifically, update the values ​​of the parameters in the computing layer N) based on the recalculated activation value and the difference between the inference result and the label. In specific implementation, the computing node 101 can first calculate the gradient based on the activation value and the difference. The gradient can be used to indicate the change in the value of the parameter in the computing layer N in a certain direction, so that the computing node 101 can use the gradient to update the parameters in the computing layer N.

[0057] After completing the parameter update for computation layer N, computation node 101 may continue to calculate the gradient corresponding to computation layer N-1 based on the activation value of computation layer N-1 and the gradient corresponding to computation layer N, so as to use the gradient corresponding to computation layer N-1 to update the parameters in computation layer N-1. If the activation value generated by computation layer N-1 is saved during the forward computation phase, computation node 101 may directly use the saved activation value and the gradient corresponding to computation layer N to calculate the gradient corresponding to computation layer N-1, so as to use the gradient to update the parameters in computation layer N-1. If computation node 101 does not save the activation value generated by computation layer N-1 during the forward computation phase, computation node 101 may use the output data of computation layer N-2 and re-calculate the output data using the activation function in computation layer N-1 to obtain the activation value, so as to use the re-calculated activation value and the calculated gradient corresponding to computation layer N to calculate the gradient corresponding to computation layer N-1, so as to use the gradient to update the parameters in computation layer N-1.

[0058] And so on, until the parameter update of the computing layer 1 in the AI ​​model is completed. At this point, the reverse calculation phase ends, and a round of training process for the AI ​​model is completed. During the actual training process, the computing node 101 (and other computing nodes) can refer to the above method to perform multiple rounds of iterative training on the AI ​​model until the AI ​​model meets the termination conditions of the model training, such as the number of iterative training reaches a preset number, or the AI ​​model converges, etc.

[0059] However, in actual application scenarios, computing node 101 performs recalculations at the computational layer granularity. That is, computing node 101 not only re-executes the activation function in the computational layer, but also continues to execute the forward communication operator in the computational layer after obtaining the activation value. This causes computing node 101 to perform unnecessary communication processes with other computing nodes. In this case, the delay in computing node 101 communicating with other computing nodes will reduce the efficiency of computing node 101 in updating the parameters of the computational layer, thereby affecting the overall efficiency of distributed training for AI models.

[0060] Based on this, in the distributed training system 10 shown in FIG1 , the computing node 101 can improve the overall efficiency of distributed training by optimizing the forward communication operator.

[0061] Specifically, in the reverse calculation stage, the computing node 101 can obtain the recalculation layer of the AI ​​model in the reverse calculation process, and obtain the forward communication operator in the recalculation layer. Then, the computing node 101 removes the forward communication operator from the recalculation layer, and then uses the recalculation layer without the forward communication operator to perform reverse calculation. In this way, in the reverse calculation stage, after the computing node 101 recalculates the recalculation layer to obtain the activation value, there is no need to execute the removed forward communication operator. Then, the computing node 101 updates the parameters of the original computing layer (that is, the computing layer without removing the forward communication operator) based on the activation value obtained by recalculation. For ease of understanding, the following is an example of the computing node 101 updating the parameters in the computing layer N.

[0062] Assuming that in the forward calculation phase, the computing node 101 does not save the activation value generated by the computing layer N, then, in the reverse calculation phase, the computing node 101 can determine that the computing layer N is a recalculation layer and obtain the forward communication operator in the recalculation layer. Then, the computing node 101 can first remove the forward communication operator in the recalculation layer to obtain the target computing layer N (that is, the computing layer N after removing the forward communication operator). Next, the computing node 101 can recalculate the target computing layer N to obtain the activation value generated by the activation function in the target computing layer N. Since the target computing layer N does not include the removed forward communication operator, when recalculating the target computing layer N, the computing node 101 will not execute the forward communication operator. Finally, the computing node 101 can use the recalculated activation value to calculate the gradient corresponding to the computing layer N, and use the gradient to update the parameters in the original computing layer N in the AI ​​model. That is, the target computing layer N is only used to recalculate the activation value. During the parameter update process, the computing node 101 will still update the parameters of the computing layer N including the forward communication operator.

[0063] Similarly, for other computing layers in the AI ​​model, the above method can be used to recalculate the activation value and update the parameters in the computing layer.

[0064] In this way, in the reverse calculation stage, the computing node 101 recalculates the computing layer without the forward communication operator, so as to obtain the activation value required for updating the parameters in the computing layer, and does not need to execute the forward communication operator in the computing layer (that is, the recalculation layer determined based on the computing layer). This can avoid the communication delay caused by the computing node 101 executing unnecessary forward communication operators and affect the efficiency of updating the model parameters, thereby improving the overall efficiency of distributed training for AI models.

[0065] It is worth noting that the distributed training system 10 shown in FIG1 above is only an exemplary illustration and is not intended to be limiting. For example, in actual application scenarios, the distributed training system 10 may include a larger number of computing nodes and switching nodes; or, the distributed training system 10 may also include other nodes, such as a scheduling node, for instructing computing nodes 101 to 104 to start, pause, or end the distributed training process. In addition, the distributed training system 10 may support providing one or more distributed training frameworks for AI models, such as the PyTorch framework, the Megatron framework, etc., without limitation.

[0066] For ease of understanding, an embodiment of the model training method provided in this application is described below in conjunction with the accompanying drawings.

[0067] Referring to Figure 2, Figure 2 is a flow chart of a model training method provided in an embodiment of the present application. This method can be applied to the distributed training system 10 described in Figure 1, or can be applied to other applicable distributed training systems. For ease of explanation, this embodiment uses the distributed training system 10 shown in Figure 1 and the process of computing node 101 updating parameters in a computing layer in the AI ​​model as an example for exemplary description.

[0068] The model training method shown in FIG2 may specifically include:

[0069] S201: During the forward calculation of the AI ​​model, the computing node 101 saves the input data of the computing layer N. The computing layer N includes a computing module and a forward communication operator. In the forward calculation phase, the execution order of the computing module precedes the execution order of the forward communication operator.

[0070] Wherein, N is a positive integer greater than 1.

[0071] In this embodiment, the AI ​​model may include multiple computing layers. For ease of understanding, the AI ​​model is described as including N computing layers (i.e., computing layers 1 to N). Computation layer N-1 and computing layer N are two computing layers included in the AI ​​model; and in the forward computing phase, computing layer N-1 is the previous computing layer of computing layer N. That is, after computing layer N-1 completes the calculation, computing node 101 can use the output data of computing layer N-1 as the input data of computing layer N.

[0072] The computing layer N may include at least one computing module and at least one forward communication operator, taking a computing module and a communication operator as an example. The computing module refers to a neural network module in the computing layer that includes an activation function and has a certain computing logic, so that after the computing node 101 executes the computing module, it can calculate the corresponding activation value based on the activation function in the computing module. For example, the computing module can be an attention computing module, or a feedforward neural network (FNN), etc. The forward communication operator refers to an operator used to interact with other computing nodes in the forward computing phase. For example, in a tensor-parallel distributed training scenario, the forward communication operator can be an All-Reduce operator; in a sequence-parallel distributed training scenario, the forward communication operator can be a Reduce-Scatter operator, etc., and this is not limited.

[0073] In the forward calculation phase, the execution order of the calculation module precedes the execution order of the forward communication operator, that is, the computing node 101 executes the forward communication operator after executing the calculation module.

[0074] In the forward calculation phase, the computing node 101 can use the computing module in the computing layer N to calculate the input data, obtain the corresponding activation value, and further calculate the intermediate data based on the activation value, or determine the activation value as the intermediate data. Then, the computing node 101 can execute the forward communication operator in the computing layer N to interact with other computing nodes to execute the intermediate data generated by the computing layer N. In this way, the computing node 101 can process the intermediate data generated by each computing node. For example, in a tensor parallel distributed training scenario, the computing node 101 can splice the intermediate data generated by multiple computing nodes; in a sequence parallel distributed training scenario, the computing node 101 can aggregate the intermediate data generated by multiple computing nodes, etc.

[0075] In this embodiment, computing node 101 may store the output data of computing layer N, but not the activation values ​​generated by computing layer N, to prevent the activation values ​​from occupying storage resources in computing node 101. For other computing layers, computing node 101 may store both the output data of the other computing layers and the activation values ​​generated by them, or computing node 101 may store only the output data of the other computing layers, etc., and this is not limited to the present invention.

[0076] In actual application, the computing node 101 can refer to the above method and perform corresponding calculations based on each computing layer to complete the forward computing process.

[0077] S202: During the reverse calculation of the AI ​​model, the computing node 101 obtains the recalculation layer of the AI ​​model during the reverse calculation.

[0078] In this embodiment, it is assumed that computing node 101 needs to update the values ​​of parameters in computing layer N based on the activation value generated by computing layer N. However, during the forward computation, computing node 101 does not save the activation value generated by computing layer N. Therefore, during the reverse computation, computing node 101 may determine that computing layer N needs to be recalculated to obtain the activation value generated by computing layer N. In this case, computing node 101 may determine that computing layer N is the recalculated layer in the reverse computation.

[0079] Similarly, for a computing layer in an AI model that needs to perform parameter updates based on activation values ​​during the reverse computing phase, computing node 101 may determine the computing layer as a re-computation layer.

[0080] S203: The computing node 101 obtains the forward communication operator in the recomputation layer.

[0081] It is understood that among the multiple recomputation layers determined by computing node 101, some may contain at least one forward communication operator, while others may not contain any. Therefore, computing node 101 may further filter the determined recomputation layers to obtain a recomputation layer that includes a forward communication operator, and further determine the forward communication operator in the recomputation layer.

[0082] In this embodiment, the forward communication operator obtained may be the terminal forward communication operator in the recomputation layer. The terminal forward communication operator refers to the forward communication operator located at the end of the recomputation layer, that is, the forward communication operator executed after the recomputation layer completes all calculation modules during the forward calculation phase. In this case, the forward communication operator is the terminal forward communication operator described in this embodiment.

[0083] S204: The computing node 101 removes the forward communication operator from the recomputation layer.

[0084] In this embodiment, assuming that computing layer N is a recomputation layer, computing node 101 will first remove the forward communication operator in computing layer N to obtain a new computing layer N, that is, the recomputation layer after removing the forward communication operator. For ease of distinction, computing layer N after removing the forward communication operator will be referred to as target computing layer N below. Exemplarily, computing node 101 can copy the program code of computing layer N and delete the code portion corresponding to the forward communication operator from the program code, so that the remaining program code is the program code of target computing layer N. The computing layer / target computing layer outputting data or generating an activation value described in this embodiment specifically refers to the computing node 101 executing the program code corresponding to the computing layer or target computing layer to implement the operation of outputting data or generating an activation value.

[0085] In actual application, in the recalculation layer, after the forward communication operator is removed, other calculation modules may be included, such as a residual connection module (used to solve the problem of gradient disappearance or gradient explosion), etc. For ease of understanding, it is referred to as a post-processing module below. As shown in Figure 3, the forward communication operator is located between the calculation module and the post-processing module, that is, the execution order of the forward communication operator in the recalculation layer is after the execution order of the calculation module and before the execution order of the post-processing module. At this time, if the matrix dimension (shape) of the activation value calculated by the calculation module is inconsistent with the matrix dimension of the input data of the post-processing module, after removing the forward communication operator located in the middle, the matrix dimension of the activation value output by the calculation module does not match the matrix dimension of the input data of the post-processing module, which may cause the computing node 101 to detect an abnormality in the input data during the execution of the post-processing module and stop executing the post-processing module, thereby causing the model training to be interrupted.

[0086] Therefore, in a further possible implementation, after removing the forward communication operator, the computing node 101 can also add a matrix dimension alignment module between the computing module and the post-processing module, as shown in FIG3 , such as adding the program code of the matrix dimension alignment module in the program code of the target computing layer N. Among them, the matrix dimension alignment module refers to a module that processes data according to the matrix dimension. In this way, after removing the forward communication operator, the matrix dimension alignment module can fill the output data of the computing module (i.e., the above-mentioned intermediate data) according to the matrix dimension of the input data of the post-processing module. Specifically, it can be to fill a plurality of elements in the matrix corresponding to the output data, and the value of each element can be 0 (or other values). In this way, the matrix dimension of the output data of the computing module after filling is consistent with the matrix dimension of the input data of the post-processing module, so that in the reverse calculation stage, the matrix dimension alignment module can input the filled output data into the post-processing module for corresponding calculation.

[0087] Similarly, for other heavy calculation layers including the forward communication operator, the computing node 101 can remove the forward communication operator by referring to the above processing method for the calculation layer N, which is not described in detail.

[0088] S205: The computing node 101 performs reverse computing by using the recomputation layer that removes the forward communication operator.

[0089] In a possible implementation, during the reverse calculation process, the computing node 101 may first perform a recalculation layer that removes the forward communication operator to obtain an activation value.

[0090] In a specific implementation, taking the target computation layer N as an example, the computation node 101 can recompute the target computation layer N based on the output data of computation layer N-1 saved during the forward computation phase to obtain the activation value generated by the target computation layer N. Computing node 101 can use the output data of computation layer N-1 as the input data for the target computation layer N and calculate the input data based on the activation function (such as the Sigmoid function) in the computation module to obtain the activation value generated by the activation function. Typically, the calculated activation value is the same as the activation value calculated by computation layer N during the forward computation phase. In this way, by recomputing the target computation layer N, computing node 101 can obtain the activation value not saved during the forward computation phase. Furthermore, computing node 101 does not need to execute the removed forward computation operator during the recomputation of the target computation layer N, which effectively avoids the communication delay caused by computing node 101 executing the forward computation operator.

[0091] After obtaining the activation value by recalculating the target computing layer N, the computing node 101 can update the parameters in the original computing layer N in the AI ​​model according to the activation value.

[0092] Specifically, the computing node 101 can calculate the gradient corresponding to each parameter in the computing layer N based on the activation value, and then the computing node 101 uses the gradient corresponding to each parameter to update the value of the parameter, which can specifically be to increase or decrease the value of the parameter according to the direction indicated by the gradient.

[0093] Among them, when the computing layer N is the last computing layer in the AI ​​model, the computing node 101 can calculate the gradient corresponding to each parameter in the computing layer N based on the activation value, the inference result of the AI ​​model and the actual result, so as to update the value of each parameter in the computing layer N according to the gradient.

[0094] When computing layer N is not the last computing layer in the AI ​​model (assuming the AI ​​model includes N+1 computing layers), computing node 101 can also obtain the gradient of the next computing layer after computing layer N (i.e., computing layer N+1), such as by executing a reverse communication operator to obtain the gradient used by computing layer N+1 when updating parameters. Then, computing node 101 can calculate the gradient corresponding to each parameter in computing layer N based on the gradient of computing layer N+1 and the activation value generated by computing layer N, so as to update the values ​​of each parameter in computing layer N according to the gradient.

[0095] In this way, in the reverse calculation phase, the computing node 101 can implement parameter updates for a computing layer (i.e., computing layer N) based on the method described in steps S202 to S205 above. For other computing layers in the AI ​​model, the computing node 101 can also refer to the above similar method. When the activation value has been saved, the saved activation value is used to update the parameters in the computing layer; when the activation value is not saved, the computing layer can be determined as a recalculation layer, and the activation value is recalculated by executing the recalculation layer without the forward communication operator, and then the recalculated activation value is used to update the parameters in the computing layer. And so on, until all computing layers in the AI ​​model have completed parameter updates, at which point the reverse calculation process ends. In addition, each computing node can refer to the above implementation method to perform forward calculation and reverse calculation processes on the AI ​​model deployed thereon, thereby completing a round of model training process for the AI ​​model in the distributed training system 10.

[0096] In this embodiment, computing node 101 does not need to execute the removed forward communication operator during the recalculation process. This also prevents the communication delay generated by executing the forward communication operator from affecting the update efficiency of model parameters, thereby improving the overall efficiency of distributed training for AI models. In actual testing scenarios, optimizing the forward communication operator can reduce the time consumption introduced by the recalculation process by more than 25%.

[0097] In the embodiment shown in FIG2 above, an example is given in which a computing layer includes a forward communication operator. In actual application scenarios, some computing layers in the AI ​​model may include multiple forward communication operators. As shown in FIG4 , in the computing layer M (that is, the recomputation layer M in the reverse calculation phase), two computing modules and two forward communication operators may be included, and the execution order of each computing module and the forward communication operator is shown in FIG4 . At this time, during the reverse calculation process, the computing node 101 can remove all forward communication operators included in the calculation.

[0098] Below, in conjunction with Figure 5, a flow chart of another model training method provided by an embodiment of the present application is introduced. In the embodiment shown in Figure 5, the computing layer M in the AI ​​model includes two forward communication operators (M is a positive integer greater than 1). As shown in Figure 5, the method may specifically include:

[0099] S501: The computing node 101 and other computing nodes perform a first round of distributed training on the AI ​​model, and determine the amount of storage resources of the computing node 101 occupied during the first round of distributed training.

[0100] In actual application, for a computation layer whose activation value is not saved in the forward computation phase, the computing node 101 may determine that the computation layer is a recomputation layer for the reverse computation phase. Furthermore, when the recomputation layer includes multiple forward communication operators, the computing node 101 may split the recomputation layer into multiple sub-recomputation layers so that the forward communication operators in each sub-recomputation layer can be subsequently removed for recomputation. Accordingly, the computing node 101 needs to save the input data of each sub-recomputation layer during the forward computation process so that the sub-recomputation layer can be recomputation based on the saved input data during the reverse computation process.

[0101] To this end, computing node 101 may first estimate whether the number of storage resources of computing node 101 meets the recomputation layer division condition, and when the number of storage resources of computing node 101 meets the recomputation layer division condition, computing node 101 may determine to divide the recomputation layer into multiple sub-recomputation layers; and when the number of storage resources of computing node 101 does not meet the recomputation layer division condition, computing node 101 may not perform the operation of dividing the recomputation layer into multiple sub-recomputation layers.

[0102] In this embodiment, the computing node 101 can complete a round of distributed training process for the AI ​​model without dividing the recomputation layer into multiple sub-recomputation layers to obtain the number of storage resources of the computing node 101 occupied in the first round of distributed training process, so that the computing node 101 can determine whether the number of storage resources of the computing node 101 meets the recomputation layer division condition based on the number of occupied storage resources.

[0103] Exemplarily, the occupied storage resources may be, for example, memory resources of the computing node 101 .

[0104] The storage resources occupied are used to store the output data of each calculation layer in the forward calculation phase, and can also be used to store the activation values ​​generated by some calculation layers (these calculation layers are not identified as recalculation layers in the reverse calculation process).

[0105] Among them, the computing node 101 can refer to the embodiment shown in Figure 2 above to perform the forward computing process and the reverse computing process to complete the first round of distributed training for the AI ​​model.

[0106] S502: The computing node 101 determines whether the amount of storage resources of the computing node 101 occupied during the first round of distributed training is less than a threshold. If not, step S503 is executed; if less than the threshold, step S504 is executed.

[0107] The threshold value may be set by a technician based on the total number of storage resources of computing node 101, or may be calculated by computing node 101 based on the total number of storage resources of computing node 101, and this is not limited thereto. For example, the threshold value may be 85% of the maximum number of storage resources of computing node 101, and this is not limited thereto.

[0108] S503: When the amount of storage resources of the computing node 101 occupied during the first round of distributed training is greater than or equal to the threshold, the computing node 101 performs a subsequent iterative training process on the AI ​​model based on the first round of distributed training of the AI ​​model.

[0109] S504: When the amount of storage resources of the computing node 101 occupied during the first round of distributed training is less than a threshold, the computing node 101 determines the computing layer M including multiple forward communication operators as the recomputation layer M, and divides the recomputation layer M into two sub-recomputation layers according to the number of forward communication operators included in the recomputation layer M, namely, sub-recomputation layer M1 and sub-recomputation layer M2, each sub-recomputation layer including one forward communication operator.

[0110] The forward communication operator in each sub-recomputation layer may be the terminal forward communication operator in the sub-recomputation layer. Furthermore, computing node 101 may refer to the process described in FIG. 2 to determine the recomputation layer and the forward communication operator in the recomputation layer during the reverse calculation process, which will not be further described here.

[0111] In this embodiment, the computing node 101 can perceive the occupancy of storage resources during the model training process by performing a round of distributed training on the AI ​​model, and can further determine whether the maximum number of storage resources of the computing node 101 meets the conditions for splitting the recalculation layer (hereinafter referred to as the recalculation layer division condition). If it is satisfied, that is, the number of storage resources occupied by the computing node 101 in performing a round of distributed training is less than a threshold value (such as the maximum number of storage resources or 85% of the maximum number), etc., then the computing node 101 can divide the recalculation layer including multiple forward communication operators into multiple sub-recalculation layers. If it is not satisfied, that is, the number of storage resources occupied by the computing node 101 in performing a round of distributed training is greater than or equal to the threshold value, then the computing node 101 may not split the recalculation layer including multiple forward communication operators.

[0112] Taking the recalculation layer M shown in Figure 4 as an example, it includes calculation module 1, forward communication operator 1, calculation module 2, and forward communication operator 2 in sequence. For example, calculation module 1 can be an attention calculation module, calculation module 2 can be a feedforward neural network module, and forward communication operator 1 and forward communication operator 2 can both be All-Reduce operators. Then, the computing node 101 can divide the recalculation layer M into the sub-recalculation layer M1 and the sub-recalculation layer M2 shown in Figure 6. Among them, the sub-recalculation layer M1 includes calculation module 1 and forward communication operator 1, and the sub-recalculation layer M2 includes calculation module 2 and forward communication operator 2.

[0113] In one possible implementation, after completing the first round of distributed training of the AI ​​model, the computing node 101 may determine the usage of storage resources for a round of distributed training. When the amount of storage resources used in a round of distributed training is less than a threshold value, it indicates that the computing node 101 still has more storage resources to store other data. At this time, the computing node 101 may divide the recalculation layer including multiple forward communication operators into multiple sub-recalculation layers, each sub-recalculation layer may include only one forward communication operator, and the forward communication operator is located at the end of the sub-recalculation layer (the forward communication operator may be referred to as the terminal forward communication operator), so that the forward communication operator in the sub-recalculation layer can be removed in the subsequent recalculation stage. In this embodiment, the example of dividing the recalculation layer M into multiple sub-recalculation layers is used for illustration. For other recalculation layers determined in the reverse calculation process, the recalculation layer may also be divided into multiple sub-recalculation layers in a similar manner.

[0114] In addition, when the amount of storage resources used during a round of distributed training is greater than or equal to a threshold, it indicates that the storage resources of computing node 101 during the training process are relatively tight and insufficient to store other data. In this case, computing node 101 can perform multiple rounds of iterative training on the AI ​​model in the manner of the embodiment shown in Figure 2 above.

[0115] S505: The computing node 101 and other computing nodes start to execute the second round of distributed training process for the AI ​​model, and in the forward computing phase, the computing node 101 saves the input data of the sub-computing layer M1 and the input data of the sub-computing layer M2.

[0116] In this embodiment, after the computing node 101 divides the recalculation layer M into multiple sub-recalculation layers, it can save the input data corresponding to each sub-recalculation layer in the forward calculation phase, so that the saved input data can be used to perform the corresponding recalculation process on the sub-recalculation layer in the reverse calculation phase. Among them, the input data of the sub-recalculation layer M1 is the input data of the recalculation layer M; the output data of the sub-recalculation layer M1 is the input data of the sub-recalculation layer M2; and the output data of the sub-recalculation layer M2 is the output data of the recalculation layer M. That is, compared to the first round of distributed training process, the output data of the sub-recalculation layer M1 will be additionally saved in the second round of distributed training (so that the output data can be used as the input data of the sub-recalculation layer M2 for recalculation in the reverse calculation process).

[0117] Among them, when there are other recomputation layers in the AI ​​model that are also divided into multiple sub-recomputation layers, in the forward calculation stage, the computing node 101 can save the input data of each sub-recomputation layer included in the other recomputation layers.

[0118] S506: In the reverse calculation phase, the computing node 101 removes the forward communication operator in the sub-computation layer M1 to obtain the target layer M1, and removes the forward communication operator in the sub-computation layer M2 to obtain the target layer M2.

[0119] Among them, the implementation method of the computing node 101 removing the forward communication operators in each sub-computing layer can be referred to the description of the relevant parts about removing the communication operators in the computing layer N in the embodiment shown in Figure 2 above, which will not be repeated here.

[0120] S507: The computing node 101 recalculates the target layer M2 according to the input data of the target layer M2 saved in the forward computing phase, and obtains the activation value 2 generated by the target layer M2.

[0121] In this embodiment, after the computation layers are layered, since the parameter values ​​in the target layer M2 are first updated in the reverse computation phase, the computation node 101 can first recompute the target layer M2 in the order in which the parameters are updated in the reverse computation phase to obtain the corresponding activation value 2. The computation node 101 then executes step S508 to recompute the target layer M1.

[0122] S508: The computing node 101 recalculates the target layer M1 according to the input data of the target layer M1 saved in the forward computing phase, and obtains the activation value 1 generated by the target layer M1.

[0123] In this embodiment, the specific implementation process of the computing node 101 recalculating the target layer M1 and the target layer M2 can be found in the description of the recalculation of the target computing layer N in the embodiment shown in Figure 2 above, and will not be repeated here.

[0124] S509: The computing node 101 updates the parameters in the recomputation layer M according to the activation value 1 and the activation value 2.

[0125] In a specific implementation, the computing node 101 can generate a reverse computing graph 1 based on the computing logic in the target layer M1, and generate a reverse computing graph 2 based on the computing logic in the target layer M2, wherein the reverse computing graph 1 is used to indicate the logic for performing reverse step-by-step computing on the target layer M1, and the reverse computing graph 2 is used to indicate the logic for performing reverse step-by-step computing on the target layer M2. Then, the computing node 101 can calculate the gradient 2 corresponding to each parameter in the computing module 2 based on the activation value 2. Specifically, the gradient 2 corresponding to each parameter in the computing module 2 can be calculated based on the activation value 2 and the gradient used to update the parameters in the computing layer M+1, and based on the computing logic indicated by the reverse computing graph 2, the values ​​of each parameter in the computing module 2 are updated according to the gradient 2. Finally, computing node 101 calculates the gradient 1 corresponding to each parameter in computing module 1 based on activation value 1. Specifically, computing node 101 calculates the gradient 1 corresponding to each parameter in computing module 1 based on activation value 1 and gradient 2 used to update the parameters in computing module 2, and updates the values ​​of each parameter in computing module 1 according to the gradient 1 based on the calculation logic indicated in reverse calculation diagram 2.

[0126] Similarly, for other computing layers in the AI ​​model, the above method can also be used to update the parameter values ​​in the computing layer until the parameter updates of all computing layers in the AI ​​model are completed. At this point, the second round of distributed training process for the AI ​​model is completed. In actual application, computing node 101 can refer to steps S504 to S509 above to perform multiple rounds of iterative training on the AI ​​model until the AI ​​model meets the termination conditions of model training.

[0127] It should be noted that the execution order of steps S501 to S509 in this embodiment is only an implementation example and is not intended to be limiting. That is, in other embodiments, the computing node 101 may also use other step sequences to execute the model training process. Moreover, in this embodiment, the computing node 101 determines whether to divide the recalculation layer including multiple forward communication operators based on the number of storage resources occupied during the first round of distributed training. In other embodiments, the computing node 101 may also determine whether the number of storage resources of the computing node 101 meets the recalculation layer division conditions in other ways. For example, the computing node 101 can estimate whether the number of storage resources occupied by the computing node 101 when training the AI ​​model after splitting the computing layer is greater than a threshold based on the number of parameters in the AI ​​model, and then determine whether the number of storage resources of the computing node 101 meets the computing layer division conditions, etc. In this way, when it is determined that the number of storage resources of the computing node 101 can meet the computing layer division conditions, the computing node 101 can divide the recalculation layer including multiple forward communication operators into multiple sub-recalculation layers in the reverse calculation stage during the first round of distributed training of the AI ​​model, and recalculate each sub-recalculation layer that excludes the forward communication operators.

[0128] In this embodiment, when the storage resources of computing node 101 are relatively sufficient, computing node 101 can divide the computing layer including multiple forward communication operators into multiple sub-computation layers, and in the reverse computing phase, by recomputing the sub-computation layers without the forward communication operators, all forward communication operators in the recomputation layers can be removed, thereby further avoiding the communication delay generated by the execution of the forward communication operators from affecting the update efficiency of the model parameters, thereby improving the overall efficiency of distributed training for the AI ​​model. In actual application, in the reverse computing phase, for all recomputation layers in the AI ​​model, computing node 101 can remove all forward communication operators in all recomputation layers.

[0129] The embodiments shown in Figures 5 and 6 above can be applicable to tensor parallel distributed training scenarios. At this time, the forward communication operator included in the computing layer can be an All-Reduce operator, that is, after the computing node 101 executes the All-Reduce operator, it can obtain the intermediate data generated by other computing nodes (obtained according to the activation value) and perform aggregation calculations (such as summation, etc.) on the intermediate data generated by computing nodes 101 to computing nodes 104 respectively, so that the computing node can perform subsequent calculation processes based on the aggregation results.

[0130] In other possible distributed training scenarios, such as serial parallel distributed training scenarios, the forward communication operator included in the computing layer is used to instruct the splicing of intermediate data generated by multiple computing nodes, such as the forward communication operator can be a Reduce-Scatter operator. At this time, in the reverse calculation phase, the computing node 101 needs to execute the corresponding communication operator (hereinafter referred to as the reverse communication operator) to exchange gradients with other computing nodes and perform gradient aggregation, so as to update the parameters in the computing layer based on the aggregated gradients, such as the reverse communication operator can be an All-Reduce operator. Therefore, the computing node 101 can execute the reverse communication operator before updating the parameters in the computing layer. Below, this is exemplified in conjunction with Figure 7.

[0131] Referring to Figure 7, a flow chart of another model training method is shown. As shown in Figure 7, the method may include:

[0132] S701: The computing node 101 obtains the model file and startup script corresponding to the AI ​​model.

[0133] Among them, the model file is used to record the program code of the AI ​​model to be trained, and the model file can indicate the structure of the AI ​​model and the initialization values ​​of the model parameters. The startup script is used to record the program code required to be executed when starting model training, and the startup script can also include configuration contents such as whether to turn on the recalculation configuration, whether to turn on the configuration of dividing the recalculation layer (that is, dividing the recalculation layer including multiple forward communication operators into multiple sub-recalculation layers), and the parallel mode of distributed training; in actual application, the user can configure this in the startup script. In this embodiment, the user sets the configuration of turning on recalculation, turning on the configuration of dividing the recalculation layer, and the parallel mode of distributed training as serial parallel in the startup script as an example for explanation.

[0134] In the distributed training system 10, when the AI ​​model is distributedly trained using multiple computing nodes 101, the model file and startup script corresponding to the AI ​​model can be deployed on each computing node, so that each computing node can start the training process for the AI ​​model based on the startup script and the model file.

[0135] S702: The computing node 101 starts distributed training of the AI ​​model according to the startup script, and identifies the forward communication operators in the multiple recalculation layers and each recalculation layer included in the AI ​​model in the reverse calculation phase according to the model file.

[0136] In a specific implementation, the computing node 101 can analyze the model file to determine one or more computing layers in the AI ​​model that do not save activation values ​​in the forward computing phase but need to use the activation values ​​to calculate gradients in the reverse computing phase. This computing layer is also the recalculation layer that needs to recalculate the activation value in the reverse computing phase. Then, the computing node 101 can identify the forward communication operators included in the computing layer, such as the Reduce-Scatter operator, and mark the identified forward communication operators, that is, mark the forward communication operators as operators that can be removed in the reverse computing phase.

[0137] S703: The computing node 101 and other computing nodes perform a first round of distributed training on the AI ​​model, and determine the amount of storage resources of the computing node 101 occupied during the first round of distributed training.

[0138] In the reverse calculation phase included in the first round of distributed training, the computing node 101 may not perform the process of removing the forward communication operator and dividing the recalculation layer.

[0139] S704: The computing node 101 determines whether the amount of storage resources of the computing node 101 occupied during the first round of distributed training is less than a threshold. If not, step S705 is executed; if less than the threshold, step S706 is executed.

[0140] S705: When the amount of storage resources of the computing node 101 occupied during the first round of distributed training is greater than or equal to the threshold, the computing node 101 performs a subsequent iterative training process on the AI ​​model based on the first round of distributed training of the AI ​​model.

[0141] S706: When the amount of storage resources of the computing node 101 occupied during the first round of distributed training is less than the threshold, the computing node 101 determines the computing layer M as the recomputation layer M, and divides the recomputation layer M into two sub-recomputation layers according to the number of forward communication operators included in the recomputation layer M, namely sub-recomputation layer M1 and sub-recomputation layer M2, each of which includes a forward communication operator.

[0142] S707: The computing node 101 and other computing nodes start to execute the second round of distributed training process for the AI ​​model, and in the forward computing phase, the computing node 101 saves the input data of the sub-computing layer M1 and the input data of the sub-computing layer M2.

[0143] S708: In the reverse calculation phase, the computing node 101 removes the forward communication operator in the sub-computation layer M1 to obtain the target layer M1, and removes the forward communication operator in the sub-computation layer M2 to obtain the target layer M2.

[0144] S709: The computing node 101 inserts a reverse communication operator 1 for gradient interaction into the reverse computation graph 1 corresponding to the target layer M1, and inserts a reverse communication operator 2 for gradient interaction into the reverse computation graph 2 corresponding to the target layer M2.

[0145] As an implementation example, the computing node 101 can be pre-configured with a mapping relationship, such as a mapping relationship that can be pre-configured in the computing node 101 by a technician. The mapping relationship is a correspondence between a forward communication operator and a reverse communication operator. For example, when the forward communication operator is Reduce-Scatter, the reverse communication operator can be All-Gather, etc. In the forward computing phase, the computing node 101 executes the forward communication operator to exchange intermediate data with other computing nodes; in the reverse computing phase, the computing node 101 executes the reverse communication operator to exchange gradients (for updating model parameters) with other computing nodes.

[0146] During the reverse computation phase, compute node 101 can generate a reverse computation graph 1 based on the computation logic in target layer M1. This reverse computation graph 1 indicates the logic for performing reverse, step-by-step computations on target layer M1, typically the opposite of the computation logic in target layer M1. Since target layer M1 does not include forward communication operator 1, the reverse computation graph 1 generated by compute node 101 does not include reverse communication operator 1. Therefore, compute node 101 can insert the reverse communication operator 1 corresponding to the removed forward communication operator 1 into reverse computation graph 1. Specifically, compute node 101 can first search for a mapping relationship based on forward communication operator 1 to determine the reverse communication operator 1 corresponding to the forward communication operator 1. Computing node 101 can then insert the reverse communication operator 1 into reverse computation graph 1, for example, at the starting position of reverse computation graph 1. Thus, when performing a parameter update based on reverse computation graph 1, compute node 101 can first execute reverse communication operator 1 to obtain the gradients calculated by multiple compute nodes for target layer M1, and then perform a parameter update based on the gradients calculated by these multiple compute nodes.

[0147] The computing node 101 may automatically insert the reverse communication operator 1 into the reverse computing graph 1, or the computing node 101 may insert the reverse communication operator 1 into the reverse computing graph 1 under manual instruction of the user, which is not limited to this.

[0148] Similarly, for the reverse calculation graph 2 generated by the target layer M2, the computing node 101 can also refer to the above-mentioned similar method to insert the reverse communication operator 2 corresponding to the forward communication operator 2 into the reverse calculation graph 2, which will not be repeated here.

[0149] It should be noted that this embodiment uses the example of computing node 101 inserting the reverse communication operator into the reverse computation graph for explanation. In other embodiments, the process of inserting the reverse communication operator into the reverse computation graph may not be performed. For example, in the reverse computation phase, computing node 101 may generate a reverse computation graph 1 based on the computation logic in the sub-computation layer M1. Since the sub-computation layer M1 includes the forward communication operator 1, the reverse computation graph 1 generated by computing node 101 may include the reverse communication operator 1 corresponding to the forward communication operator 1. In this way, when computing node 101 performs parameter updates according to the reverse computation graph 1, it may directly execute the reverse communication operator 1 indicated in the reverse computation graph 1 to obtain the gradients calculated by multiple computing nodes respectively, and then perform parameter updates based on the gradients calculated by the multiple computing nodes, without having to perform the process of inserting the reverse communication operator into the reverse computation graph 1.

[0150] S710: The computing node 101 recalculates the target layer M1 according to the input data of the target layer M1 saved in the forward calculation phase to obtain the activation value 1 generated by the target layer M1, and recalculates the target layer M2 according to the input data of the target layer M2 saved in the forward calculation phase to obtain the activation value 2 generated by the target layer M2.

[0151] The input data of the target layer M1 is the output data of the calculation layer M-1, and the input data of the target layer M2 is the output data of the target layer M1.

[0152] S711: The computing node 101 updates the parameters in the computing layer M according to the activation value 1, the activation value 2, the reverse computing graph 1, and the reverse computing graph 2.

[0153] In the first implementation example, when it is necessary to update the parameters of the computing module 2, the computing node 101 can first recalculate the target layer M2 based on the input data of the target layer M2 saved in the forward calculation phase (that is, the output data of the target layer M1) to obtain the activation value 2. Then, the computing node 101 can execute the reverse communication operator 2 in the reverse calculation diagram 2 to obtain the gradient used by each computing node when updating the parameters in the computing layer M+1 (hereinafter referred to as the input gradient 2). Therefore, the computing node 101 can calculate the gradient 2 used to update the computing module 2 based on the input gradient 2 and the calculated activation value 2. In this way, the computing node 101 can update the parameters in the computing module 2 in sequence based on the calculated gradient 2 and the calculation logic indicated by the reverse calculation diagram 2. Then, computing node 101 updates the parameters of computing module 1. At this time, as shown in Figure 8, computing node 101 can first recalculate the target layer M1 to obtain the activation value 1, and can perform the reverse communication operator 1 in the reverse calculation diagram 1 to obtain the gradient used by each computing node when updating the parameters in computing module 1 (hereinafter referred to as input gradient 1). Therefore, computing node 101 can calculate the gradient 1 used to update computing module 1 based on the input gradient 1 and the calculated activation value 1. In this way, computing node 101 can sequentially update the parameters in computing module 1 based on the calculated gradient 1 and the calculation logic indicated by the reverse calculation diagram 1, as shown in Figure 8. In this way, computing node 101 implements the parameter update for computing layer M.

[0154] In the second implementation example, when it is necessary to update the parameters of the computing module 2, the computing node 101 can first recalculate the target layer M2 based on the input data of the target layer M2 saved in the forward calculation stage. Since there is no dependency between the process of the computing node 101 performing recalculation on the target layer M2 and the process of the computing node 101 executing the reverse communication operator 2, during the process of recalculating the target layer M2, the computing node 101 can simultaneously execute the reverse communication operator 2 to obtain the input gradient 2 used by each computing node when updating the parameters in the computing layer M+1. In this way, after recalculating the target layer M2 and obtaining the activation value 2, the computing node 101 can calculate the gradient 2 used to update the computing module 2 based on the input gradient 2 and the calculated activation value 2, so that the computing node 101 can update the parameters in the computing module 2 in sequence based on the calculation logic indicated by the reverse calculation diagram 2 according to the gradient 2. After completing the recalculation of the target layer M2, as shown in Figure 9, the computing node 101 can begin to recalculate the target layer M1 based on the input data of the target layer M1 saved in the forward calculation phase. During the recalculation of the target layer M1, the computing node 101 can simultaneously execute the reverse communication operator 1 to obtain the input gradient 1 used by each computing node when updating the parameters in the calculation module 2. In this way, after recalculating the target layer M1 and obtaining the activation value 1, the computing node 101 can calculate the gradient 1 used to update the calculation module 1 based on the input gradient 1 and the calculated activation value 1. Therefore, the computing node 101 can update the parameters in the calculation module 1 in sequence based on the gradient 1 and the calculation logic indicated by the reverse calculation diagram 1, thereby achieving the parameter update of the calculation layer M, as shown in Figure 9.

[0155] In this way, the computing node 101 can hide the process of executing reverse communication operator 2 and reverse communication operator 1 in the process of recalculating the target layer M2 and target layer M1, respectively, as shown in Figure 9. This can avoid the communication delay caused by the computing node 101 executing reverse communication operator 2 and reverse communication operator 1, affecting the efficiency of updating the parameter values ​​in the computing layer M, thereby further improving the overall efficiency of distributed training for AI models. Among them, the computing node 101 hides the process of executing the reverse communication operator after the recalculation process for the target layer M1, and the reduced communication delay is shown in Figure 9.

[0156] It is worth noting that in the implementation shown in Figure 9, the process of executing reverse communication operators 1 and 2 is hidden in the recalculation process for the target layer M1 and the target layer M2 as an example for explanation. In actual application, the computing node 101 can also hide the calculation process in the recalculation process. For example, after obtaining the activation value 2, the computing node 101 can start the recalculation process for the target layer M1 (to obtain the activation value 1), and at the same time execute part or all of the process of executing the reverse communication operator 2, calculating the activation value 2 to obtain the gradient 2, and updating the parameters of the computing module 2. In this way, the computing node 101 can further improve the overall efficiency of distributed training for the AI ​​model by executing other calculations or communication processes in parallel during the recalculation process.

[0157] It is worth noting that other reasonable step combinations that can be thought of by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be familiar with that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by this application.

[0158] The above introduces the model training method provided in the embodiment of the present application in combination with Figures 1 to 9. Next, the structure of the model training device and computing equipment provided in the embodiment of the present application is introduced in combination with the accompanying drawings.

[0159] Referring to FIG10 , a schematic diagram of the structure of a model training device is shown. The model training device 1000 shown in FIG10 is applied to a distributed training system, which includes multiple computing nodes. The multiple computing nodes are used to perform distributed training on an artificial intelligence (AI) model. The AI ​​model includes a forward communication operator. During the forward computation process, data is transmitted between the multiple computing nodes by executing the forward communication operator.

[0160] As shown in FIG10 , the model training device 1000 includes:

[0161] An acquisition unit 1001 is used to acquire the recalculation layer of the AI ​​model during the reverse calculation process and to acquire the forward communication operator in the recalculation layer;

[0162] a removal unit 1002, configured to remove the forward communication operator from the recalculation layer;

[0163] The reverse calculation unit 1003 is configured to perform reverse calculation using a recalculation layer with the forward communication operator removed.

[0164] In a possible implementation, the forward communication operator is a terminal forward communication operator in the recomputation layer.

[0165] In a possible implementation, the forward communication operators are all forward communication operators in the recomputation layer.

[0166] In one possible implementation, the recomputation layer includes multiple forward communication operators;

[0167] Then, the removal unit 1002 is specifically configured to divide the recomputation layer into a plurality of sub-recomputation layers, each of the plurality of sub-recomputation layers including a forward communication operator; and remove the forward communication operator included in each of the plurality of sub-recomputation layers.

[0168] The reverse calculation unit 1003 is specifically configured to perform reverse calculation using each of the multiple sub-recalculation layers with the forward communication operator removed.

[0169] In a possible implementation, the model training device 1000 further includes:

[0170] The storage unit 1004 is used to store input data of each sub-computation layer in the plurality of sub-computation layers during the forward calculation process, and the input data is used by the sub-computation layer to perform reverse calculation.

[0171] In one possible implementation, the model training device 1000 is applied to a first computing node among a plurality of computing nodes, and the first computing node performs reverse calculation using a recalculation layer that removes a forward communication operator;

[0172] Then, the model training device 1000 further includes:

[0173] The estimation unit 1005 is configured to estimate whether the amount of storage resources of the first computing node meets the recomputation layer division condition before dividing the recomputation layer into multiple sub-recomputation layers;

[0174] When dividing the recomputation layer into multiple sub-recomputation layers, the removing unit 102 is specifically configured to divide the recomputation layer into multiple sub-recomputation layers when the amount of storage resources of the first computing node meets the recomputation layer division condition.

[0175] In a possible implementation, the model training device 1000 further includes:

[0176] A determining unit 1006 is configured to determine a reverse communication operator corresponding to the forward communication operator based on the mapping relationship and the forward communication operator;

[0177] The reverse calculation unit 1003 is specifically configured to:

[0178] Execute the recalculation layer that removes the forward communication operator to obtain the activation value;

[0179] Execute the reverse communication operator to obtain the gradient used to update the parameters in the recalculation layer;

[0180] Based on the activation value and gradient, update the parameters in the recalculation layer.

[0181] In a possible implementation, the reverse calculation unit 1003 is specifically configured to execute the reverse communication operator during a process of executing a recalculation layer that removes the forward communication operator.

[0182] In one possible implementation, the recomputation layer includes a computation module, a forward communication operator, and a post-processing module, wherein the forward communication operator is executed before the post-processing module and after the computation module in the recomputation layer.

[0183] The model training device 1000 further includes:

[0184] The filling unit 1007 is configured to fill the output data of the calculation module according to the second matrix dimension when the first matrix dimension of the output data of the calculation module does not match the second matrix dimension of the input data of the post-processing module.

[0185] Since the model training device 1000 shown in Figure 10 corresponds to the computing node 101 in the embodiments shown in Figures 2, 5 and 7 above, the specific implementation method of the model training device 1000 shown in Figure 10 and its technical effects can be found in the relevant descriptions in the embodiments shown in Figures 2, 5 and 7 above, and will not be repeated here.

[0186] FIG11 is a schematic diagram of the hardware structure of a computing device 1100 provided in the present application. The computing device 1100 may, for example, implement the computing node 101 in the embodiments shown in FIG2 , FIG5 and FIG7 .

[0187] As shown in Figure 11, the computing device 1100 includes a processor 1101, a memory 1102, and a communication interface 1103. The processor 1101, the memory 1102, and the communication interface 1103 communicate via a bus 1104, and may also communicate via other means such as wireless transmission. The memory 1102 is used to store instructions, and the processor 1101 is used to execute the instructions stored in the memory 1102. Furthermore, the computing device 1100 may also include a memory unit 1105, and the memory unit 1105 may be connected to the processor 1101, the storage medium 1102, and the communication interface 1103 via the bus 1104. The memory 1102 stores program code, and the processor 1101 may use the program code stored in the memory 1102 to perform the following operations:

[0188] Obtaining a recomputation layer of the AI ​​model during a reverse computation process, and obtaining a forward communication operator in the recomputation layer; wherein the distributed training system includes multiple computing nodes for performing distributed training on the AI ​​model, and, during the forward computation process, transmitting data between the multiple computing nodes by executing the forward communication operator;

[0189] removing the forward communication operator from the recomputation layer;

[0190] A reverse calculation is performed using a recalculation layer that removes the forward communication operator.

[0191] It should be understood that in this embodiment, the processor 1101 may be a CPU, or may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0192] The memory 1102 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1101. The memory 1102 may also include a nonvolatile random access memory.

[0193] The memory 1102 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. The nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0194] The communication interface 1103 is used to communicate with other devices connected to the computing device 1100. The bus 1104 may include, in addition to the data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all buses are labeled as bus 1104 in the figure.

[0195] It should be understood that the computing device 1100 according to the embodiment of the present application may correspond to the model training device 1000 in the embodiment of the present application, and may correspond to executing the method executed by the computing node 101 in the method shown in Figures 2, 5 and 7 in the embodiment of the present application. The above-mentioned and other operations and / or functions implemented by the computing device 1100 are respectively for implementing the processes of the corresponding methods in Figures 2, 5 and 7. For the sake of brevity, they will not be repeated here.

[0196] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned model training method.

[0197] The present application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the computer program product fully or partially generates the process or function described in the present application.

[0198] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0199] The computer program product may be a software installation package, which may be downloaded and executed on a computing device when any of the aforementioned model training methods is required.

[0200] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0201] The terms used in the above embodiments are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification of this application and the appended claims, the singular expressions "a", "an", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless the context clearly indicates otherwise. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; the character " / " generally indicates that the objects before and after are in an "or" relationship. In the embodiments of the present application. "Simultaneously" refers to the same time period, including the situation at the same moment.

[0202] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0203] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A model training method, characterized in that: The method is applied to a distributed training system, the distributed training system including multiple computing nodes, the multiple computing nodes being used to perform distributed training on an artificial intelligence (AI) model, the AI ​​model including a forward communication operator, and during a forward computation process, data is transmitted between the multiple computing nodes by executing the forward communication operator. The method includes: Obtaining a recalculation layer of the AI ​​model during a reverse calculation process, and obtaining a forward communication operator in the recalculation layer; removing the forward communication operator from the recomputation layer; A reverse calculation is performed using a recalculation layer that removes the forward communication operator.

2. The method according to claim 1, characterized in that The forward communication operator is a terminal forward communication operator in the recalculation layer.

3. The method according to claim 1, characterized in that The forward communication operators are all forward communication operators in the recalculation layer.

4. The method according to claim 3, characterized in that The recalculation layer includes a plurality of forward communication operators; The removing the forward communication operator from the recalculation layer comprises: Dividing the recomputation layer into a plurality of sub-recomputation layers, each of the plurality of sub-recomputation layers comprising a forward communication operator; removing a forward communication operator included in each sub-computation layer of the plurality of sub-computation layers; The performing reverse calculation by using the recalculation layer that removes the forward communication operator includes: Reverse calculation is performed using each of the plurality of sub-computation layers with the forward communication operator removed.

5. The method according to claim 4, characterized in that The method further comprises: During the forward calculation process, input data of each sub-computation layer in the plurality of sub-computation layers is saved, and the input data is used for the sub-computation layer to perform reverse calculation.

6. The method according to claim 4 or 5, characterized in that A first computing node among the plurality of computing nodes performs reverse computing using the recomputation layer from which the forward communication operator is removed. Therefore, before dividing the recomputation layer into a plurality of sub-recomputation layers, the method further includes: estimating whether the quantity of storage resources of the first computing node meets the recomputing layer division condition; The dividing the recomputation layer into a plurality of sub-recomputation layers comprises: When the amount of storage resources of the first computing node meets the recomputation layer division condition, the recomputation layer is divided into multiple sub-recomputation layers.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Determining a reverse communication operator corresponding to the forward communication operator according to the mapping relationship and the forward communication operator; The performing reverse calculation by using the recalculation layer that removes the forward communication operator includes: executing the recalculation layer removing the forward communication operator to obtain an activation value; Executing the reverse communication operator to obtain a gradient for updating parameters in the recalculation layer; Update parameters in the recalculation layer according to the activation value and the gradient.

8. The method according to claim 7, characterized in that During the process of removing the recalculation layer of the forward communication operator, the reverse communication operator is executed.

9. The method according to any one of claims 1 to 8, characterized in that The recalculation layer includes a calculation module, the forward communication operator and a post-processing module, and the execution order of the forward communication operator in the recalculation layer is before the execution order of the post-processing module and after the execution order of the calculation module; The method further comprises: When a first matrix dimension of the output data of the calculation module does not match a second matrix dimension of the input data of the post-processing module, the output data of the calculation module is padded according to the second matrix dimension.

10. A model training device, characterized in that: The model training device is applied to a distributed training system, which includes multiple computing nodes. The multiple computing nodes are used to perform distributed training on an artificial intelligence (AI) model. The AI ​​model includes a forward communication operator. During the forward calculation process, data is transmitted between the multiple computing nodes by executing the forward communication operator. The model training device comprises: An acquisition unit, configured to acquire a recalculation layer of the AI ​​model during a reverse calculation process, and to acquire a forward communication operator in the recalculation layer; a removal unit, configured to remove the forward communication operator from the recalculation layer; A reverse calculation unit is used to perform reverse calculation by using a recalculation layer that removes the forward communication operator.

11. The device according to claim 10, characterized in that The forward communication operator is a terminal forward communication operator in the recalculation layer.

12. The device according to claim 10, characterized in that The forward communication operators are all forward communication operators in the recalculation layer.

13. The device according to claim 12, characterized in that The recalculation layer includes a plurality of forward communication operators; The removal unit is specifically configured to divide the recomputation layer into a plurality of sub-recomputation layers, each of the plurality of sub-recomputation layers comprising a forward communication operator; and remove the forward communication operator included in each of the plurality of sub-recomputation layers; The reverse calculation unit is specifically configured to perform reverse calculation using each sub-computation layer of the multiple sub-computation layers with the forward communication operator removed.

14. The device according to claim 13, characterized in that The model training device also includes: The storage unit is used to store input data of each sub-computation layer in the multiple sub-computation layers during the forward calculation process, and the input data is used for the sub-computation layer to perform reverse calculation.

15. The device according to claim 13 or 14, characterized in that The model training device is applied to a first computing node among the multiple computing nodes, and the first computing node performs reverse calculation using the recalculation layer with the forward communication operator removed; Then, the model training device further includes: an estimating unit, configured to estimate whether the amount of storage resources of the first computing node meets a recomputation layer division condition before dividing the recomputation layer into a plurality of sub-recomputation layers; When dividing the recomputation layer into multiple sub-recomputation layers, the removal unit is specifically configured to divide the recomputation layer into multiple sub-recomputation layers when the quantity of storage resources of the first computing node meets the recomputation layer division condition.

16. The device according to any one of claims 10 to 15, characterized in that The model training device also includes: A determining unit, configured to determine a reverse communication operator corresponding to the forward communication operator according to the mapping relationship and the forward communication operator; The reverse calculation unit is specifically used to: executing the recalculation layer removing the forward communication operator to obtain an activation value; Executing the reverse communication operator to obtain a gradient for updating parameters in the recalculation layer; Update parameters in the recalculation layer according to the activation value and the gradient.

17. The device according to claim 16, characterized in that The reverse calculation unit is specifically configured to execute the reverse communication operator during the process of removing the recalculation layer of the forward communication operator.

18. The device according to any one of claims 10 to 17, characterized in that The recalculation layer includes a calculation module, the forward communication operator and a post-processing module, and the execution order of the forward communication operator in the recalculation layer is before the execution order of the post-processing module and after the execution order of the calculation module; The model training device also includes: A filling unit is used to fill the output data of the calculation module according to the second matrix dimension when the first matrix dimension of the output data of the calculation module does not match the second matrix dimension of the input data of the post-processing module.

19. A distributed training system, characterized in that: The distributed training system includes multiple computing nodes, which are used to perform distributed training on an artificial intelligence (AI) model. The AI ​​model includes a forward communication operator. During the forward calculation process, data is transmitted between the multiple computing nodes by executing the forward communication operator. Each of the multiple computing nodes is used to execute the steps of the method as described in any one of claims 1 to 9.

20. A computer program product comprising instructions, characterized in that When the method is executed on at least one computing device, the method enables the at least one computing device to execute the method according to any one of claims 1 to 9.

Images