AI model training method and related equipment

Abstract

The invention provides an AI model training method and related equipment, and the method comprises the steps: dividing a training task of an AI model into a first training task and a second training task which are independent from each other, dividing the first training task into a plurality of first sub-tasks, dividing the second training task into a plurality of second sub-tasks, and dividing the first sub-tasks into a plurality of first sub-tasks and a plurality of second sub-tasks; each sub-task is a calculation sub-task or a communication sub-task, so that the calculation sub-tasks in the plurality of first sub-tasks and the communication sub-tasks in the plurality of second sub-tasks are mutually independent, and / or the communication sub-tasks in the plurality of first sub-tasks and the calculation sub-tasks in the plurality of second sub-tasks are mutually independent; and then a plurality of computing nodes in the distributed training system execute mutually independent computing sub-tasks and communication sub-tasks in parallel in the process of executing the plurality of first sub-tasks and the plurality of second sub-tasks to train the AI model, and the execution time consumption of the communication sub-tasks is masked, so that the training time consumption of the AI model is shortened, and the training efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligent (AI), and in particular, relates to an AI model training method and related equipment. BACKGROUND

[0002] With the development of AI technology, AI models (also known as neural networks) have been widely applied in fields such as computer vision (CV), natural language processing (NLP), graph learning, and recommendation systems. Before being put into use, AI models usually need to be trained.

[0003] Currently, in order to efficiently and accurately train AI models, distributed training is usually used. Distributed training refers to using multiple computing nodes in a distributed training system to jointly perform the training task of an AI model to accelerate the training process and reduce training time consumption. Specifically, the process of multiple computing nodes jointly performing the training task of an AI model is as follows: the training task of an AI model is split into multiple subtasks with a sequential execution order according to execution stages (including a computation stage and a communication stage), the subtasks corresponding to the computation stage are computation subtasks, and the subtasks corresponding to the communication stage are communication subtasks. Then, in the forward propagation process of each iteration of AI model training, multiple computing nodes jointly execute each subtask according to the sequential execution order of the multiple subtasks, and in the backward propagation process of each iteration of AI model training, multiple computing nodes jointly execute each subtask according to the reverse order of the sequential execution order of the multiple subtasks.

[0004] However, the above-mentioned AI model training process has the problems of long training time consumption and low training efficiency. SUMMARY

[0005] The present application provides an AI model training method and related equipment for shortening the training time consumption of an AI model and improving the training efficiency.

[0006] In a first aspect, an AI model training method is provided, which is applied to a distributed training system including multiple computing nodes, the training task of an AI model is split into a first training task and a second training task which are independent of each other, the first training task is split into multiple first subtasks, the second training task is split into multiple second subtasks, and the multiple computing nodes jointly execute each subtask in the multiple first subtasks and the multiple second subtasks, each subtask being a computation subtask or a communication subtask. The method can include the following steps:

[0007] The plurality of computing nodes obtain the plurality of first sub-tasks and the plurality of second sub-tasks, then execute the plurality of first sub-tasks and the plurality of second sub-tasks, and perform the first computing sub-task and the first communication sub-task in parallel in the process of executing the plurality of first sub-tasks and the plurality of second sub-tasks, wherein the first computing sub-task and the first communication sub-task are independent of each other, and the first computing sub-task and the first communication sub-task belong to the plurality of first sub-tasks and the plurality of second sub-tasks.

[0008] In the above scheme, by splitting the training task of the AI model into the first training task and the second training task which are independent of each other, then splitting the first training task into the plurality of first sub-tasks and splitting the second training task into the plurality of second sub-tasks, and each sub-task in the plurality of first sub-tasks and the plurality of second sub-tasks being a computing sub-task or a communication sub-task, it can be understood that, since the first training task and the second training task are independent of each other, each first sub-task in the plurality of first sub-tasks and each second sub-task in the plurality of second sub-tasks are independent of each other, so that the computing sub-task in the plurality of first sub-tasks and the communication sub-task in the plurality of second sub-tasks are independent of each other, and / or the communication sub-task in the plurality of first sub-tasks and the computing sub-task in the plurality of second sub-tasks are independent of each other, then by using the characteristic that there are independent computing sub-tasks and communication sub-tasks in the plurality of first sub-tasks and the plurality of second sub-tasks, the plurality of computing nodes in the distributed training system perform the independent computing sub-tasks and communication sub-tasks (i.e., the above independent first computing sub-task and first communication sub-task) in parallel in the process of training the AI model by executing the plurality of first sub-tasks and the plurality of second sub-tasks, the execution time of the communication sub-task is masked, thereby shortening the training time of the AI model and improving the training efficiency.

[0009] In a possible implementation, the AI model includes a first sub-AI model and a second sub-AI model, the first training task is used for training the first sub-AI model, and the second training task is used for training the second sub-AI model.

[0010] Implementing the above implementation can shorten the training time of the AI model including two sub-AI models and improve the model training efficiency.

[0011] In a possible implementation, the AI model is a wide & deep model.

[0012] Implementing the above implementation can shorten the training time of the wide & deep model and improve the model training efficiency.

[0013] In a possible implementation, the first training task is a training task for an odd-indexed embdding vector in an embdding vector table corresponding to the AI model, and the second training task is a training task for an even-indexed embdding vector in the embdding vector table corresponding to the AI model.

[0014] In a possible implementation, the AI model is a wide model or a deep model.

[0015] The implementation of the above implementation can shorten the training time of the wide model, the deep model, and other AI models, and improve the model training efficiency.

[0016] In a possible implementation, the computing node is an accelerator, and the accelerator is a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or other special AI processing chips.

[0017] In a second aspect, an AI model training method is provided, applied to a management node connected to a distributed training system, the distributed training system including a plurality of computing nodes that collectively perform a training task of an AI model. The method includes the following steps:

[0018] The management node splits the training task of the AI model into a first training task and a second training task that are independent of each other.

[0019] The management node splits the first training task into a plurality of first subtasks and splits the second training task into a plurality of second subtasks, each of the plurality of first subtasks and the plurality of second subtasks being a computing subtask or a communication subtask.

[0020] The management node instructs the distributed training system to perform the method provided in the first aspect or any possible implementation of the first aspect.

[0021] In a possible implementation, the AI model includes a first sub-AI model and a second sub-AI model, the first training task is used to train the first sub-AI model, and the second training task is used to train the second sub-AI model.

[0022] In a possible implementation, the AI model is a wide & deep model.

[0023] In a possible implementation, the first training task is a training task for an odd-indexed embedding vector in an embedding vector table corresponding to the AI model, and the second training task is a training task for an even-indexed embedding vector in the embedding vector table corresponding to the AI model.

[0024] In a possible implementation, the AI model is a wide model or a deep model.

[0025] In a possible implementation, the computing node is an accelerator, and the accelerator is a GPU, an NPU, a TPU, or another special AI processing chip.

[0026] In a third aspect, a management node is provided, which is connected to a distributed training system including a plurality of computing nodes that collectively perform a training task of an AI model. The management node specifically includes:

[0027] a splitting unit configured to split the training task of the AI model into a first training task and a second training task that are independent of each other;

[0028] The splitting unit is further configured to split the first training task into a plurality of first subtasks and split the second training task into a plurality of second subtasks, each of the plurality of first subtasks and the plurality of second subtasks being a computing subtask or a communication subtask.

[0029] a notification unit configured to instruct the distributed training system to perform the method provided in the first aspect or any possible implementation of the first aspect.

[0030] In a fourth aspect, a management node is provided, which includes a processor and a memory. The processor is configured to execute instructions stored in the memory, so that the management node implements the method provided in the second aspect or any possible implementation of the second aspect.

[0031] In a fifth aspect, a computer-readable storage medium is provided, which stores instructions for implementing the method provided in the first aspect, the second aspect, or any possible implementation of the first aspect or the second aspect.

[0032] In a sixth aspect, a computing device cluster is provided, the computing device cluster comprising at least one computing device, each of the at least one computing device comprising a processor and a memory, the processor of the at least one computing device configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to implement the method provided in the first aspect, the second aspect, or any possible implementation of the first aspect, any possible implementation of the second aspect.

[0033] In a seventh aspect, a computer program product is provided, comprising a computer program which, when executed by a computing device cluster, causes the computing device cluster to carry out the method provided in the first aspect, the second aspect, or any possible implementation of the first aspect, any possible implementation of the second aspect.

[0034] In an eighth aspect, a computing system is provided, the computing system comprising a management node configured to implement the method provided in the second aspect, or any possible implementation of the second aspect, and a distributed training system configured to implement the method provided in the first aspect, or any possible implementation of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a structure diagram of a distributed training system related to the present application;

[0036] Figure 2 is a training process diagram of a wide&deep model related to the present application;

[0037] Figure 3 is a training process diagram of a wide model related to the present application;

[0038] Figure 4 is a flow diagram of a training method of an AI model provided by the present application;

[0039] Figure 5 is a diagram of splitting a training task of a wide&deep model provided by the present application;

[0040] Figure 6 is a diagram of splitting a training task of a wide model provided by the present application;

[0041] Figure 7A is a training process diagram of a wide&deep model provided by the present application;

[0042] Figure 7B is another training process diagram of a wide&deep model provided by the present application;

[0043] Figure 8A This is a schematic diagram of the training process of a wide model provided in this application;

[0044] Figure 8B This is a schematic diagram of the training process of another wide model provided in this application;

[0045] Figure 9 This is a schematic diagram of the structure of a management node provided in this application;

[0046] Figure 10 This is a schematic diagram of another management node structure provided in this application;

[0047] Figure 11 This is a schematic diagram of the structure of a computing device cluster provided in this application;

[0048] Figure 12 This is a schematic diagram of the structure of a computing system provided in this application. Detailed Implementation

[0049] This application involves distributed training scenarios for AI models, which will be discussed below. Figure 1 The distributed training system shown provides a detailed description of this scenario.

[0050] like Figure 1 As shown, the distributed training system 100 includes an AI model 110 and multiple computing nodes 120. Figure 1 In this example, multiple computing nodes 120, including computing nodes 1201 to 1209, are used.

[0051] AI model 110 can be a model used to implement any AI business, such as a model used to implement face recognition, fingerprint recognition, object recognition, natural language translation, recommendation, etc. Examples include convolutional neural networks (CNN), large language models, wide models, deep models, wide&deep models, deep&cross networks, etc. This application does not specifically limit AI model 110. Among them, a wide model is a linear model with memorization capabilities; a deep model is a deep neural network (DNN) with generalization capabilities; a wide&deep model is a model combining wide and deep models, possessing both the memorization capabilities of a wide model and the generalization capabilities of a deep model; and a deep&cross network is a model combining deep models and cross networks.

[0052] The plurality of computing nodes 120 can be located in the same computing device or in different computing devices. Figure 1 For example, the plurality of computing nodes 120 are located in three computing devices, each of the first computing device to the third computing device includes 3 computing nodes, the first computing device includes the computing node 1201, the computing node 1202 and the computing node 1203, the second computing device includes the computing node 1204, the computing node 1205 and the computing node 1206, and the third computing device includes the computing node 1207, the computing node 1208 and the computing node 1209. The computing device can be a server of a cloud data center, an edge server or a server of a local data center, the server is a heterogeneous server including a central processing unit (CPU) and the computing node 120; the computing node 120 can be an accelerator, and the accelerator can be a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU) or other special AI processing chip, etc., which is not limited in the present application.

[0053] The plurality of computing nodes 120 are collectively used for deploying the AI model 110 and collectively performing a training task of the AI model 110, and the deployment manner of the plurality of computing nodes 120 can be pipeline parallelism (PP), tensor parallelism (TP), combination of pipeline parallelism and tensor parallelism, etc., which is not limited in the present application. The pipeline parallelism is a strategy of distributing different network layers of the AI model 110 to different computing nodes for parallel computing, each computing node is responsible for training part of the network layers of the entire AI model 110, and then the intermediate result is transmitted to the next computing node to realize parallel computing of the AI model 110; the tensor parallelism is a strategy of decomposing large tensor operations (such as large matrix multiplication or deep convolution operation) in the AI model 110 and distributing to multiple computing nodes for parallel computing.

[0054] At present, Figure 1The process of training the AI model 110 by the illustrated distributed training system 100 is generally as follows: the distributed training system 100 obtains a plurality of sub-tasks with a sequential execution order, which are split from a training task of the AI model 110 according to execution stages (including computation stages and communication stages), a sub-task corresponding to a computation stage in the plurality of sub-tasks is a computation sub-task, and a sub-task corresponding to a communication stage in the plurality of sub-tasks is a communication sub-task, then, in a forward propagation process in each iteration of training the AI model 110, the plurality of computing nodes 120 in the distributed training system 100 jointly execute each sub-task according to the sequential execution order of the plurality of sub-tasks, and in a backward propagation process in each iteration of training the AI model 110, the plurality of computing nodes 120 jointly execute each sub-task according to an inverse order of the sequential execution order of the plurality of sub-tasks.

[0055] The process of training the AI model 110 by the distributed training system 100 is described in detail below by taking the AI model 110 as a wide&deep model and a wide model respectively as examples.

[0056] Example 1: the AI model 110 is a wide&deep model.

[0057] Since the forward propagation process in each iteration of training the wide&deep model includes two all2all communication stages, two all2all communication pre-computation stages, and two all2all communication post-computation stages, the first all2all communication stage in the two all2all communication stages is an all2all communication operation in the training process of the wide model, the second all2all communication stage is an all2all communication operation in the training process of the deep model, the first all2all communication pre-computation stage in the two all2all communication pre-computation stages is a computation operation for the wide model before the execution of the first all2all communication stage, the second all2all communication pre-computation stage is a computation operation for the deep model before the execution of the second all2all communication stage, the first all2all communication post-computation stage in the two all2all communication pre-computation stages is a computation operation for the wide model after the execution of the first all2all communication stage, and the second all2all communication post-computation stage is a computation operation for the deep model after the execution of the second all2all communication stage, and the backward propagation process in each iteration of training the wide&deep model includes a plurality of execution stages, which are the same as the plurality of execution stages included in the forward propagation process and have an inverse order, wherein all2all refers to global information exchange, for example, each computing node in the plurality of computing nodes participating in the training of the wide&deep model transmits its own computation data to other computing nodes, and therefore, for example, Figure 2As shown, the training task of the wide&deep model is split into multiple subtasks according to the execution stage (including the calculation stage and the communication stage) as follows: wide pre, wideall2all, wide post, deeppre, deepall2all, deeppost, and the execution order of the multiple subtasks is: wide pre, wideall2all, wide post, deeppre, deepall2all, deeppost, wherein the subtask "wideall2all" is used to implement the first all2all communication stage, the subtask "wide pre" is used to implement the calculation stage before the first all2all communication, the subtask "wide post" is used to implement the calculation stage after the first all2all communication, the subtask "deepall2all" is used to implement the second all2all communication stage, the subtask "deeppre" is used to implement the calculation stage before the second all2all communication, and the subtask "deeppost" is used to implement the calculation stage after the second all2all communication.

[0058] Since in the multiple subtasks split from the training task of the wide&deep model, wide pre, wide post, deeppre, and deeppost are all calculation subtasks (i.e., subtasks of the calculation type), wideall2all and deepall2all are both communication subtasks (i.e., subtasks of the communication type), wideall2all depends on wide pre, wide post depends on wideall2all, deepall2all depends on deeppre, and deeppost depends on deepall2all, and moreover, the multiple computing nodes 120 only jointly execute one calculation subtask in the same time period and only jointly execute one communication subtask in the same time period, therefore, as shown, Figure 2 the process of the multiple computing nodes 120 training the wide&deep model is: in the forward propagation process in each iteration of the wide&deep model training, the multiple computing nodes 120 jointly execute wide pre, wideall2all, wide post, deeppre, deepall2all, and deeppost according to the execution order of the multiple subtasks, and in the backward propagation process in each iteration of the wide&deep model training, the multiple computing nodes jointly execute deeppost, deepall2all, deeppre, wide post, wideall2all, and wide pre in the reverse order of the execution order of the multiple subtasks.

[0059] However, the training process of the wide & deep model has problems of long training time and low training efficiency.

[0060] Example 2, the AI model 110 is a wide model.

[0061] Since the forward propagation process in each iteration of the wide model training includes an all2all communication phase, an all2all communication pre-computation phase and an all2all communication post-computation phase, the all2all communication phase is an all2all communication operation in the wide model training process, the all2all communication pre-computation phase is a computation operation to be performed for the wide model before the execution of the all2all communication phase, and the all2all communication post-computation phase is a computation operation to be performed for the wide model after the execution of the all2all communication phase, and the back propagation process in each iteration of the wide model training includes multiple execution phases, which are the same as the multiple execution phases included in the forward propagation process and in reverse order, therefore, as shown in Figure 3 , the training task of the wide model is split into multiple sub-tasks: wide pre, wide all2all, and wide post, and the execution order of the multiple sub-tasks is: wide pre, wide all2all, and wide post. For more information about wide pre, wide all2all, and wide post, please refer to Figure 2 .

[0062] Since in the multiple sub-tasks split from the training task of the wide model, wide pre and wide post are both computation sub-tasks, and wide all2all is a communication sub-task, wide all2all depends on wide pre, and wide post depends on wide all2all, and moreover, the multiple computing nodes 120 only jointly execute one computation sub-task at the same time period, therefore, as shown in Figure 3 , the process of the multiple computing nodes 120 training the wide model is: in the forward propagation process in each iteration of the wide model training, the multiple computing nodes 120 jointly execute wide pre, wide all2all, and wide post according to the execution order of the multiple sub-tasks, and in the back propagation process in each iteration of the wide model training, the multiple computing nodes 120 jointly execute wide post, wide all2all, and wide pre in reverse order of the execution order of the multiple sub-tasks.

[0063] However, the training process of the wide model has problems of long training time and low training efficiency.

[0064] It should be understood that, Figure 3 The wide model and Figure 2 The wide & deep model are only two examples of the AI model 110, and in actual implementation, the AI model 110 can be other non-mixed models similar to the wide model, such as a deep model, a CNN, etc., or other mixed models containing multiple sub-AI model structures similar to the wide & deep model, such as a deep & cross network, and the present application does not specifically limit the AI model 110. In actual scenarios, when the AI model 110 is a non-mixed model similar to the wide model (such as a deep model, a CNN, etc.), the training process of the AI model 110 is generally similar to that of the wide model shown in Figure 3 When the AI model is other mixed models containing multiple sub-AI model structures similar to the wide & deep model (such as a deep & cross network), the training process of the AI model 110 is generally similar to that of the wide & deep model shown in Figure 2 Therefore, the training process of the AI model 110 generally has the problems of long training time and low training efficiency.

[0065] To solve the problems of long training time and low training efficiency of the AI model, the present application provides an AI model training method. The training task of the AI model is divided into a first training task and a second training task which are independent of each other, and then the first training task is divided into a plurality of first sub-tasks and the second training task is divided into a plurality of second sub-tasks. Each sub-task in the plurality of first sub-tasks and the plurality of second sub-tasks is a computing sub-task or a communication sub-task. It can be understood that since the first training task and the second training task are independent of each other, each first sub-task in the plurality of first sub-tasks and each second sub-task in the plurality of second sub-tasks are independent of each other, so that the computing sub-tasks in the plurality of first sub-tasks and the communication sub-tasks in the plurality of second sub-tasks are independent of each other, and / or the communication sub-tasks in the plurality of first sub-tasks and the computing sub-tasks in the plurality of second sub-tasks are independent of each other. Then, by using the characteristics that there are independent computing sub-tasks and communication sub-tasks in the plurality of first sub-tasks and the plurality of second sub-tasks, the plurality of computing nodes in the distributed training system perform the independent computing sub-tasks and communication sub-tasks in parallel in the process of training the AI model by the plurality of first sub-tasks and the plurality of second sub-tasks, the execution time of the communication sub-tasks is masked, thereby shortening the training time of the AI model and improving the training efficiency.

[0066] To facilitate a clearer understanding of the AI model training method provided by the present application, the following will be described in detail in conjunction with the flowchart of the AI model training method provided by the present application shown in Figure 4 The present application provides an AI model training method.

[0067] like Figure 4 As shown, it includes the following steps:

[0068] S401: The management node splits the training task of the AI ​​model into two independent training tasks: the first training task and the second training task.

[0069] The management node can be a computing device deployed independently of the distributed training system and specifically responsible for task decomposition, or it can be a computing device within the distributed training system; this application does not impose specific limitations. For information on distributed training systems and computing devices, please refer to [link to relevant documentation]. Figure 1 The section on distributed training systems and computing devices will not be repeated here.

[0070] The first training task and the second training task are independent of each other, meaning that the first training task and the second training task do not depend on each other, their execution processes do not affect each other, and the two tasks can be executed in parallel.

[0071] As can be seen from the above introduction to AI models, an AI model may be a hybrid model containing multiple sub-AI models, such as the wide & deep model, deep & cross network, etc., or it may be a non-hybrid model, such as the wide model, deep model, CNN, etc. The following will introduce S401 in detail with these two types of AI models.

[0072] (a) AI models are hybrid models, such as wide & deep models, deep & cross networks, etc.

[0073] For ease of description, the multiple sub-AI models in the hybrid model are referred to as the first sub-AI model, the second sub-AI model, and so on. Taking the wide&deep hybrid model as an example, the wide model in the wide&deep model is the first sub-AI model, and the deep model in the wide&deep model is the second sub-AI model. Similarly, taking the deep&cross network hybrid model as an example, the deep model in the deep&cross network is the first sub-AI model, and the cross network in the deep&cross network is the second sub-AI model. It should be noted that, for ease of description, in the following embodiments, the AI ​​model training method provided in this application will be described using the example of a hybrid model containing both a first sub-AI model and a second sub-AI model.

[0074] When the AI ​​model is a hybrid model comprising a first sub-AI model and a second sub-AI model, the training task of the AI ​​model can be split into two independent training tasks: a first training task and a second training task. The first training task is for training the first sub-AI model, and the second training task is for training the second sub-AI model. Taking the hybrid model Wide & Deep as an example...Figure 5 As shown in FIG. 8, the training task of the wide&deep model can be split into the first training task and the second training task which are independent of each other, wherein the first training task is the training task of the sub-AI model wide in the wide&deep model, and the second training task is the training task of the sub-AI model deep in the wide&deep model.

[0075] (II) The AI model is a non-mixed model, such as a wide model, a deep model, a CNN, etc.

[0076] When the AI model is a non-mixed model, the training task of the AI model can be split into the first training task and the second training task which are independent of each other, wherein the first training task is the training task for the odd table of the embedding corresponding to the AI model, and the second training task is the training task for the even table of the embedding corresponding to the AI model. Taking the non-mixed model wide as an example, as shown in FIG. 9, the training task of the wide model can be split into the first training task and the second training task which are independent of each other, wherein the first training task is the training task for the odd table of the embedding corresponding to the wide model, and the second training task is the training task for the even table of the embedding corresponding to the wide model. Figure 6

[0077] ​wherein, the embdding odd table and the embdding even table, in actual scenarios, the training task of the AI model is used not only for training the AI model itself, but also for training the embdding vector table corresponding to the AI model. The embdding vector table corresponding to the AI model refers to a table containing a large number of embdding vectors. In the table, each row usually represents an entity (such as a word, a user ID, a commodity ID, etc.), and the columns are the dimensions of the embdding vectors. For example, if there is a vocabulary table containing a large number of words, each word in the vocabulary table has a 300-dimensional embdding vector, and the 300-dimensional vector table corresponding to all the words in the entire vocabulary table is an embdding vector table. The size of the embdding vector table is the number of words in the vocabulary table * 300. Before starting to train the AI model, the embdding vectors in the embdding vector table are usually initialized with random values, and then by training the embdding vector table, the AI model can learn the contextual relationship and semantic features of each entity (such as words, user IDs, commodity IDs, etc.) during the training process. Each embdding vector in the embdding vector table corresponds to a unique index, which is used to represent and quickly access the embdding vector corresponding to the index. Therefore, the embdding vector table can be split into two tables: the embdding odd table and the embdding even table according to the index corresponding to the embdding vector. The embdding odd table includes embdding vectors with odd indexes in the embdding vector table, and the embdding even table includes embdding vectors with even indexes in the embdding vector table.

[0078] It should be understood that the above method of splitting the training task of the AI model into the first training task and the second training task when the AI model is a hybrid model or a non-hybrid model is only an example. In actual scenarios, the splitting can be performed in other ways, which are not limited in the present application.

[0079] S402: The management node splits the first training task into a plurality of first sub-tasks, and splits the second training task into a plurality of second sub-tasks. Each first sub-task and each second sub-task is a computing sub-task or a communication sub-task.

[0080] Wherein, the communication sub-task is an all2all or allreduce type task, which is not limited in the present application. Wherein, allreduce refers to sharing and aggregating data among all participating computing nodes. The aggregation operation can be in various forms, such as summation, averaging, maximum value, etc.

[0081] It can be understood that, since the first training task and the second training task are independent of each other, the plurality of first sub-tasks split from the first training task and the plurality of second sub-tasks split from the second training task are also independent of each other.

[0082] The implementation process of S402 will be described below in combination with two specific examples.

[0083] Example 1, taking the first training task shown in Figure 5 as the training task of the sub-AI model wide in the wide&deep model, and the second training task as the training task of the sub-AI model deep in the wide&deep model, as shown in Figure 5 , the first training task can be split into the following plurality of first sub-tasks: wide pre, wide all2all, wide post, and the second training task can be split into the following plurality of second sub-tasks: deep pre, deep all2all, deep post. For the meanings of these sub-tasks, please refer to the related description, which will not be expanded here for the sake of brevity of the description. Figure 2

[0084] Since the first training task and the second training task are independent of each other, each of the plurality of first sub-tasks (including wide pre, wide all2all, and wide post) is independent of each of the plurality of second sub-tasks (including deep pre, deep all2all, and deep post).

[0085] Example 2, taking the first training task shown in Figure 6 as the training task for the odd embdding table corresponding to the wide model, and the second training task as the training task for the even embdding table corresponding to the wide model, as shown in Figure 6 ​As shown, the first training task can be split into a plurality of first sub-tasks: odd table pre, odd table all2all, odd table post, and the second training task can be split into a plurality of second sub-tasks: even table pre, even table all2all, even table post. Among them, the sub-task "odd table all2all" is used to implement the all2all communication operation in the training process of the embdding odd table corresponding to the wide model, the sub-task "odd table pre" is the calculation operation that needs to be performed for the embdding odd table corresponding to the wide model before the execution of the sub-task "odd table all2all", the sub-task "odd table post" is the calculation operation that needs to be performed for the embdding odd table corresponding to the wide model after the execution of the sub-task "odd table all2all", the sub-task "even table all2all" is used to implement the all2all communication operation in the training process of the embdding even table corresponding to the wide model, the sub-task "even table pre" is the calculation operation that needs to be performed for the embdding even table corresponding to the wide model before the execution of the sub-task "even table all2all", and the sub-task "even table post" is the calculation operation that needs to be performed for the embdding even table corresponding to the wide model after the execution of the sub-task "even table all2all".

[0086] Since the first training task and the second training task are independent of each other, each of the plurality of first sub-tasks (including odd table pre, odd table all2all and odd table post) is independent of each of the plurality of second sub-tasks (including even table pre, even table all2all and even table post).

[0087] S403: The management node sends an instruction to the distributed training system, instructing the plurality of computing nodes in the distributed training system to execute the plurality of first sub-tasks and the plurality of second sub-tasks, and in the process of executing the plurality of first sub-tasks and the plurality of second sub-tasks, the first computing sub-task and the first communication sub-task are executed in parallel, and the first computing sub-task and the first communication sub-task are independent of each other.

[0088] Among them, the first computing sub-task and the first communication sub-task belong to the plurality of first sub-tasks and the plurality of second sub-tasks. It can be understood that since the first computing sub-task and the first communication sub-task are independent of each other, the first computing sub-task and the first communication sub-task can be executed in parallel.

[0089] As can be known from the description in S402, each of the plurality of first sub-tasks is independent of each of the plurality of second sub-tasks, then the computing sub-task in the plurality of first sub-tasks is independent of the communication sub-task in the plurality of second sub-tasks, and the communication sub-task in the plurality of first sub-tasks is independent of the computing sub-task in the plurality of second sub-tasks, therefore, the first computing sub-task can be the computing sub-task in the plurality of first sub-tasks, and the first communication sub-task can be the communication sub-task in the plurality of second sub-tasks, or the first computing sub-task can be the computing sub-task in the plurality of second sub-tasks, and the first communication sub-task can be the communication sub-task in the plurality of first sub-tasks.

[0090] The first computing sub-task and the first communication sub-task are described below in combination with two specific examples.

[0091] Example 1, taking the plurality of first sub-tasks as Figure 5 the plurality of first sub-tasks (including wide pre, wide all2all and wide post) and the plurality of second sub-tasks as Figure 5 the plurality of second sub-tasks (including deep pre, deep all2all and deep post), then the first computing sub-task can be wide pre / wide post, and the first communication sub-task can be deep all2all, or the first computing sub-task can be deep pre / deep post, and the first communication sub-task can be wide all2all.

[0092] Example 2, taking the plurality of first sub-tasks as Figure 6 the plurality of first sub-tasks (including odd table pre, odd table all2all and odd table post), and the plurality of second sub-tasks as Figure 6 the plurality of second sub-tasks (including even table pre, even table all2all and even table post), then the first computing sub-task can be odd table pre / odd table post, and the first communication sub-task can be even table all2all, or the first computing sub-task can be even table pre / even table post, and the first communication sub-task can be odd table all2all. Specifically, the indication sent by the management node to the distributed training system can carry the plurality of first sub-tasks and the plurality of second sub-tasks, and carry the execution order information of the plurality of first sub-tasks and the plurality of second sub-tasks, wherein the execution order information of the plurality of first sub-tasks and the plurality of second sub-tasks contains information indicating that the first computing sub-task and the first communication sub-task are executed in parallel.

[0093] The execution order information of the plurality of first sub-tasks and the plurality of second sub-tasks is introduced below in combination with two detailed examples.

[0094] Example 1, referring toFigure 7A In Figure 7A , the multiple first sub-tasks and the multiple second sub-tasks are Figure 5 The multiple first sub-tasks (including wide pre, wide all2all and wide post) and the multiple second sub-tasks (including deep pre, deep all2all and deep post) shown in FIG. 3, taking the first computing sub-task as wide pre and the first communication sub-task as deep all2all as an example, since the first computing sub-task “wide pre” and the first communication sub-task “deep all2all” are independent of each other, the execution order information of the multiple first sub-tasks and the multiple second sub-tasks can be Figure 7A The execution order shown in FIG. 3: in the forward propagation process in each iteration of the wide & deep model training, deep pre is executed first, after the execution of deep pre is completed, wide pre and deep all2all are executed in parallel, after the execution of wide pre and deep all2all is completed, wide all2all, wide post and deep post are executed in series; in the back propagation process in each iteration of the wide & deep model training, deep post, wide post and wide all2all are executed in series first, after the execution of wide all2all is completed, wide pre and deep all2all are executed in parallel, after the execution of wide pre and deep all2all is completed, deep pre is executed.

[0095] Example two, referring to Figure 8A In Figure 8A , the multiple first sub-tasks and the multiple second sub-tasks are Figure 6 The multiple first sub-tasks (including odd table pre, odd table all2all and odd table post) and the multiple second sub-tasks (including even table pre, even table all2all and even table post) shown in FIG. 4, taking the first computing sub-task as odd table post and the first communication sub-task as even table all2all as an example, since the first computing sub-task “odd table post” and the first communication sub-task “even table all2all” are independent of each other, the execution order information of the multiple first sub-tasks and the multiple second sub-tasks can be Figure 8AThe execution order shown: in the forward propagation process in each iteration of wide model training, odd table pre, odd table all2all and even table pre are executed in series first, after the execution of even table pre is completed, odd table post and even table all2all are executed in parallel, after the execution of odd table post and even table all2all is completed, even table post is executed; in the backward propagation process in each iteration of wide model training, even table post is executed first, after the execution of even table post is completed, odd table post and even table all2all are executed in parallel, after the execution of odd table post and even table all2all is completed, even table pre, odd table all2all and odd table pre are executed in series.

[0096] Comparison Figure 7A The training process of the wide&deep model shown and Figure 2 The training process of the wide&deep model shown can be seen that:

[0097] The difference between the two training processes is that in Figure 7A , the calculation subtask wide pre and the communication subtask deepall2all are executed in parallel, and the execution time of the communication subtask deepall2all is hidden, while in Figure 2 , the calculation subtask wide pre and the communication subtask deepall2all are executed in series, and the execution time of the communication subtask deepall2all is not hidden, therefore, Figure 7A The training time of the wide&deep model shown is reduced, and the training efficiency is higher.

[0098] Comparison Figure 7A The training process of the wide model shown and Figure 3 The training process of the wide model shown can be seen that:

[0099] The difference between the two training processes is that in Figure 8A , the calculation subtask "even table pre" and the communication subtask "odd table all2all" are executed in parallel, and the execution time of the communication subtask "odd table all2all" is hidden, while in Figure 3 , the communication subtask "wideall2all" is composed of odd table all2all and even table all2all, and the execution time of the communication subtask "wideall2all" is not hidden, and it can be deduced that the execution time of the odd table all2all is not hidden, therefore, Figure 8A The training time of the wide model shown is reduced, and the training efficiency is higher.

[0100] In one possible embodiment, in addition to the mutually independent first computation subtask and first communication subtask, the plurality of first subtasks and the plurality of second subtasks may also include other mutually independent computation subtasks (hereinafter referred to as second computation subtasks) and other communication subtasks (hereinafter referred to as second communication subtasks). The second computation subtasks and second communication subtasks are illustrated below with two specific examples.

[0101] Example 1, using multiple first subtasks and multiple second subtasks as... Figure 5 The diagram shows multiple first subtasks (including wide pre, wideall2all, and wide post) and multiple second subtasks (including deeppre, deepall2all, and deeppost). Taking widepre as the first computation subtask and deepall2all as the first communication subtask as an example, deeppre / deeppost and wideall2all are independent of each other in these multiple first and second subtasks. Therefore, the second computation subtask can be deeppre / deeppost, and the second communication subtask can be wideall2all.

[0102] Example 2, using multiple first subtasks and multiple second subtasks as... Figure 6 The diagram shows multiple third subtasks (including odd table pre, odd table all2all, and odd table post) and multiple fourth subtasks (including even table pre, even table all2all, and even table post). Taking the example of the first computation subtask being odd table post and the first communication subtask being even table all2all, even table pre / even table post and odd table all2all are also independent of each other in these multiple subtasks. Therefore, the second computation subtask can be even table pre / even table post, and the second communication subtask can be odd table all2all.

[0103] It can be understood that, in the case that the plurality of sub-tasks include the first computing sub-task and the first communication sub-task which are independent of each other, and also include the second computing sub-task and the second communication sub-task which are independent of each other, if the plurality of computing nodes not only perform the first computing sub-task and the first communication sub-task in parallel in the process of performing the plurality of sub-tasks, but also perform the second computing sub-task and the second communication sub-task in parallel, not only the execution time consumption of the first communication sub-task can be masked, but also the execution time consumption of the second communication sub-task can be masked, further reducing the AI model training time consumption and improving the training efficiency. Therefore, in a possible embodiment, in S403, the management node not only instructs the plurality of computing nodes to perform the first computing sub-task and the first communication sub-task in parallel, but also instructs the plurality of computing nodes to perform the second computing sub-task and the second communication sub-task in parallel. In this case, the execution order information of the plurality of first sub-tasks and the plurality of second sub-tasks carried by the indication sent by the management node to the distributed training system also contains information indicating that the second computing sub-task and the second communication sub-task are performed in parallel.

[0104] The execution order information of the plurality of first sub-tasks and the plurality of second sub-tasks will be introduced below in combination with two detailed examples.

[0105] Example one, see Figure 7B In Figure 7B , the plurality of first sub-tasks and the plurality of second sub-tasks are Figure 5 The plurality of first sub-tasks (including wide pre, wideall2all and wide post) and the plurality of second sub-tasks (including deeppre, deepall2all and deeppost) shown in the table, the first computing sub-task is wide pre, the first communication sub-task is deepall2all, the second computing sub-task is deeppost, and the second communication sub-task is wideall2all. Since the first computing sub-task “wide pre” and the first communication sub-task “deepall2all” are independent of each other, and the second computing sub-task “deeppost” and the second communication sub-task “wideall2all” are independent of each other, the execution order information of the plurality of first sub-tasks and the plurality of second sub-tasks can be Figure 7BThe execution order shown: in the forward propagation process in each iteration of the wide&deep model training, first execute deep pre, after the execution of deep pre is completed, execute wide pre and deep all2all in parallel, after the execution of wide pre and deep all2all is completed, execute deep post and wide all2all in parallel, after the execution of deep post and wide all2all is completed, execute wide post; in the back propagation process in each iteration of the wide&deep model training, first execute wide post, after the execution of wide post is completed, execute deep post and wide all2all in parallel, after the execution of deep post and wide all2all is completed, execute wide pre and deep all2all in parallel, after the execution of wide pre and deep all2all is completed, execute deep pre.

[0106] Example two, see Figure 8B , in Figure 8B , a plurality of first subtasks and a plurality of second subtasks are Figure 6 The plurality of first subtasks (including odd table pre, odd table all2all and odd table post) and the plurality of second subtasks (including even table pre, even table all2all and even table post) shown, the first calculation subtask is odd table post, the first communication subtask is even table all2all, the second calculation subtask is even table pre, and the second communication subtask is odd table all2all. As the first calculation subtask "odd table post" and the first communication subtask "even table all2all" are independent of each other, and the second calculation subtask "even table pre" and the second communication subtask "odd table all2all" are independent of each other, the execution order information of the plurality of first subtasks and the plurality of second subtasks can be Figure 8B The execution order shown: in the forward propagation process in each iteration of the wide model training, first execute odd table pre, after the execution of odd table pre is completed, execute even table pre and odd table all2all in parallel, after the execution of even table pre and odd table all2all is completed, execute odd table post and even table all2all in parallel, after the execution of odd table post and even table all2all is completed, execute even table post; in the back propagation process in each iteration of the wide model training, first execute even table post, after the execution of even table post is completed, execute odd table post and even table all2all in parallel, after the execution of odd table post and even table all2all is completed, execute even table pre and odd table all2all in parallel, after the execution of even table pre and odd table all2all is completed, execute odd table pre.

[0107] Comparison Figure 7B The training process of the wide & deep model shown in FIG. 1 and Figure 2 The training process of the wide & deep model shown in FIG. 1 can be seen as follows:

[0108] The difference between the two training processes is that in the Figure 7B , the calculation subtask wide pre and the communication subtask deep all2all are executed in parallel, and the calculation subtask deep post and the communication subtask wide all2all are executed in parallel, and the execution time of the communication subtask deep all2all and the communication subtask wide all2all is hidden, while in the Figure 2 , the calculation subtask wide pre and the communication subtask deep all2all are executed in series, and the calculation subtask deep post and the communication subtask wide all2all are executed in series, and the execution time of the communication subtask deep all2all and the communication subtask wide all2all is not hidden, therefore, Figure 7B The training time of the wide & deep model shown in FIG. 1 is reduced, and the training efficiency is higher.

[0109] Comparison Figure 7B The training process of the wide & deep model shown in FIG. 1 and Figure 7A The training process of the wide & deep model shown in FIG. 1 can be seen as follows:

[0110] The difference between the two training processes is that in the Figure 7B , the calculation subtask wide pre and the communication subtask deep all2all are executed in parallel, and the calculation subtask deep post and the communication subtask wide all2all are executed in parallel, and the execution time of the communication subtask deep all2all and the communication subtask wide all2all is hidden, while in the Figure 7A , the calculation subtask wide pre and the communication subtask deep all2all are executed in parallel, the calculation subtask deep post and the communication subtask wide all2all are executed in series, the execution time of the communication subtask deep all2all is hidden, and the execution time of the communication subtask wide all2all is not hidden, therefore, Figure 7B The training time of the wide & deep model shown in FIG. 1 is reduced, and the training efficiency is higher.

[0111] Comparison Figure 8B The training process of the wide model shown in FIG. 1 and Figure 3 The training process of the wide model shown in FIG. 1 can be seen as follows:

[0112] The difference between the two training processes is that in the training process shown in Figure 8B , the computation subtask "even table pre" is executed in parallel with the communication subtask "odd table all2all", and the computation subtask "odd table post" is executed in parallel with the communication subtask "even table all2all", and the execution time of the communication subtasks "odd table all2all" and "even table all2all" is masked, while in the training process shown in Figure 3 , the communication subtask "wide all2all" is composed of the odd table all2all and the even table all2all, and the execution time of the communication subtask "wide all2all" is not masked, and it can be deduced that the execution time of the odd table all2all and the even table all2all is not masked, and therefore, Figure 8B the training time of the wide model shown in is reduced, and the training efficiency is higher.

[0113] Comparing the training process of the wide model shown in Figure 8B with the training process of the wide model shown in Figure 8A , it can be seen that:

[0114] The difference between the two training processes is that in the training process shown in Figure 8B , the computation subtask "even table pre" is executed in parallel with the communication subtask "odd table all2all", and the computation subtask "odd table post" is executed in parallel with the communication subtask "even table all2all", and the execution time of the communication subtasks "odd table all2all" and "even table all2all" is masked, while in the training process shown in Figure 8A , the computation subtask "even table pre" is executed in parallel with the communication subtask "odd table all2all", and the computation subtask "odd table post" is executed in series with the communication subtask "even table all2all", and the execution time of the communication subtask "odd table all2all" is masked, while the execution time of the communication subtask "even table all2all" is not masked, and therefore, Figure 8B the training time of the wide model shown in is reduced, and the training efficiency is higher.

[0115] It should be noted that Figure 7A , Figure 7B , Figure 8A and Figure 8B the execution order of the plurality of first subtasks and the plurality of second subtasks in the model training process shown is only an example and should not be considered as a specific limitation, for example, in a specific implementation, the execution order with the largest parallel masking degree can be selected according to the actual scene.

[0116] S404: Multiple computing nodes execute multiple first subtasks and multiple second subtasks, and in the process of executing multiple first subtasks and multiple second subtasks, execute the first computing subtask and the first communication subtask in parallel.

[0117] Specifically, after receiving instructions from the management node, multiple computing nodes execute multiple first subtasks and multiple second subtasks according to the execution order information of multiple first subtasks and multiple second subtasks to complete the training of the AI ​​model. For detailed execution process, please refer to S403. Figure 7A , Figure 7B , Figure 8A and Figure 8B Related descriptions.

[0118] In summary, the AI ​​model training method provided in this application splits the AI ​​model training task into mutually independent first and second training tasks. The first training task is further divided into multiple first sub-tasks, and the second training task is further divided into multiple second sub-tasks. Each first and second sub-task is either a computational or a communication sub-task. The computational sub-tasks within the multiple first sub-tasks are independent of the communication sub-tasks within the multiple second sub-tasks, and / or the communication sub-tasks within the multiple first sub-tasks are independent of the computational sub-tasks within the multiple second sub-tasks. Then, during the training of the AI ​​model by multiple computing nodes in the distributed training system, the mutually independent computational and communication sub-tasks are executed in parallel, masking the execution time of the communication sub-tasks. This shortens the training time of the AI ​​model and improves training efficiency.

[0119] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0120] The training method of the AI ​​model provided in this application has been described in detail above. In order to facilitate the better implementation of the above-described scheme of the embodiments of this application, the following also provides related devices and equipment for cooperating in the implementation of the above scheme.

[0121] See Figure 9 , Figure 9 This is a structural diagram of a management node 900 provided in this application. The management node 900 can be... Figure 4 The management node described in the embodiments, such as Figure 9 As shown, the management node 900 may include a splitting unit 910 and a notification unit 920. The functions of each unit module of the management node 900 are described below by way of example.

[0122] The splitting unit 910 is configured to split the training task of the AI model into a first training task and a second training task which are independent of each other.

[0123] The splitting unit 910 is further configured to split the first training task into a plurality of first sub-tasks and split the second training task into a plurality of second sub-tasks, each of the plurality of first sub-tasks and the plurality of second sub-tasks being a computing sub-task or a communication sub-task.

[0124] The notification unit 920 is configured to instruct a plurality of computing nodes in the distributed training system to perform the plurality of first sub-tasks and the plurality of second sub-tasks, and perform a first computing sub-task and a first communication sub-task in parallel in a process of performing the plurality of first sub-tasks and the plurality of second sub-tasks, the first computing sub-task and the first communication sub-task being independent of each other, and the first computing sub-task and the first communication sub-task belonging to the plurality of first sub-tasks and the plurality of second sub-tasks.

[0125] In some possible embodiments, the AI model comprises a first sub-AI model and a second sub-AI model, the first training task is used for training the first sub-AI model, and the second training task is used for training the second sub-AI model.

[0126] In some possible embodiments, the AI model is a wide&deep model.

[0127] In some possible embodiments, the first training task is a training task for an embdding vector with an odd index in an embdding vector table corresponding to the AI model, and the second training task is a training task for an embdding vector with an even index in the embdding vector table corresponding to the AI model.

[0128] In some possible embodiments, the AI model is a wide model or a deep model.

[0129] In specific implementations, the splitting unit 910 and the notification unit 920 can be implemented by software or by hardware. For example, the implementation of the splitting unit 910 is described below. Similarly, the implementation of other units such as the splitting unit 910 can be implemented by referring to the implementation of the splitting unit 910.

[0130] As an example of implementation of the unit by software, the splitting unit 910 can include code running on a compute instance. The compute instance can include at least one of a physical host (a computing device), a virtual machine, a container. Further, the compute instance can be one or more. For example, the splitting unit 910 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers for running the code can be distributed in the same availability zone (AZ), or in different AZs, each of which includes one data center or multiple data centers in close geographical proximity. Generally, one region can include multiple AZs.

[0131] Similarly, the multiple hosts / virtual machines / containers for running the code can be distributed in the same virtual private cloud (VPC), or in multiple VPCs. Generally, one VPC is set up in one region, and communication between two VPCs in the same region, or between VPCs in different regions, needs to be set up in each VPC to set up a communication gateway, and the interconnection between VPCs is realized through the communication gateway.

[0132] As an example of implementation of the unit by hardware, the splitting unit 910 can include at least one computing device, such as a server, etc. Alternatively, the splitting unit 910 can also be implemented by using a CPU, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), an NPU, a system on chip (SoC), an offload card, an acceleration card, or any combination thereof.

[0133] The splitting unit 910 includes multiple computing devices when it includes multiple computing devices, which can be distributed in the same region or in different regions. The multiple computing devices included in the splitting unit 910 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the splitting unit 910 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, acceleration cards, and the like.

[0134] It should be noted that in other embodiments, the notification unit 920 can be configured to perform any step performed by the management node in the AI model training method provided in the present application, and the splitting unit 910 can be configured to perform any step performed by the management node in the AI model training method provided in the present application, Figure 9 The steps implemented by each unit in the above are specified as needed to implement all functions of the management node 900.

[0135] It should be understood that the functions of the various unit modules described above are only the functions that the management node 900 can have in some embodiments of the present application, and the present application does not limit the functions of the various unit modules.

[0136] It should also be understood that Figure 9 is an exemplary division manner, and the management node 900 can include more or fewer unit modules, and the division manner of the unit modules in the management node 900 can be adjusted flexibly based on actual business scenarios, and the present application does not make specific limitations.

[0137] Referring to Figure 10 , Figure 10 is another structure diagram of a management node provided in the present application. The management node can be Figure 4 the management node in the method, or used to deploy Figure 9 the management node 900 shown in FIG. 9.

[0138] In Figure 10 , the computing device 1000 represents the management node provided in the present application, as Figure 10As shown, the computing device 1000 includes a bus 1001, a processor 1002, a memory 1003, a network interface 1004, and an output interface 1005. The processor 1002, the memory 1003, the network interface 1004, and the output interface 1005 communicate with each other through the bus 1001. The computing device 1000 can be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device 1000 can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone. It should be understood that the number of processors and memories in the computing device 1000 is not limited in the present application.

[0139] The bus 1001 can be a peripheral component interconnect express (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), or the like. The bus 1001 can be divided into an address bus, a data bus, a control bus, and the like.

[0140] In addition to including a data bus, the bus 1001 can also include a power bus, a control bus, a status signal bus, and the like. However, for the sake of clarity, all buses are labeled as bus 1001 in the figure. For ease of representation, Figure 10 Only one thick line is used in the figure to represent the bus 1001, but it does not mean that there is only one bus or only one type of bus.

[0141] The processor 1002 can include any one or more of a CPU, a GPU, a microprocessor (MP), or a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, an acceleration card, and the like computing device.

[0142] The processor 1002 executes various types of digital storage instructions, such as software or firmware programs stored in the memory 1020, which enables the computing device 1000 to provide a wide variety of services.

[0143] In a specific implementation, as an embodiment, the processor 1002 includes one or more CPUs.

[0144] In a particular implementation, as an example, the computing device 1000 also includes multiple processors, each of which can be a single-CPU or a multi-CPU. The processor herein refers to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0145] The memory 1003 can include volatile memory (e.g., random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD)), or combinations of the above.

[0146] The memory 1003 stores executable programs, and the processor 1002 executes the executable programs to implement the functions of the aforementioned splitting unit 910 and the notification unit 920, respectively, thereby implementing the steps performed by the management node in the AI model training method described above. That is, the memory 1003 stores instructions for performing the steps performed by the management node in the AI model training method described above.

[0147] The network interface 1004 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement the acquisition of files or network data.

[0148] The output interface 1005 uses a wired interface such as, but not limited to, a wired interface to implement communication between the computing device 1000 and other devices or communication networks.

[0149] As a possible implementation, the computing device 1000 can also include a chip system including the processor 1002 and a power supply circuit for supplying power to the processor 1002 for performing the operation steps corresponding to the AI model training method. For brevity, details are not repeated here. The processor 1002 can be implemented by a CPU, or by a GPU, DPU, NPU, XPU, SoC, offload card, acceleration card, or other computing device or AI chip.

[0150] As a possible implementation, the computing device 1000 can include multiple types of processors 1002, i.e., the computing device 1000 is a heterogeneous device, for example, the computing device 1000 includes a CPU and a GPU, and the operation steps corresponding to the training method of the AI model can be performed by at least one of the processors. For brevity, details are not repeated here.

[0151] The computing device 1000 described above is used to execute the training method of the AI model provided in the present application, and the specific implementation process is described in the above method embodiments, which are not repeated here.

[0152] It should be understood that the computing device 1000 is only an example provided by the embodiments of the present application, and the computing device 1000 can have more or fewer components than those shown, can combine two or more components, or can have a different configuration of components. For what is not shown or described in the embodiments of the present application, please refer to the relevant description in the foregoing Figure 10 embodiments, which are not repeated here. Figures 1-9

[0153] Referring to Figure 11 , Figure 11 is a structural schematic diagram of a computing device cluster provided in the present application. The computing device cluster provided in the present application includes at least one computing device. The computing device can be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a desktop computer, a notebook computer, or a terminal device such as a smart phone.

[0154] As Figure 11 shown, the computing device cluster includes at least one computing device 1000. The memory 1003 in one or more computing devices 1000 in the computing device cluster can have the same instructions for executing the training method of the AI model.

[0155] In some possible implementations, the memory 1003 of one or more computing devices 1000 in the computing device cluster can also respectively have partial instructions for executing the training method of the AI model. In other words, the combination of one or more computing devices 1000 can collectively execute the instructions for executing the training method of the AI model.

[0156] It should be noted that the memories 1003 in different computing devices 1000 in the computing device cluster can store different instructions, respectively used to execute part of the functions of the management node 900.

[0157] The present application also provides a computing system, which includes the foregoing management node (such as the management node 900 shown in Figure 9 or Figure 10 ​The computing device 1000 shown is used to execute the training method of the AI model, and the plurality of computing nodes (i.e., a distributed training system) are used to execute the training method of the AI model, wherein the management node is used to execute Figure 4 The plurality of computing nodes are used to execute S401-S403 shown, and the plurality of computing nodes are used to execute S404 shown. Figure 4 The plurality of computing nodes are used to execute S401-S403 shown, and the plurality of computing nodes are used to execute S404 shown. Figure 12 The computing system shown includes the management node and the 4 computing nodes (i.e., GPU 1101 to GPU 1104) connected by the bus in the computing device 1100. The computing device 1100 can be a server, for example, a central server, an edge server, or a local server in a local data center, etc.

[0158] It should be understood that, Figure 12 For example, in a specific implementation, the plurality of computing nodes are a plurality of CPUs / NPUs / TPUs in a single computing device, or a plurality of CPUs / GPUs / NPUs / TPUs in a plurality of computing devices, or any combination of CPUs, GPUs, NPUs, and TPUs. Optionally, the computing system is a single computing device, one CPU / GPU / NPU / TPU in the single computing device is the management node, and the other plurality of CPUs / GPUs / NPUs / TPUs or combinations in the single computing device are the plurality of computing nodes. The type and number of the computing system and the plurality of computing nodes are not limited in the present application.

[0159] The present application also provides a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute the training method of the AI model.

[0160] The present application also provides a computer readable storage medium. The computer readable storage medium can be any available medium that the computing device can store or a data storage device such as a data center containing one or more available media. The available media can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk), etc. The computer readable storage medium includes instructions for instructing the computing device to execute the training method of the AI model.

[0161] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training an artificial intelligence (AI) model, characterized in that, The method is applied to a distributed training system, which includes multiple computing nodes. The training task of the AI ​​model is divided into two independent training tasks: a first training task and a second training task. The first training task is further divided into multiple first sub-tasks, and the second training task is divided into multiple second sub-tasks. The multiple computing nodes jointly execute each of the multiple first and second sub-tasks. Each sub-task is either a computational sub-task or a communication sub-task. The method includes: The plurality of computing nodes acquire the plurality of first subtasks and the plurality of second subtasks; The plurality of computing nodes execute the plurality of first subtasks and the plurality of second subtasks, and in the process of executing the plurality of first subtasks and the plurality of second subtasks, execute the first computing subtask and the first communication subtask in parallel. The first computing subtask and the first communication subtask are independent of each other and belong to the plurality of first subtasks and the plurality of second subtasks.

2. The method according to claim 1, characterized in that, The AI ​​model includes a first sub-AI model and a second sub-AI model. The first training task is used to train the first sub-AI model, and the second training task is used to train the second sub-AI model.

3. The method according to claim 2, characterized in that, The AI ​​model is a wide & deep model.

4. The method according to claim 1, characterized in that, The first training task is to train the embedding vectors with odd indices in the embedding vector table corresponding to the AI ​​model, and the second training task is to train the embedding vectors with even indices in the embedding vector table corresponding to the AI ​​model.

5. The method according to claim 4, characterized in that, The AI ​​model is either a wide model or a deep model.

6. The method according to any one of claims 1 to 5, wherein the computing node is an accelerator, and the accelerator is a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or other dedicated AI processing chip.

7. A method for training an AI model, characterized in that, The method is applied to a management node connected to a distributed training system, which includes multiple computing nodes that jointly execute the training task of the AI ​​model. The management node splits the training task of the AI ​​model into a first training task and a second training task that are independent of each other. The management node splits the first training task into multiple first sub-tasks and the second training task into multiple second sub-tasks, wherein each of the multiple first sub-tasks and the multiple second sub-tasks is a computation sub-task or a communication sub-task. The management node instructs the distributed training system to perform the method as described in any one of claims 1 to 6.

8. A management node, characterized in that, The management node is connected to a distributed training system, which includes multiple computing nodes that jointly execute the training task of the AI ​​model. The management node includes: A splitting unit is used to split the training task of the AI ​​model into a first training task and a second training task that are independent of each other. The splitting unit is further configured to split the first training task into multiple first sub-tasks and split the second training task into multiple second sub-tasks, wherein each of the multiple first sub-tasks and the multiple second sub-tasks is a computation sub-task or a communication sub-task. A notification unit is used to instruct the distributed training system to perform the method as described in any one of claims 1 to 6.

9. A management node, characterized in that, The management node includes a processor and a memory, the memory storing code, and the processor executing the code to implement the method as described in claim 7.

10. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in claim 7.

11. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device, the computing device performs the method as described in claim 7.

12. A computing system, characterized in that, The computing system includes a distributed training system and a management node, the distributed training system being used to perform the method as described in any one of claims 1 to 6, and the management node performing the method as described in claim 7.

Images