Artificial intelligence (AI) model training method and apparatus, and computing device

Abstract

The present application relates to the technical field of artificial intelligence (AI), and discloses an AI model training method and apparatus, and a computing device. The method comprises: when performing forward computation on a plurality of modules of an AI model, for a first module on which recomputation is to be performed and a second module that depends on output data of the first module, once the forward computation of the second module is completed, discarding the output data of the first module; and when performing backward computation on the plurality of modules, scheduling the recomputation of the first module to be performed prior to the backward computation of the second module. In this way, the performing of the backward computation on the second module can depend on data obtained by performing recomputation on the first module. By means of the flexible recomputation scheduling method, the output data of the first module does not need to be stored from a forward computation process to a backward computation process, thereby saving memory resources of accelerator cards.

Description

Training method and device of artificial intelligence AI model and computing device

[0001] The present application claims priority from the Chinese patent application No. 202410869302.6 filed on June 29, 2024 and entitled "Training method and device of artificial intelligence AI model and computing device", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a training method and device of an artificial intelligence AI model and a computing device. BACKGROUND

[0003] With the rapid development of AI technology, a series of acceleration cards emerge to provide computing power for computing matrices, vectors and the like to accelerate the computing process of AI models. Currently, recomputation is adopted in the training process of AI models, which can effectively reduce the memory occupation of the acceleration card in the model training process.

[0004] In the related art, taking an AI model including a module A and a module B as an example, the implementation of recomputation in the model training process is as follows: the module A is taken as a module that needs to perform recomputation, in the forward calculation process, the acceleration card sequentially performs the forward calculation of the module A and the module B, wherein the acceleration card discards the intermediate activation value of the module A and stores the output activation value of the module A when performing the forward calculation of the module A; in the backward calculation process, the acceleration card performs the backward calculation of the module B based on the output activation value of the module A stored in the forward calculation process, then performs the forward calculation of the module A again to obtain the intermediate activation value and the output activation value of the module A (i.e., performs the recomputation of the module A), and then performs the backward calculation of the module A based on the intermediate activation value of the module A, and discards the intermediate activation value of the module A after performing the backward calculation of the module A to reduce the memory occupation of the acceleration card.

[0005] In the above method, since the recomputation and the backward calculation of the module A need to be performed together, and the backward calculation of the module A is performed after the backward calculation of the module B, in order to ensure that the acceleration card can obtain the output activation value of the module A when performing the backward calculation of the module B, the acceleration card must retain the output activation value of the module A when performing the forward calculation of the module A, and retain it until after the backward calculation of the module B, which causes the waste of the memory resources of the acceleration card. SUMMARY

[0006] The embodiment of the application provides a training method, device and computing equipment of an artificial intelligence AI model, which can save memory resources of a graphics processing unit in the training process of the AI model.

[0007] In a first aspect, the application provides a training method of an AI model, the method comprising:

[0008] performing forward calculation on a plurality of modules of the AI model in response to a training request of the AI model, the plurality of modules comprising a first module and a second module, the first module being a module that needs to perform re-calculation in a backward calculation process, and the second module being a module that needs to use output data of the first module in a forward calculation process;

[0009] discarding the output data of the first module after the forward calculation of the second module is completed;

[0010] performing backward calculation on the plurality of modules of the AI model, performing re-calculation on the first module before performing backward calculation on the second module, and performing backward calculation on the second module based on data obtained by performing re-calculation on the first module.

[0011] In the above manner, when performing forward calculation on the plurality of modules of the AI model, for the first module to be executed and the second module dependent on the output data of the first module, the output data of the first module is discarded after the forward calculation of the second module is completed, and when performing backward calculation on the plurality of modules, the re-calculation of the first module is scheduled to be performed before the backward calculation of the second module, so that when performing backward calculation on the second module, the data obtained by performing re-calculation on the first module can be relied on. Through this flexible re-calculation scheduling manner, the output data of the first module does not need to be stored from the forward calculation process to the backward calculation process, thereby saving the memory resources of the graphics processing unit.

[0012] In some embodiments, the second module is a next module of the first module in the plurality of modules.

[0013] In some embodiments, the AI model comprises a plurality of network layers, at least one network layer is executed on each graphics processing unit, the at least one network layer comprises a plurality of modules executed in sequence, and the second module is the last module in the plurality of modules.

[0014] In some embodiments, the AI model is a Transformer model, and the network layer is a Transformer layer.

[0015] In a second aspect, the application provides a training method of an artificial intelligence AI model, the method comprising:

[0016] In response to a training request of the AI model, determining a plurality of modules of the AI model, the plurality of modules comprising a first module to be executed re-calculation;

[0017] generating a forward computation task of each module of the plurality of modules, a backward computation task of each module, and a re-computation task of the first module;

[0018] The re-computation task is sent to the accelerator card before the first backward computation task related to the training request is sent to the accelerator card, and there is a dependency relationship between the first backward computation task and the re-computation task.

[0019] In some embodiments, the training request is executed in a pipeline parallel manner by N accelerator cards, N being a positive integer;

[0020] The re-computation task is sent to the accelerator card before the first backward computation task related to the training request is sent to the accelerator card, and there is a dependency relationship between the first backward computation task and the re-computation task.

[0021] In some embodiments, the method further comprises:

[0022] The second backward computation task is sent to a second accelerator card of the N accelerator cards, the second accelerator card being configured to execute a computation task related to a third module, the third module being a next module of the first module, and the second backward computation task being a backward computation task of the third module;

[0023] The first backward computation task is sent to the first accelerator card, the first backward computation task indicating that the first module is backward computed based on an execution result of the second backward computation task.

[0024] In some embodiments, the training request is executed in a virtual pipeline parallel manner by M accelerator cards, M being a positive integer;

[0025] In response to a training request of an artificial intelligence (AI) model, a plurality of modules of the AI model are determined, comprising: in response to the training request, based on a number of virtual pipelines, a number of pipelines, and a number of the M accelerator cards, a first module is determined;

[0026] The re-computation task is sent to the accelerator card before the first backward computation task related to the training request is sent to the accelerator card, and there is a dependency relationship between the first backward computation task and the re-computation task.

[0027] The method further comprises: sending the forward computation task of the first module to the third accelerator card, the forward computation task of the first module indicating that the intermediate activation value of the first module is discarded after the forward computation of the first module is performed.

[0028] In a third aspect, an embodiment of the present application provides a device for training an AI model, the device comprising at least one functional unit configured to perform the method for training an AI model according to the first aspect or any possible implementation of the first aspect.

[0029] In a fourth aspect, an embodiment of the present application provides a device for training an AI model, the device comprising at least one functional unit configured to perform the method for training an AI model according to the second aspect or any possible implementation of the second aspect.

[0030] In a fifth aspect, the present application provides a computing device comprising a processor and a memory, the processor configured to execute at least one piece of program code stored in the memory, so that the computing device implements the method for training an AI model according to the first aspect or any possible implementation of the first aspect, or implements the method for training an AI model according to the second aspect or any possible implementation of the second aspect.

[0031] In a sixth aspect, an embodiment of the present application provides a computing system comprising a host and at least one accelerator card, the computing system configured to implement the method for training an AI model according to the first aspect or any possible implementation of the first aspect, or implement the method for training an AI model according to the second aspect or any possible implementation of the second aspect.

[0032] In a seventh aspect, the present application provides a computer-readable storage medium configured to store at least one piece of program code, when the at least one piece of program code is executed by a computing device, so that the computing device implements the method for training an AI model according to the first aspect or any possible implementation of the first aspect, or implements the method for training an AI model according to the second aspect or any possible implementation of the second aspect. The storage medium includes but is not limited to a volatile memory, such as a random access memory, a non-volatile memory, such as a flash memory, a hard disk drive (HDD), a solid state drive (SSD).

[0033] In an eighth aspect, the present application provides a computer program product, which, when running on a computing device, causes the computing device to implement the AI model training method provided by the first aspect or any possible implementation manner of the first aspect, or implement the AI model training method provided by the second aspect or any possible implementation manner of the second aspect. The computer program product can be a software installation package, and can be downloaded and executed on the computing device in a case where the AI model training method needs to be implemented. BRIEF DESCRIPTION OF DRAWINGS

[0034] FIG. 1 is a schematic diagram of an application scenario of a related art and the present application;

[0035] FIG. 2 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0036] FIG. 3 is a structural schematic diagram of a computing device provided by an embodiment of the present application;

[0037] FIG. 4 is a flowchart of an AI model training method provided by an embodiment of the present application;

[0038] FIG. 5 is a flowchart of another AI model training method provided by an embodiment of the present application;

[0039] FIG. 6 is a flowchart of another AI model training method provided by an embodiment of the present application;

[0040] FIG. 7 is a flowchart of another AI model training method provided by an embodiment of the present application;

[0041] FIG. 8 is a schematic diagram of an AI model training method based on pipeline parallelism provided by an embodiment of the present application;

[0042] FIG. 9 is a flowchart of another AI model training method provided by an embodiment of the present application;

[0043] FIG. 10 is a schematic diagram of an AI model training method based on virtual pipeline parallelism provided by an embodiment of the present application;

[0044] FIG. 11 is a schematic diagram of another AI model training method based on virtual pipeline parallelism provided by an embodiment of the present application;

[0045] FIG. 12 is a structural schematic diagram of an AI model training apparatus provided by an embodiment of the present application;

[0046] FIG. 13 is a structural schematic diagram of another AI model training apparatus provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] For the purposes of the present application, the technical solutions and advantages will be further described in detail below with reference to the drawings. It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.), and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data need to comply with relevant laws, regulations, and standards of relevant countries and regions. For example, the AI model, computing task-related data, etc. involved in the present application are obtained under full authorization.

[0048] For the convenience of understanding, the key terms and key concepts involved in the present application will be described first.

[0049] An artificial intelligence (AI) model is a mathematical algorithm model that uses machine learning ideas to solve practical problems. Generally, an AI model includes a large number of parameters and calculation formulas (or calculation rules).

[0050] An operator (OP) is a computing unit or computing function running on a computing device. In the field of deep learning, neural network layers and even entire models are composed of operators, which correspond to the computing logic in the neural network layers. For example, a convolution layer is an operator; the weight summation process in a fully-connected layer (FC layer) is an operator.

[0051] An acceleration card, also known as an acceleration device, an accelerator, or an acceleration chip, is a special hardware accelerator or computer system designed to accelerate AI applications, especially neural networks, machine vision, and machine learning. For example, it is used to provide computing power for matrix and vector calculations to accelerate the computation of AI models. Illustratively, the acceleration device is, for example, a graphics processing unit (GPU), a neural network processing unit (XPU), an intelligent processing unit (IPU), a tensor processing unit (TPU), a domain specific architecture (DSA) chip, etc., without being limited thereto.

[0052] Memory peak, in this application, refers to the maximum value of memory usage of the accelerator card during the AI model training process. Generally, the size of the memory peak depends on multiple factors, such as model size (number of layers, number of parameters), size of training data, batch size, storage of intermediate calculation results (such as activation values generated by forward calculation). For example, because the AI model training process requires that the activation values generated by forward calculation be left for backward calculation, the activation values generated by forward calculation will increase before backward calculation begins, resulting in excessive memory usage, and eventually reaching the memory peak, causing memory shortage. In other words, the memory peak is generally reached during forward calculation, and after backward calculation begins, the memory usage gradually decreases as the backward calculation progresses.

[0053] Recomputation refers to, during the training of an AI model, not retaining the intermediate activation values (activation) of each module during forward calculation, but re-executing the forward calculation of the corresponding module before executing the backward calculation of each module, re-generating the intermediate activation values, and discarding the intermediate activation values after the backward calculation of the module is completed, thereby alleviating the problem of excessive memory peak of the accelerator card.

[0054] Pipelined parallelism (PP) is a model parallel training method that divides different layers of a model to different devices (such as accelerator cards), and each device is responsible for training some layers, thereby reducing the memory consumption of a single device and achieving large-scale model training. For example, a model includes 12 network layers, accelerator card 0 executes layers 0-3, accelerator card 1 executes layers 4-7, accelerator card 2 executes layers 8-11, and the accelerator cards exchange data through a communication link. During forward calculation, card 0 performs forward calculation of layers 0-3 and sends the results to card 1, card 1 performs forward calculation of layers 4-7 and sends the results to card 2, and card 2 performs forward calculation of layers 8-11. The backward calculation process is similar, starting from card 2 to perform backward calculation to obtain gradient information, which is sequentially passed back to card 1 and card 0. In some scenarios, this pipelined parallelism is also called naive pipelined parallelism.

[0055] Virtual pipeline parallelism (VPP) is an extended training manner of pipeline parallelism, in which more pipeline stages are divided under the condition that the number of devices remains unchanged. For example, in a VPP scale of vpp, all layers of a model are divided into vpp parts, and after pipeline parallelism is performed on each part, the result of the last device is sent to the first device to continue pipeline parallelism on the next part until all parts are completed. For example, based on the foregoing example of pipeline parallelism, if the VPP scale vpp = 2, 12 network layers are divided into two parts, 0-5 layers for one part and 6-11 layers for the other part, where card 0 performs layers 0-1, card 1 performs layers 2-3, card 2 performs layers 4-5, card 0 performs layers 6-7, card 1 performs layers 8-9, and card 2 performs layers 10-11.

[0056] Micro batch pipeline parallelism is an extended training manner of pipeline parallelism. Generally, in model training, a sample group is used for one iteration to form a global batch, and the gradients generated based on the samples are accumulated together for weight update. The global batch is divided into multiple micro batches, each of which includes at least one sample. The micro batch is used as a basic unit for parallel scheduling, so that the utilization of devices in pipeline parallelism can be improved and the idle time of devices can be reduced. For example, in pipeline parallelism, the task performed by each accelerator is forward calculation or backward calculation of some layers based on a micro batch.

[0057] Bubble refers to the idle state of a device in pipeline parallelism, in which the device needs to wait for the execution result of another device before continuing to calculate.

[0058] The application scenario and implementation environment of the present application are introduced as follows.

[0059] The present application is applied to a scenario in which re-computation is used to reduce memory occupation of an accelerator during AI model training, and the type of AI model and the type of accelerator are not limited in the present application. The application scenario of the present application is introduced with reference to FIG. 1. FIG. 1 is a schematic diagram of an application scenario of a related technology and the present application.

[0060] As shown in (a) of FIG. 1, in the related art, taking an AI model including a module A and a module B as an example, in the model training process, the implementation of re-computation is as follows: the module A is taken as a module that needs to perform re-computation, in the forward computation process, the accelerator sequentially performs forward computation of the module A and the module B, wherein, in performing the forward computation of the module A, the accelerator discards the intermediate activation value of the module A, and stores the output activation value Output A of the module A; in the backward computation process, the accelerator performs backward computation of the module B based on the output activation value Output A of the module A stored in the forward computation process, then re-performs the forward computation of the module A once again to obtain the intermediate activation value and the output activation value Output A of the module A (i.e., performs re-computation of the module A), and then performs backward computation of the module A based on the intermediate activation value of the module A, after performing the backward computation of the module A, the intermediate activation value of the module A is discarded to reduce the memory occupation of the accelerator. It can be seen that, since the re-computation and the backward computation of the module A need to be performed together, and the backward computation of the module A is performed after the backward computation of the module B, in order to ensure that the accelerator can obtain the output activation value Output A of the module A when performing the backward computation of the module B, the accelerator must reserve the output activation value Output A of the module A when performing the forward computation of the module A, and reserve it until after the backward computation of the module B, which causes waste of memory resources of the accelerator.

[0061] As shown in (b) of FIG. 1, the present application provides an implementation of re-computation in the AI model training process, which can effectively reduce the memory peak of the accelerator and reduce the waste of memory resources. In the present application, the re-computation and the backward computation of the module A are split into two functions, that is, the re-computation is separated and implemented as a separate function, so that the forward computation, the backward computation and the re-computation involved in the AI model training process can be independently scheduled. For example, in the forward computation process, the accelerator sequentially performs forward computation of the module A and the module B, wherein, in performing the forward computation of the module A, the accelerator discards the intermediate activation value of the module A, and stores the output activation value Output A of the module A, and after performing the forward computation of the module B, the real storage of Output A is discarded, and Output A is reserved in the logical view, so as to ensure the accuracy of the calculation logic on the basis of releasing the memory space of the accelerator; in the backward computation process, the accelerator first performs re-computation of the module A to obtain the intermediate activation value and the output activation value Output AThe reverse calculation of module B and the reverse calculation of module A are sequentially performed, that is, when the accelerator performs the reverse calculation of module B, the accelerator can rely on the output activation value Output of module A obtained by re-computation to activate module B A In addition, after performing the reverse calculation of module A, the accelerator discards the intermediate activation value of module A, reducing the memory occupation of the accelerator. It can be seen that the technical solution provided in the present application increases the flexibility of re-computation, reduces the memory peak of the accelerator, and improves the model training performance.

[0062] It should be understood that the above Figure 1 is exemplified by taking two modules in the model as an example. In some scenarios, the model includes a larger number of modules, and the above content applies by analogy. For example, the model includes module A, module B, and module C. If module A is a module to be re-computed, when the embodiments provided in the present application are used, the re-computation of module A can be scheduled to be performed between the reverse calculation of module C and the reverse calculation of module B. In other words, since the reverse calculation of module B relies on the output activation value of module A, the re-computation of module A is scheduled to be performed before the reverse calculation of module B, or in other words, the re-computation of module A is scheduled to be performed after the accelerator reaches the memory peak. In this way, the memory occupation of the output activation value of module A on the accelerator can be reduced as much as possible, thereby improving the model training performance.

[0063] The implementation environment of the present application will be introduced below with reference to Figure 2. Figure 2 is a schematic diagram of an implementation environment provided in an embodiment of the present application. As shown in Figure 2, the implementation environment includes a computing system, which includes a host 100 and at least one accelerator 200. The host 100 and the at least one accelerator 200 are in communication connection.

[0064] The host 100 is used to provide AI services. In the embodiments of the present application, the host 100 can control the accelerator 200 to perform the calculation tasks involved in the AI model training process. For example, the host 100 sends the forward calculation tasks, the backward calculation tasks, and the re-computation tasks of each module in the AI model to the accelerator 200 in response to the training request of the AI model, so as to implement the forward calculation, the reverse calculation, and the re-computation of each module. This process can also be understood as loading the calculation tasks involved in the AI model training process into the accelerator 200 for execution. In addition, the number of hosts 100 can be one or more, which is not limited in the present application.

[0065] The accelerator card 200 is used to provide computing power for the training process of the AI model to accelerate the training process of the AI model. For example, the accelerator card 200 is a GPU, an XPU, an IPU, a TPU, a DSA chip, etc., and the present application is not limited thereto. Illustratively, the accelerator card 200 receives the computing task (such as a forward computing task, a backward computing task, a re-computing task, etc.) of the AI model sent by the host 100, calls the corresponding operator, processes the input data of the operator, and obtains the output data, that is, the execution result of the computing task. In addition, the number of accelerator cards 200 can be one or more, and the present application is not limited thereto. In the case where the number of accelerator cards 200 is N, the computing system has distributed computing capability and can execute the training request of the AI model in a pipeline parallel manner, and N is a positive integer. The pipeline parallelism includes but is not limited to naive pipeline parallelism, virtual pipeline parallelism, micro-batch pipeline parallelism, etc.

[0066] The host 100 and the accelerator card 200 described above can be integrated in one server, or can be separately arranged, and the present application is not limited thereto. Illustratively, taking the example that the host 100 and the accelerator card 200 are integrated in one server, the host 100 and the accelerator card 200 are connected in communication through a peripheral component interconnect express (PCIe) link, and the host 100 and the accelerator card 200 exchange data through the PCIe link. The server can be a standalone physical server, or a server cluster or distributed file system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, etc. basic cloud computing services. Taking the computing device as an example, the computing device can also be referred to as a cloud platform (i.e., a short name for a cloud computing platform), which refers to a service based on hardware resources and software resources, providing computing, network and storage capabilities. Through the network "cloud", huge data computing is processed and analyzed in the far end and returned to the user, with the characteristics of large scale, distribution, virtualization, high availability, scalability, on-demand service and security, etc. The cloud platform can realize the rapid allocation and release of configurable computing resources with small management cost or low interaction complexity between the user and the service provider.

[0067] In addition, the networks involved above include, but are not limited to, any combination of data center networks, storage area networks (SANs), local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), mobile, wired or wireless networks, private networks or virtual private networks. In some implementations, technologies and / or formats including hyper text markup language (HTML), extensible markup language (XML), etc. are used to represent data exchanged over the networks. In addition, all or part of the links can be encrypted using conventional encryption technologies such as secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), internet protocol security (IPsec), etc. In other embodiments, custom and / or proprietary data communications technologies can be employed in place of, or in addition to, the data communications technologies described above.

[0068] The hardware structure of the host 100 in the computing system is described below.

[0069] The present application provides a computing device, which can be configured as the host 100 described above. Referring to FIG. 3, FIG. 3 is a structural schematic diagram of a computing device provided by an embodiment of the present application. As shown in FIG. 3, the computing device 300 includes a memory 301, a processor 302, a communication interface 303 and a bus 304. The memory 301, the processor 302 and the communication interface 303 are communicatively connected to each other through the bus 304.

[0070] The memory 301 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM), or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium capable of storing desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto. Illustratively, the memory 301 is used to store at least one program code, and when the program code stored in the memory 301 is executed by the processor 302, the processor 302 is used to execute the steps performed by the host in the method embodiments described below.

[0071] The processor 302 can be a network processor (NP), a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or an integrated circuit for controlling program execution of the scheme of the present application. The processor 302 can be a single-CPU processor or a multi-CPU processor. The number of processors 302 can be one or more.

[0072] The communication interface 303 uses a transceiving module such as a transceiver to realize communication between the computing device 300 and other devices or communication networks. For example, data can be obtained through the communication interface 303.

[0073] The memory 301 and the processor 302 can be separately arranged or integrated together.

[0074] The bus 304 can include a path for transmitting information between various components (e.g., the memory 301, the processor 302, the communication interface 303) of the computing device 300.

[0075] Based on the above contents shown in FIG. 1 to FIG. 3, the application scenario and implementation environment provided by the present application are introduced, and the training method of the AI model provided by the present application is introduced below.

[0076] FIG. 4 is a flowchart of a training method of an AI model provided by an embodiment of the present application. As shown in FIG. 4, taking the case that the method is applied to a host in a computing system for example, the method schematically includes the following steps 401 to 403.

[0077] 401, in response to a training request of an AI model, performing forward calculation on a plurality of modules of the AI model, the plurality of modules including a first module and a second module, the first module being a module that needs to perform re-computation in a backward calculation process, and the second module being a module that needs to use output data of the first module in a forward calculation process.

[0078] In the embodiment of the present application, the computing system includes a host and an acceleration card, and the host can access a wired network or a wireless network to provide AI services for users. Performing forward calculation on a plurality of modules of the AI model means that the host sends a forward calculation task of each module to the acceleration card related to the training request, and controls the acceleration card to perform forward calculation of the corresponding module according to the received forward calculation task. Wherein, the AI model includes a plurality of modules, for any one module of the AI model, the module can be one or more network layers of the AI model, or a component, a basic element, etc. in a network layer, in addition, for any one module, the module can be a nonlinear module or a linear module, and the present application does not limit the division granularity and type of the module in the AI model.

[0079] The plurality of modules include a first module and a second module, wherein the first module is a module that needs to perform recalculation in the backward calculation process, that is, in the process of forward calculation of the first module, the intermediate activation value of the first module can not be retained, and in the subsequent reverse calculation process, the forward calculation of the first module is performed again to generate the intermediate activation value again. In this way, the memory occupation of the accelerator card is reduced. The second module is a module that needs to use the output data of the first module in the forward calculation process, that is, in the forward calculation process, the forward calculation of the second module depends on the output data of the first module, and the output data of the first module is the output activation value of the first module, and the same applies hereinafter, which will not be repeated. For example, the second module is the next module of the first module in the plurality of modules, and for another example, in some embodiments, the AI model includes a plurality of network layers, each accelerator card on which a training request is executed performs at least one network layer, the at least one network layer includes the plurality of modules executed in sequence, and the second module is the last module in the plurality of modules. For example, the AI model is a Transformer model, and the network layer is a Transformer layer, also known as a Transformer block. It should be understood that the Transformer block generally refers to the basic building layer in the Transformer model, which can include a multi-head self-attention mechanism (Multi-head Self-Attention), a feed forward neural network (Feed Forward Network), and normalization and residual connection components. A plurality of Transformer blocks are stacked in sequence to form a complete Transformer model to achieve effective feature extraction and representation learning on input sequences. Illustratively, taking the network layer executed on the accelerator card as an example, if the network layer includes modules A, B, C, and D executed in sequence, the first module can be module A or module B or module C, and the second module is module D.

[0080] Illustratively, the host, in response to a training request of the AI model, performs operator compilation based on a model framework of the AI model, that is, generates operators of the AI model. For example, the operator generation includes three processes, namely, input tensor description, weight data conversion, and output tensor description. In the input tensor description, the input dimension, memory size, and other information of each operator are calculated, and the form of the operator input data is defined. In the weight data conversion, the weight parameters used by the operator are processed in terms of data format, shape conversion, data compression, and the like. In the output tensor description, the output dimension, memory size, and other information of the operator are calculated. After the host performs operator compilation based on the model framework of the AI model, the host performs backend device interface adaptation on the operators (that is, adapts the upper-layer operators to the underlying hardware devices), identifies the properties of the operators, generates corresponding computing tasks according to the identified operators, and sends the computing tasks to the accelerator card related to the training request.

[0081] 402、After the forward calculation of the second module is performed, the output data of the first module is discarded.

[0082] In the embodiments of the present application, performing forward calculation on the second module means that the host sends a forward calculation task of the second module to the accelerator card, controls the accelerator card to perform forward calculation on the second module based on the output data of the first module according to the received forward calculation task. After the forward calculation of the second module is performed, the output data of the first module is discarded, that is, the host sends a discard task to the accelerator card after the forward calculation of the second module is performed, and controls the accelerator card to discard the output data of the first module based on the discard task. The discard task instructs to discard the real storage of the output data of the first module in the accelerator card, and to retain the logical view of the output data. In this way, the accuracy of the calculation logic can be ensured on the basis of releasing the memory space of the accelerator card. It should be understood that, since the output data of the first module will not be used temporarily after the forward calculation of the second module is performed, the discard task can be executed to discard the output data and release the memory space of the accelerator card.

[0083] 403、Performing backward calculation on the plurality of modules of the AI model, performing re-calculation on the first module before performing backward calculation on the second module, and performing backward calculation on the second module based on the data obtained by performing re-calculation on the first module.

[0084] In the embodiments of the present application, performing backward calculation on the plurality of modules of the AI model means that the host sends a backward calculation task of each module to the training request related acceleration card, and controls the acceleration card to perform backward calculation of the corresponding module according to the received backward calculation task. In this process, before performing backward calculation on the second module, the first module is re-calculated, and the backward calculation on the second module is performed based on the data obtained by re-calculating the first module, that is, before sending the backward calculation task of the second module to the acceleration card, the re-calculation task of the first module is sent to the acceleration card, so that the acceleration card performs the backward calculation task of the second module based on the data obtained by re-calculating the first module. In other words, the re-calculation of the first module is scheduled to be performed before the backward calculation of the second module. In this way, after re-calculating the first module, the backward calculation of the second module can rely on the data obtained by re-calculating the first module. That is, the output data of the first module does not need to be stored from the forward calculation process to the backward calculation process of the second module.

[0085] In the foregoing manner, when performing forward calculation on the plurality of modules of the AI model, for the first module to be re-calculated and the second module that relies on the output data of the first module, the output data of the first module is discarded after the forward calculation of the second module is performed, and when performing backward calculation on the plurality of modules, the re-calculation of the first module is scheduled to be performed before the backward calculation of the second module. In this way, when performing the backward calculation of the second module, the data obtained by re-calculating the first module can be relied on. Through this flexible re-calculation scheduling manner, the output data of the first module does not need to be stored from the forward calculation process to the backward calculation process, thereby saving the memory resources of the acceleration card.

[0086] In addition, based on the foregoing, for the first module in the AI model, the re-calculation of the first module is separated into two functions, that is, the re-calculation is separated and implemented as a separate function, so that the forward calculation, the backward calculation and the re-calculation involved in the AI model training process can be independently scheduled. Through this separate scheduling of re-calculation, the flexibility of re-calculation is improved, so that the user can schedule the re-calculation to a suitable position according to the business requirements, thereby reducing the memory occupation of the acceleration card in the AI model training process, or improving the model training performance. Moreover, the number of acceleration cards in the computing system can be one or multiple, that is, the implementation of the re-calculation in the AI model training process provided by the present application is applicable to a single acceleration card and also applicable to multiple acceleration cards. Based on this, the following describes several other AI model training methods provided by the present application with reference to the embodiments shown in FIGS. 5 to 11.

[0087] FIG. 5 is a flowchart of another method for training an AI model according to an embodiment of the present application. As shown in FIG. 5, the method is applied to a computing system as an example, and the method includes steps 501-503, which are described below with reference to the interaction between a host and an acceleration card.

[0088] 501. The host determines a plurality of modules of the AI model in response to a training request for the AI model, wherein the plurality of modules includes a first module to be executed for re-computation.

[0089] In the embodiments of the present application, the host can access a wired network or a wireless network to provide AI services for users. The AI model includes a plurality of modules. For any one of the modules of the AI model, the module can be one or more network layers of the AI model, or a component or a basic element in a network layer. In addition, for any one of the modules, the module can be a nonlinear module or a linear module. The present application does not limit the granularity and type of the modules in the AI model.

[0090] The plurality of modules includes a first module, which is a module to be executed for re-computation. That is, during forward computation of the first module, the intermediate activation values of the first module are not retained, and during subsequent backward computation, the forward computation of the first module is re-executed to generate the intermediate activation values again. In this way, the memory occupation of the acceleration card is reduced. In addition, the present application does not limit the position of the first module in the AI model. The first module can be determined according to service requirements. For example, the first module is the first module in the plurality of modules, or each module in the AI model is the first module, and the like. It should be understood that the first module in the present embodiment is used to refer to a module to be executed for re-computation, which can be one or more, and the present application does not limit this.

[0091] 502. The host generates a forward computation task for each module of the plurality of modules, a backward computation task for each module, and a re-computation task for the first module.

[0092] In the embodiments of the present application, for any one of the plurality of modules, the host generates a forward computation task and a backward computation task for the module, and for the first module, the host generates a re-computation task for the first module. For the forward computation task of the first module, the forward computation task indicates that the forward computation of the first module is performed, and after the forward computation, the intermediate activation values of the first module are discarded, and the output data of the first module is stored. This process is also referred to as splitting the re-computation and the backward computation of the first module in the AI model into two functions, and separating the re-computation as a separate function. In this way, the forward computation, the backward computation, and the re-computation involved in the training process of the AI model can be independently scheduled.

[0093] In addition, the application does not limit the timing of the host generating the calculation tasks corresponding to each module. The host can generate each calculation task in sequence according to the training framework of the AI model, and in the case of generating any one calculation task, the host can assign the calculation task to the accelerator card related to the training request for execution. The host can also generate the calculation tasks corresponding to each module according to the training framework of the AI model, and then assign the calculation tasks to the accelerator card related to the training request in sequence according to the execution sequence of each calculation task, and so on. This process can be set according to business needs.

[0094] 503、The host sends a re-computation task of the first module to the accelerator card before sending the first back-computation task to the accelerator card related to the training request. The first back-computation task and the re-computation task have a dependency relationship.

[0095] In the embodiments of the application, the host and the accelerator card are in communication connection, and the host can control the accelerator card to execute corresponding calculation tasks. The first back-computation task is a back-computation task in the back-computation tasks of the plurality of modules, and the dependency relationship between the first back-computation task and the re-computation task means that the execution of the first back-computation task depends on the execution result of the re-computation task. Illustratively, the first back-computation task can be determined according to the training framework of the AI model. Before the host sends the first back-computation task to the accelerator card, the host sends a re-computation task of the first module to the accelerator card, so that the accelerator card can rely on the execution result of the re-computation task when executing the first back-computation task. Through this separate scheduling of the re-computation task, the flexibility of re-computation is improved, so that the user can schedule the re-computation task to a suitable position according to business needs, thereby reducing the memory occupation of the accelerator card in the AI model training process, or improving the model training performance.

[0096] In summary, in the AI model training method provided in the embodiments of the application, after determining the plurality of modules of the AI model according to the training request of the AI model, the forward computation tasks and the back-computation tasks corresponding to each module are generated, and for a first module in the plurality of modules to be executed for re-computation, a re-computation task of the first module is separately generated, and when assigning the calculation tasks to the accelerator card related to the training request, the re-computation task is scheduled before the first back-computation task having a dependency relationship with the re-computation task. Through this separate scheduling of the re-computation task, the flexibility of re-computation is effectively improved.

[0097] Based on the embodiment shown in the foregoing FIG. 5, the application provides an AI model training method for separately scheduling a re-computation task. The first back-computation task can be determined according to the training framework of the AI model. The following takes different scenarios as examples to further illustrate the AI model training method.

[0098] FIG. 6 is a flowchart of another method for training an AI model according to an embodiment of the present application. As shown in FIG. 6, the method is applied to a computing system including a host and an acceleration card, and the method includes the following steps 601-614.

[0099] 601. The host determines a plurality of modules of the AI model in response to a training request of the AI model, wherein the plurality of modules include a first module to be executed for re-computation.

[0100] The plurality of modules further include a second module, and the second module is a module that needs to use the output data of the first module in the forward computation process, for example, the second module is a next module of the first module in the plurality of modules.

[0101] 602. The host generates a forward computation task of each module in the plurality of modules, a backward computation task of each module, and a re-computation task of the first module.

[0102] 603. The host sends the forward computation task of the first module to the acceleration card related to the training request.

[0103] 604. The acceleration card executes the forward computation task of the first module.

[0104] The acceleration card performs forward computation on the first module to obtain intermediate activation values and output data of the first module, discards the intermediate activation values of the first module, and stores the output data of the first module into the memory of the acceleration card.

[0105] 605. The host sends the forward computation task of the second module to the acceleration card.

[0106] The forward computation task of the second module indicates to perform forward computation on the second module of the AI model.

[0107] 606. The acceleration card executes the forward computation task of the second module.

[0108] The acceleration card performs forward computation on the second module based on the output data of the first module to obtain intermediate activation values and output data of the second module, and stores the intermediate activation values and the output data of the second module into the memory of the acceleration card. It should be noted that the second module can also be a module to be executed for re-computation, for example, if the second module and the first module are both modules to be executed for re-computation, then in this step, the acceleration card discards the intermediate activation values of the second module after obtaining the intermediate activation values and the output data of the second module, and stores the output data of the second module into the memory of the acceleration card, which is not limited in the present application.

[0109] 607. The host sends a discard task to the acceleration card, and the discard task indicates to discard the output data of the first module.

[0110] In the embodiments of the present application, the discarding task instructing to discard the output data of the first module refers to discarding the real storage of the output data of the first module in the acceleration card and retaining the logical view of the output data, so that the accuracy of the computing logic can be ensured on the basis of releasing the memory space of the acceleration card.

[0111] 608. The acceleration card executes the discarding task.

[0112] It should be understood that, since the output data of the first module will not be used temporarily after the acceleration card executes the forward computing task of the second module, the discarding task can be executed to discard the output data, so as to release the memory space of the acceleration card.

[0113] 609. The host sends a recalculation task of the first module to the acceleration card, and the recalculation task instructs to perform forward calculation on the first module again.

[0114] 610. The acceleration card executes the recalculation task.

[0115] In this way, the acceleration card performs forward calculation on the first module again, and obtains the intermediate activation value and the output data of the first module again, that is, the execution result of the recalculation task includes the output data of the first module.

[0116] 611. The host sends a first backward calculation task to the acceleration card, and the first backward calculation task is a backward calculation task of the second module.

[0117] 612. The acceleration card executes the first backward calculation task.

[0118] In this way, the acceleration card performs backward calculation on the second module based on the output data of the first module and the intermediate activation value of the second module stored in the memory. In this way, the output data of the first module relied on by the acceleration card when performing backward calculation on the second module is obtained through the recalculation task of the first module, instead of being stored in the forward calculation process, that is, the output data of the first module does not need to be stored from the forward calculation process to the backward calculation process of the second module.

[0119] 613. The host sends a backward calculation task of the first module to the acceleration card.

[0120] 614. The acceleration card executes the backward calculation task of the first module.

[0121] In this way, the acceleration card performs backward calculation on the first module based on the intermediate activation value of the first module stored in the memory. It should be understood that, if the first module is the first module of the AI model, the acceleration card also needs to combine the sample input into the AI model when executing the first backward calculation task; if there are other modules before the first module, the acceleration card also needs to combine the output data of the other modules when executing the first backward calculation task, which will not be described herein.

[0122] In steps 603 to 614, the first module and the second module in the AI model are taken as examples for description. It should be understood that in some embodiments, if the AI model further includes more modules, between the foregoing steps 608 and 609, steps of sequentially sending, by the host, forward calculation tasks of other modules and backward calculation tasks of the other modules to the acceleration card are further included.

[0123] In the foregoing manner, after determining the plurality of modules of the AI model according to the training request of the AI model, the forward calculation task and the backward calculation task corresponding to each module are generated, and for the first module to be executed for re-computation in the plurality of modules, the re-computation task of the first module is separately generated, and when the calculation task is issued to the acceleration card related to the training request, the re-computation task is scheduled before the backward calculation task of the second module having a dependency relationship with the re-computation task. Through this separate scheduling manner of the re-computation task, the flexibility of re-computation is effectively improved, and when the acceleration card executes the backward calculation task of the second module, the output data of the first module obtained based on the re-computation task can be relied on. Thus, after the acceleration card executes the forward calculation task of the second module, the output data of the first module can be discarded in time, the memory space of the acceleration card is released, and the memory occupation of the acceleration card is reduced.

[0124] In addition, in the embodiment shown in FIG. 6, the second module is the next module of the first module, and in other embodiments, the second module is a subsequent module of the first module. In some embodiments, the AI model includes a plurality of network layers, each of which is executed on an acceleration card related to the training request to perform at least one network layer, which includes the aforementioned plurality of modules executed in sequence, and the second module is the last module in the plurality of modules. For example, the AI model is a Transformer model, and the network layer is a Transformer layer, also known as a Transformer block. It should be understood that the Transformer block generally refers to the basic building layer in the Transformer model, which can include components such as multi-head self-attention mechanism, feed forward network, normalization, and residual connection. A plurality of Transformer blocks are stacked in sequence to form a complete Transformer model to achieve effective feature extraction and representation learning on input sequences. Illustratively, taking any one of the Transformer blocks in the Transformer model as an example, if the network layer executed on the acceleration card includes modules A, B, C, and D executed in sequence, the first module can be module A, B, or C, and the second module is module D. Based on this, by using the above method, the recalculation of at least one of modules A, B, and C is scheduled before the backpropagation calculation task of module D, which can reduce the memory occupancy of the acceleration card as much as possible, thereby improving the model training performance.

[0125] FIG. 7 is a flowchart of another method for training an AI model according to an embodiment of the present application. As shown in FIG. 7, the method is applied to a computing system including a host and N acceleration cards. Illustratively, the method includes the following steps 701-703.

[0126] 701. The host determines a plurality of modules of an AI model in response to a training request of the AI model, wherein the plurality of modules includes a first module to be executed for recalculation.

[0127] In the embodiments of the present application, the host is in communication connection with N acceleration cards, and the N acceleration cards execute the training request of the AI model in a pipeline parallel manner. N is a positive integer. The pipeline parallel manner includes but is not limited to naive pipeline parallel, virtual pipeline parallel, micro-batch pipeline parallel, etc. The plurality of modules of the AI model are executed by the N acceleration cards to perform corresponding calculation tasks.

[0128] 702. The host generates a forward calculation task of each module in the plurality of modules, a backpropagation calculation task of each module, and a recalculation task of the first module.

[0129] To more clearly explain the process in which the N acceleration cards execute the training request in a pipelined parallel manner, in this embodiment, the training method of the AI model is introduced with reference to FIG. 8. FIG. 8 is a schematic diagram of a training method of an AI model based on pipelined parallelism provided in an embodiment of the present application. As shown in FIG. 8, the computing system includes N acceleration devices, N = 4, and the N acceleration devices are respectively denoted as card 0, card 1, card 2, and card 3. The N acceleration cards are taken as an example to introduce the pipelined parallelism based on micro-batches and virtual pipelines, where the micro-batch b = 8, the virtual pipeline vpp = 2, and the AI model includes 8 layers. In the figure, F represents forward calculation, B represents backward calculation, and R represents re-computation (for example, re-computation is performed on each layer). The 8 network layers are divided into two parts, the first layer to the fourth layer are one part, and the fifth layer to the eighth layer are another part. Card 0 performs the first layer and the fifth layer, card 1 performs the second layer and the sixth layer, card 2 performs the third layer and the seventh layer, and card 3 performs the fourth layer and the eighth layer. The box without filling represents the network layer in front of the model (i.e., the first layer to the fourth layer), the box with filling represents the network layer behind the model (i.e., the fifth layer to the eighth layer), and the numbers 1-8 in the box represent the 8 micro-batches b1 to b8. It should be noted that FIG. 8 is used to introduce the principle of the scheme, and the size of each box shown in the figure is only used for illustration and does not constitute a limitation on the execution time of each computing task. For example, in actual application, the execution time of the backward calculation task is usually twice the execution time of the forward calculation task.

[0130] As shown in FIG. 8, in the timing corresponding to card 0, the box without filling F1 represents forward calculation of the first layer based on the micro-batch b1, the box with filling F1 represents forward calculation of the fifth layer based on the micro-batch b1, the box without filling B1 represents backward calculation of the first layer based on the micro-batch b1, the box with filling B1 represents backward calculation of the fifth layer based on the micro-batch b1, the box without filling R1 represents re-computation of the first layer based on the micro-batch b1, and the box with filling R1 represents re-computation of the fifth layer based on the micro-batch b1.

[0131] For example, taking the micro-batch b1 as an example, if the first module is the 1st layer of the AI model, the forward calculation task of the first module is the non-padding block F1 in the time sequence corresponding to card 0, and the re-computation task of the first module is the non-padding block R1 in the time sequence corresponding to card 0; if the first module is the 5th layer of the AI model, the forward calculation task of the first module is the padding block F1 in the time sequence corresponding to card 0, and the re-computation task of the first module is the padding block R1 in the time sequence corresponding to card 0; if the first module is the 2nd layer of the AI model, the forward calculation task of the first module is the non-padding block F1 in the time sequence corresponding to card 1, and the re-computation task of the first module is the non-padding block R1 in the time sequence corresponding to card 1; if the first module is the 6th layer of the AI model, the forward calculation task of the first module is the padding block F1 in the time sequence corresponding to card 1, and the re-computation task of the first module is the padding block R1 in the time sequence corresponding to card 1, and so on, which will not be described herein again.

[0132] 703、The host sends the re-computation task of the first module to the first accelerator card before sending the first backward calculation task of the first module to the first accelerator card, the first backward calculation task being a backward calculation task of the first module.

[0133] The first accelerator card is configured to perform a calculation task related to the first module.

[0134] For example, as shown in FIG. 8, if the first module is the first layer of the AI model, the first backward calculation task is the non-padding block B1 in the time sequence corresponding to card 0, and the re-computation task of the first module is the non-padding block R1 in the time sequence corresponding to card 0. It should be understood that, since the re-computation task is separately scheduled in the process of executing the training request in a pipeline parallel manner by the N acceleration cards, the re-computation task does not need to wait for the execution result of the previous acceleration card, and only the backward calculation task of the first module needs to wait for the execution result of the previous acceleration card. Illustratively, the host sends a second backward calculation task to the second acceleration card in the N acceleration cards, the second acceleration card is used to execute a calculation task related to a third module, the third module is the next module of the first module, and the second backward calculation task is a backward calculation task of the third module; and the host sends the first backward calculation task to the first acceleration card, the first backward calculation task instructs to perform backward calculation on the first module based on the execution result of the second backward calculation task. In this way, the bubbles of the pipeline can be utilized for re-computation, thereby reducing the waste of computing power. For example, continuing to refer to FIG. 8, taking the first acceleration card as card 0 and the second acceleration card as card 1 as an example, since the padding block R6 (re-computation of the fifth layer based on the micro batch b6) and the padding block B6 (backward calculation of the fifth layer based on the micro batch b6) are separately scheduled, the card 0 can immediately execute the padding block R6 (re-computation of the fifth layer based on the micro batch b6) after executing the padding block B5 (backward calculation of the fifth layer based on the micro batch b5), and then the card 0 executes the padding block B6 (backward calculation of the fifth layer based on the micro batch b6) corresponding to the card 0 after receiving the execution result of the padding block B6 (backward calculation of the sixth layer based on the micro batch b6 corresponding to the time sequence of the card 1) sent by the card 1 after execution. Compared with the related art, the bubbles as indicated by the arrows in the figure are filled, that is, in the related art, the padding block R6 and the padding block B6 corresponding to the card 0 are bound together, and the two tasks need to wait for the execution result of the padding block B6 corresponding to the card 1 to be sent by the card 1 before being executed. Obviously, compared with the related art, the technical solution provided in the present application can effectively reduce the bubbles in the pipeline parallel process, thereby improving the utilization rate of the acceleration card.

[0135] FIG. 9 is a flowchart of another training method of an AI model provided by an embodiment of the present application. As shown in FIG. 9, the method is applied to a computing system including a host and M acceleration cards. Illustratively, the method includes the following steps 901 to 903.

[0136] 901. The host determines a plurality of modules of an AI model in response to a training request of the AI model, wherein the plurality of modules includes a first module to be executed for re-computation.

[0137] The M accelerator cards execute the training request of the AI model in a virtual pipeline parallel manner. M is a positive integer. Illustratively, the host determines the first module based on the number of virtual pipelines, the number of pipelines, and the number of the M accelerator cards. In this way, the host determines part of the modules in the AI model as the modules to be executed for re-computation, thereby releasing the memory space of the accelerator card.

[0138] To more clearly explain the process in which the M accelerator cards execute the training request in a virtual pipeline parallel manner, in this embodiment, the training method of the AI model is introduced with reference to FIG. 10. FIG. 10 is a schematic diagram of a training method of an AI model based on virtual pipeline parallelism provided in an embodiment of the present application. As shown in FIG. 10, the computing system includes M accelerator devices, M = 4, and the M accelerator devices are respectively denoted as card 0, card 1, card 2, and card 3. The M accelerator cards are taken as an example to introduce the parallelism based on micro-batches and virtual pipelines, where the micro-batch b = 8, the virtual pipeline vpp = 2, and the AI model includes 8 layers. In the figure, F represents forward computation, B represents backward computation, and R represents re-computation. The 8 network layers are divided into two parts, the first layer to the fourth layer as one part and the fifth layer to the eighth layer as the other part. Card 0 executes the first layer and the fifth layer, card 1 executes the second layer and the sixth layer, card 2 executes the third layer and the seventh layer, and card 3 executes the fourth layer and the eighth layer. The box without filling represents the network layer in the front of the model (i.e., the first layer to the fourth layer), the box with filling represents the network layer in the back of the model (i.e., the fifth layer to the eighth layer), and the numbers 1-8 in the box represent 8 micro-batches b1 to b8. It should be noted that FIG. 10 is used to introduce the principle of the scheme, and the size of each box in the figure is only used for illustration and does not constitute a limitation on the execution time of each computing unit. For example, in actual application, the execution time of the backward computation task is usually greater than the execution time of the forward computation task, for example, the execution time of the backward computation task is twice the execution time of the forward computation task.

[0139] As shown in FIG. 10, in the timing corresponding to card 0, the available memory capacity reaches the peak value when card 0 executes the forward computation of the first layer based on the micro-batch b7 (corresponding to the box without filling F7), and the available memory capacity reaches a certain threshold value when card 0 executes the forward computation of the first layer based on the micro-batch b5 (corresponding to the box without filling F5). Therefore, the modules corresponding to the box without filling F5, the box without filling F6, and the box without filling F7 can be determined as the first module, and the re-computation is performed on the first module.

[0140] 902、The host generates the forward computation task of each module in the plurality of modules, the backward computation task of each module, and the re-computation task of the first module.

[0141] For example, referring to FIG. 10, in the timing corresponding to card 0, the non-padding block R5, the non-padding block R6, and the non-padding block R7 are all re-computation tasks of the first module.

[0142] 903、The host sends a re-computation task to a third accelerator card in the M accelerator cards before sending a first backward computation task to the third accelerator card, the first backward computation task being a backward computation task of a first module, and the third accelerator card being configured to execute a computation task related to the first module.

[0143] For example, referring to the content shown in FIG. 10, if the third accelerator card is card 0, and the first backward computation task is the non-padding block B5, the non-padding block B6, and the non-padding block B7 in the timing corresponding to card 0. It should be understood that, because part of the modules are determined to be re-computed in the process of executing the training request in a virtual pipeline parallel manner by the M accelerator cards, the accelerator card can discard the intermediate activation value of the first module when executing the forward computation task of the first module. In this way, the memory space of the accelerator card can be released. Illustratively, the host sends the forward computation task of the first module to the third accelerator card, and the forward computation task of the first module indicates that the intermediate activation value of the first module is discarded after the forward computation of the first module is performed.

[0144] In addition, by the above method, when the training request is executed in a virtual pipeline parallel manner by the M accelerator cards, re-computation is performed on part of the modules, and the bubbles of the pipeline can also be utilized for re-computation, thereby reducing the waste of computing power. For example, continuing to refer to FIG. 10, taking card 0 as the third accelerator card as an example, for card 0, the non-padding block R5 (re-computation of the first layer based on the micro batch b5) can be scheduled to be executed at the bubble before the non-padding block B3, the non-padding block R6 (re-computation of the first layer based on the micro batch b6) can be scheduled to be executed at the bubble before the non-padding block B4, and the non-padding block R7 (re-computation of the first layer based on the micro batch b7) can be scheduled to be executed at the bubble before the non-padding block B5. In this way, when the non-padding block F5 (forward computation of the first layer based on the micro batch b5), the non-padding block F6 (forward computation of the first layer based on the micro batch b6), and the non-padding block F7 (forward computation of the first layer based on the micro batch b7) are executed, the intermediate activation value of the first layer can be discarded, and the memory space of the accelerator card can be released without changing the performance. Of course, in actual applications, the scheduling strategy of re-computation can be determined according to business requirements, which is not limited in the present application.

[0145] In some embodiments, for any one of the M accelerator cards, the host adopts the following scheduling manner to issue tasks to the accelerator card, so as to achieve the pipeline parallel execution effect as shown in FIG. 10. Illustratively, the number of pipelines is denoted as pp, the number of virtual pipelines is denoted as vpp, the number of micro-batches is denoted as b, and any one of the accelerator cards is denoted as the i th accelerator card (i ranges from 0 to M-1), and the scheduling manner includes the following stages:

[0146] Stage 1: The host sends X forward calculation tasks to the accelerator card based on the number of pipelines and the number of virtual pipelines, and the X forward calculation tasks are not subsequently recalculated. Illustratively, X = min (pp x vpp + pp - 2 - 2i, pp x vpp). Referring to FIG. 11, the i th accelerator card is taken as card 0, and the X forward calculation tasks are the forward calculations corresponding to the first 8 boxes.

[0147] Stage 2: The host sends Y forward calculation tasks to the accelerator card based on the number of pipelines, and the Y forward calculation tasks need to be subsequently recalculated. Illustratively, Y = max (pp - 2 - 2i, 0). Referring to FIG. 11, the i th accelerator card is taken as card 0, and the Y forward calculation tasks are the non-padding boxes F5 (forward calculation based on micro-batch b5 for the first layer) and the non-padding box F6 (forward calculation based on micro-batch b6 for the first layer).

[0148] Stage 3: The host sends Z first calculation task groups to the accelerator card based on the number of pipelines, the number of micro-batches, and the number of virtual pipelines, and one first calculation task group includes one forward calculation task and one reverse calculation task. Illustratively, Z = b x vpp - pp x vpp - pp + 2 + 2i. Wherein, if pp - 2 - 2i ≥ 0, the forward calculation task of the first first calculation task group in the Z first calculation task groups needs to be executed. Referring to FIG. 11, the i th accelerator card is taken as card 0, and the Z first calculation task groups are as indicated by the arrows in the figure, a total of 6 groups.

[0149] Stage 4: The host sends W second calculation task groups to the accelerator card based on the number of pipelines, and one second calculation task group includes one recalculation task and one reverse calculation task. Illustratively, W = max (pp - 1 - 2i, 0). Referring to FIG. 11, the i th accelerator card is taken as card 0, and the W second calculation task groups are as indicated by the arrows in the figure, a total of 3 groups.

[0150] Stage 5: The host sends K reverse calculation tasks to the accelerator card based on the number of pipelines and the number of virtual pipelines. Illustratively, K = min (pp x vpp - 1, pp x vpp + pp - 2 - 2i). Referring to FIG. 11, the i th accelerator card is taken as card 0, and the K reverse calculation tasks are as indicated by the arrows in the figure, a total of 7.

[0151] It should be noted that the above scheduling manner is only for example, in some embodiments, the module that needs to be re-computed can be set according to the business requirement, and the execution time of the re-computation task is set, etc., which is not limited in the present application.

[0152] In addition, with reference to FIG. 12, the present application also provides a training device of an AI model, which is used to implement the foregoing training method of the AI model. FIG. 12 is a structural schematic diagram of a training device of an AI model according to an embodiment of the present application. As shown in FIG. 12, the device comprises a forward calculation execution unit 1201, a discard unit 1202, and a reverse calculation execution unit 1203.

[0153] The forward calculation execution unit 1201 is configured to perform forward calculation on a plurality of modules of the AI model in response to a training request of the AI model, the plurality of modules comprising a first module and a second module, the first module being a module that needs to perform re-computation in a backward calculation process, and the second module being a module that needs to use output data of the first module in a forward calculation process.

[0154] The discard unit 1202 is configured to discard the output data of the first module after the forward calculation of the second module is completed.

[0155] The reverse calculation execution unit 1203 is configured to perform reverse calculation on the plurality of modules of the AI model, perform re-computation on the first module before performing reverse calculation on the second module, and perform reverse calculation on the second module based on the data obtained by performing re-computation on the first module.

[0156] In some embodiments, the second module is a next module of the first module in the plurality of modules.

[0157] In some embodiments, the AI model comprises a plurality of network layers, at least one network layer is executed on each acceleration card, the at least one network layer comprises a plurality of modules that are sequentially executed, and the second module is a last module in the plurality of modules.

[0158] In some embodiments, the AI model is a Transformer model, and the network layer is a Transformer layer.

[0159] It should be noted that the training device of the AI model provided in the foregoing embodiments is only used to illustrate the division of the above-mentioned functional modules when training the AI model, and in actual applications, the above-mentioned functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the training device of the AI model provided in the foregoing embodiments and the training method of the AI model belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.

[0160] Referring to FIG. 13, the application also provides a training device of an AI model, which is used to implement the foregoing training method of the AI model. FIG. 13 is a structural schematic diagram of another training device of an AI model provided by an embodiment of the application. As shown in FIG. 13, the device comprises a determination unit 1301, a generation unit 1302 and a sending unit 1303.

[0161] The determination unit 1301 is configured to, in response to a training request of an artificial intelligence (AI) model, determine a plurality of modules of the AI model, the plurality of modules comprising a first module to be executed for re-computation.

[0162] The generation unit 1302 is configured to generate a forward computation task of each module in the plurality of modules, a backward computation task of each module and a re-computation task of the first module.

[0163] The sending unit 1303 is configured to, before sending a first backward computation task to an acceleration card related to the training request, send the re-computation task to the acceleration card, wherein a dependency relationship exists between the first backward computation task and the re-computation task.

[0164] In some embodiments, the training request is executed in a pipeline parallel manner by N acceleration cards, N being a positive integer.

[0165] The sending unit 1303 is configured to, before sending a first backward computation task to a first acceleration card in the N acceleration cards, send the re-computation task to the first acceleration card, wherein the first backward computation task is a backward computation task of the first module, and the first acceleration card is configured to execute a computation task related to the first module.

[0166] In some embodiments, the sending unit 1303 is further configured to:

[0167] send a second backward computation task to a second acceleration card in the N acceleration cards, wherein the second acceleration card is configured to execute a computation task related to a third module, the third module being a next module of the first module, and the second backward computation task being a backward computation task of the third module;

[0168] send the first backward computation task to the first acceleration card, wherein the first backward computation task indicates that the first module is subjected to backward computation based on an execution result of the second backward computation task.

[0169] In some embodiments, the training request is executed in a virtual pipeline parallel manner by M acceleration cards, M being a positive integer.

[0170] The determination unit 1301 is configured to, in response to the training request, determine the first module based on a number of virtual pipelines, a number of pipelines and a number of the M acceleration cards.

[0171] The sending unit 1303 is configured to send a recalculation task to the third accelerator card before sending a first backward calculation task to the third accelerator card among the M accelerator cards, the first backward calculation task being a backward calculation task of the first module, and the third accelerator card being configured to execute a calculation task related to the first module.

[0172] The sending unit 1303 is further configured to send a forward calculation task of the first module to the third accelerator card, the forward calculation task of the first module indicating discarding intermediate activation values of the first module after forward calculation of the first module.

[0173] It should be noted that the training device of the AI model provided in the above embodiments only divides the above functions for example when training the AI model, and in actual application, the above functions can be completed by different function modules according to needs, that is, the internal structure of the device is divided into different function modules to complete all or part of the functions described above. In addition, the training device of the AI model provided in the above embodiments and the training method of the AI model embodiment belong to the same concept, and the specific implementation process is described in the method embodiment, which will not be repeated here.

[0174] The terms "first", "second", and the like are used in the present application to distinguish between items or similar items having substantially the same function and action. It should be understood that there is no logical or chronological dependency between "first", "second", and "n", and the number and execution order are not limited. It should also be understood that although the following description uses the terms first, second, and the like to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of various described examples, the first accelerator card can be referred to as the second accelerator card, and similarly, the second accelerator card can be referred to as the first accelerator card. The first accelerator card and the second accelerator card can both be accelerator cards, and in some cases, can be separate and different accelerator cards.

[0175] In the present application, the term "at least one" means one or more, and the term "multiple" in the present application means two or more, for example, multiple accelerator cards means two or more accelerator cards.

[0176] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0177] In the above embodiments, all or part of the steps can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the steps can be implemented in the form of program structure information. The program structure information includes one or more program instructions. When the program instructions are loaded and executed on a computing device, all or part of the procedures or functions in the embodiments of the present application are generated.

[0178] A person of ordinary skill in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or by a program instructing the relevant hardware, and the program can be stored in a computer readable storage medium, such as a read-only memory, a magnetic disk or an optical disk.

[0179] The above description and the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the above embodiments of the present application have been described in detail, a person of ordinary skill in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training an artificial intelligence (AI) model, characterized in that, The method includes: In response to the training request of the AI ​​model, forward computation is performed on multiple modules of the AI ​​model. The multiple modules include a first module and a second module. The first module is the module that needs to be recomputed during the backward computation process, and the second module is the module that needs to use the output data of the first module during the forward computation process. After the forward computation of the second module is completed, the output data of the first module is discarded; Inverse computation is performed on multiple modules of the AI ​​model. Before performing inverse computation on the second module, the first module is recomputed. Based on the data obtained from recomputing the first module, inverse computation is performed on the second module.

2. The method according to claim 1, characterized in that, The second module is the module following the first module among the plurality of modules.

3. The method according to claim 1, characterized in that, The AI ​​model includes multiple network layers, with at least one network layer executed on each accelerator card. The at least one network layer includes the multiple modules executed sequentially, and the second module is the last module among the multiple modules.

4. The method according to claim 3, characterized in that, The AI ​​model is a Transformer model, and the network layer is a Transformer layer.

5. A method for training an artificial intelligence (AI) model, characterized in that, The method includes: In response to a training request from an artificial intelligence (AI) model, multiple modules of the AI ​​model are identified, including a first module to be recomputed. Generate the forward computation task for each of the multiple modules, the backward computation task for each of the modules, and the recomputation task for the first module. Before sending the first reverse computation task to the accelerator card associated with the training request, the recomputation task is sent to the accelerator card, and there is a dependency between the first reverse computation task and the recomputation task.

6. The method according to claim 5, characterized in that, The training request is executed by N accelerator cards in a pipelined parallel manner, where N is a positive integer; The step of sending the recomputation task to the accelerator card before sending the first reverse computation task to the accelerator card related to the training request includes: sending the recomputation task to the first accelerator card among the N accelerator cards before sending the first reverse computation task to the first accelerator card, wherein the first reverse computation task is the reverse computation task of the first module, and the first accelerator card is used to execute the computation task related to the first module.

7. The method according to claim 6, characterized in that, The method further includes: A second reverse computation task is sent to the second accelerator card among the N accelerator cards. The second accelerator card is used to execute computation tasks related to the third module. The third module is the next module after the first module. The second reverse computation task is the reverse computation task of the third module. The first reverse computation task is sent to the first accelerator card. The first reverse computation task indicates that the first module is reverse computed based on the execution result of the second reverse computation task.

8. The method according to claim 5, characterized in that, The training request is executed by M accelerator cards in parallel via a virtual pipeline, where M is a positive integer. In response to a training request from an artificial intelligence (AI) model, determining multiple modules of the AI ​​model includes: in response to the training request, determining the first module based on the number of virtual pipelines, the number of pipelines, and the number of the M accelerator cards; The step of sending the recomputation task to the accelerator card before sending the first reverse computation task to the accelerator card related to the training request includes: sending the recomputation task to the third accelerator card among the M accelerator cards before sending the first reverse computation task to the third accelerator card, wherein the first reverse computation task is the reverse computation task of the first module, and the third accelerator card is used to execute the computation task related to the first module. The method further includes: sending a forward computation task of the first module to the third accelerator card, wherein the forward computation task of the first module indicates that after performing forward computation on the first module, the intermediate activation value of the first module is discarded.

9. A training device for an artificial intelligence (AI) model, characterized in that, The device includes: A forward computation execution unit is used to perform forward computation on multiple modules of the AI ​​model in response to the training request of the AI ​​model. The multiple modules include a first module and a second module. The first module is the module that needs to be recomputed during the backward computation process, and the second module is the module that needs to use the output data of the first module during the forward computation process. The discard unit is used to discard the output data of the first module after the forward computation of the second module is completed; The reverse computation execution unit is used to perform reverse computation on multiple modules of the AI ​​model. Before performing reverse computation on the second module, it performs recomputation on the first module and performs reverse computation on the second module based on the data obtained from recomputing the first module.

10. The apparatus according to claim 9, characterized in that, The second module is the module following the first module among the plurality of modules.

11. The apparatus according to claim 9, characterized in that, The AI ​​model includes multiple network layers, with at least one network layer executed on each accelerator card. The at least one network layer includes the multiple modules executed sequentially, and the second module is the last module among the multiple modules.

12. The apparatus according to claim 11, characterized in that, The AI ​​model is a Transformer model, and the network layer is a Transformer layer.

13. A computing device, characterized in that, The computing device includes a processor and a memory, the processor being configured to execute at least one piece of program code stored in the memory to enable the computing device to implement the training method of the AI ​​model as described in any one of claims 1 to 8.

14. A computing system, characterized in that, The computing system includes a host and at least one accelerator card, and the system is used to implement the training method of the AI ​​model as described in any one of claims 1 to 8.

15. A computer program product, characterized in that, When the computer program product is run on a computing device, the computing device enables the training method of the AI ​​model as described in any one of claims 1 to 8.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store at least one piece of program code, which is used to implement the training method of the AI ​​model as described in any one of claims 1 to 8.

Images