Artificial intelligence (AI) model training method and device and computing equipment

Abstract

The invention discloses a training method and device of an artificial intelligence AI model and computing equipment, and relates to the technical field of artificial intelligence AI. The method comprises the steps that when forward computation is executed on multiple modules of the AI model, the forward computation is executed on a first module to be subjected to re-computation and a second module depending on the first module to output data; after the execution of the forward calculation of the second module is completed, the output data of the first module is discarded, and when the reverse calculation of the plurality of modules is executed, the re-calculation of the first module is scheduled to be executed before the reverse calculation of the second module, so that when the reverse calculation of the second module is executed, the data obtained by the re-calculation of the first module can be depended, and the calculation efficiency of the second module is improved. Through the flexible re-calculation scheduling mode, the output data of the first module does not need to be stored from the forward calculation process to the reverse calculation process, and memory resources of the acceleration card are saved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence (AI), and particularly relates to an AI model training method and device and a computing device. BACKGROUND

[0002] With the rapid development of AI technology, a series of acceleration cards emerge to provide computing power for computing matrices, vectors and the like to accelerate the AI model computing process. Currently, recomputation is used in the AI model training process, which can effectively reduce the memory occupation of the acceleration card in the model training process.

[0003] In the related art, taking an AI model including a module A and a module B as an example, the implementation of recomputation in the model training process is as follows: the module A is taken as a module that needs to perform recomputation, in the forward calculation process, the acceleration card sequentially performs forward calculation of the module A and the module B, wherein the acceleration card discards the intermediate activation value of the module A and stores the output activation value of the module A when performing forward calculation of the module A; in the backward calculation process, the acceleration card performs backward calculation of the module B based on the output activation value of the module A stored in the forward calculation process, then performs forward calculation of the module A again to obtain the intermediate activation value and the output activation value of the module A (i.e., performs recomputation of the module A), and then performs backward calculation of the module A based on the intermediate activation value of the module A, and discards the intermediate activation value of the module A after performing backward calculation of the module A to reduce the memory occupation of the acceleration card.

[0004] In the above method, since the recomputation and the backward calculation of the module A need to be performed together, and the backward calculation of the module A is performed after the backward calculation of the module B, in order to ensure that the acceleration card can obtain the output activation value of the module A when performing backward calculation of the module B, the acceleration card must retain the output activation value of the module A when performing forward calculation of the module A, and retain it until after the backward calculation of the module B, which causes waste of the memory resources of the acceleration card. SUMMARY

[0005] The embodiments of the present application provide an AI model training method, device and computing device, which can save the memory resources of the acceleration card in the AI model training process.

[0006] In a first aspect, the present application provides an AI model training method, which comprises:

[0007] In response to the training request of the AI ​​model, forward computation is performed on multiple modules of the AI ​​model. These modules include a first module and a second module. The first module is the module that needs to be recomputed during the backward computation process, and the second module is the module that needs to use the output data of the first module during the forward computation process.

[0008] After the forward computation of the second module is completed, the output data of the first module is discarded;

[0009] The AI ​​model performs reverse computation on multiple modules. Before performing reverse computation on the second module, the first module is recomputed. Based on the data obtained from recomputing the first module, the second module is then reverse computed.

[0010] In this way, when performing forward computation on multiple modules of the AI ​​model, for the first module to be recomputed and the second module that depends on the output data of the first module, the output data of the first module is discarded after the forward computation of the second module is completed. When performing backward computation on multiple modules, the recomputation of the first module is scheduled to be performed before the backward computation of the second module. In this way, when performing the backward computation of the second module, it can rely on the data obtained by recomputing the first module. Through this flexible recomputation scheduling method, there is no need to store the output data of the first module from the forward computation process to the backward computation process, thus saving the memory resources of the accelerator card.

[0011] In some embodiments, the second module is the module following the first module among a plurality of modules.

[0012] In some embodiments, the AI ​​model includes multiple network layers, with at least one network layer executed on each accelerator card. The at least one network layer includes multiple modules executed sequentially, with the second module being the last of the multiple modules.

[0013] In some embodiments, the AI ​​model is a Transformer model, and the network layer is a Transformer layer.

[0014] Secondly, this application provides a method for training an artificial intelligence (AI) model, the method comprising:

[0015] In response to the training request of the artificial intelligence (AI) model, multiple modules of the AI ​​model are identified, including a first module to be recomputed.

[0016] Generate forward computation tasks for each module, backward computation tasks for each module, and recomputation tasks for the first module in multiple modules.

[0017] Before sending the first reverse computation task to the accelerator card associated with the training request, a recomputation task is sent to the accelerator card. There is a dependency between the first reverse computation task and the recomputation task.

[0018] In some embodiments, training requests are executed by N accelerator cards in a pipelined parallel manner, where N is a positive integer;

[0019] Before sending the first reverse computation task to the accelerator card related to the training request, a recomputation task is sent to the accelerator card, including: before sending the first reverse computation task to the first accelerator card among N accelerator cards, a recomputation task is sent to the first accelerator card, the first reverse computation task is the reverse computation task of the first module, and the first accelerator card is used to execute the computation tasks related to the first module.

[0020] In some embodiments, the method further includes:

[0021] Send a second reverse computation task to the second accelerator card among N accelerator cards. The second accelerator card is used to execute computation tasks related to the third module. The third module is the next module after the first module. The second reverse computation task is the reverse computation task of the third module.

[0022] A first reverse computation task is sent to the first accelerator card. The first reverse computation task instructs the first module to be reverse computed based on the execution result of the second reverse computation task.

[0023] In some embodiments, training requests are executed by M accelerator cards in parallel via a virtual pipeline, where M is a positive integer.

[0024] In response to the training request of the artificial intelligence (AI) model, multiple modules of the AI ​​model are determined, including: in response to the training request, the first module is determined based on the number of virtual pipelines, the number of pipelines, and the number of M accelerator cards;

[0025] Before sending the first reverse computation task to the accelerator card related to the training request, a recomputation task is sent to the accelerator card, including: before sending the first reverse computation task to the third accelerator card among the M accelerator cards, a recomputation task is sent to the third accelerator card. The first reverse computation task is the reverse computation task of the first module, and the third accelerator card is used to execute the computation task related to the first module.

[0026] The method also includes: sending a forward computation task of the first module to the third accelerator card, wherein the forward computation task of the first module instructs that after performing forward computation on the first module, the intermediate activation value of the first module is discarded.

[0027] Thirdly, embodiments of this application provide an AI model training apparatus, which includes at least one functional unit for executing the AI ​​model training method provided in the first aspect or any possible implementation thereof.

[0028] Fourthly, embodiments of this application provide an AI model training apparatus, which includes at least one functional unit for executing the AI ​​model training method provided in the second aspect or any possible implementation thereof.

[0029] Fifthly, this application provides a computing device including a processor and a memory, wherein the processor is configured to execute at least a segment of program code stored in the memory to enable the computing device to implement the AI ​​model training method provided by the first aspect or any possible implementation thereof, or to implement the AI ​​model training method provided by the second aspect or any possible implementation thereof.

[0030] In a sixth aspect, embodiments of this application provide a computing system including a host and at least one accelerator card. The computing system is used to implement the training method for an AI model provided by the first aspect or any possible implementation of the first aspect, or to implement the training method for an AI model provided by the second aspect or any possible implementation of the second aspect.

[0031] Seventhly, this application provides a computer-readable storage medium for storing at least one piece of program code. When the at least one piece of program code is executed by a computing device, it causes the computing device to implement the AI ​​model training method provided by the first aspect or any possible implementation thereof, or to implement the AI ​​model training method provided by the second aspect or any possible implementation thereof. The storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid-state drive (SSD).

[0032] Eighthly, this application provides a computer program product that, when run on a computing device, causes the computing device to implement the AI ​​model training method provided by the first aspect or any possible implementation thereof, or to implement the AI ​​model training method provided by the second aspect or any possible implementation thereof. The computer program product can be a software installation package; when the aforementioned AI model training method needs to be implemented, the computer program product can be downloaded and executed on the computing device. Attached Figure Description

[0033] Figure 1 This is a schematic diagram illustrating a related technology and its application scenario in this application.

[0034] Figure 2 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0035] Figure 3 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0036] Figure 4 This is a flowchart illustrating a training method for an AI model provided in an embodiment of this application;

[0037] Figure 5 This is a flowchart of another AI model training method provided in the embodiments of this application;

[0038] Figure 6 This is a flowchart of another AI model training method provided in the embodiments of this application;

[0039] Figure 7 This is a flowchart of another AI model training method provided in the embodiments of this application;

[0040] Figure 8 This is a schematic diagram of a training method for an AI model based on pipeline parallelism provided in an embodiment of this application;

[0041] Figure 9 This is a flowchart of another AI model training method provided in the embodiments of this application;

[0042] Figure 10 This is a schematic diagram of a training method for an AI model based on virtual pipeline parallelism provided in an embodiment of this application;

[0043] Figure 11 This is a schematic diagram of another training method for an AI model based on virtual pipeline parallelism provided in an embodiment of this application;

[0044] Figure 12This is a schematic diagram of the structure of an AI model training device provided in an embodiment of this application;

[0045] Figure 13 This is a schematic diagram of the structure of a training device for another AI model provided in an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the AI ​​models and related data of computing tasks involved in this application were obtained under fully authorized conditions.

[0047] To facilitate understanding, the key terms and concepts involved in this application will be explained below.

[0048] Artificial intelligence (AI) models are a class of mathematical algorithm models that use machine learning concepts to solve practical problems. Typically, AI models include a large number of parameters and calculation formulas (or calculation rules).

[0049] An operator (OP) is a computational unit or function that runs on a computing device. In the field of deep learning, neural network layers and even the entire model are composed of operators, which correspond to the computational logic within the neural network layers. For example, a convolutional layer is an operator; the weight summation process in a fully-connected layer (FClayer) is also an operator.

[0050] Acceleration cards, also known as acceleration devices, accelerators, or acceleration chips, are a type of specialized hardware accelerator or computer system designed to accelerate AI applications, especially neural networks, machine vision, and machine learning. For example, they provide computational power to perform calculations on matrices and vectors, thereby accelerating the computation of AI models. Illustratively, acceleration devices include, for example, graphics processing units (GPUs), neural network processing units (XPUs), intelligent processing units (IPUs), tensor processing units (TPUs), domain-specific architecture (DSA) chips, etc., and this application is not limited to these.

[0051] In this application, peak memory usage refers to the maximum amount of memory consumed by the accelerator card during AI model training. Typically, the magnitude of peak memory usage depends on several factors, such as model size (number of layers, number of parameters), training data size, batch size, and storage of intermediate computation results (e.g., activation values ​​generated from forward computation). For example, during AI model training, activation values ​​generated from forward computation need to be reserved for backward computation. Therefore, before backward computation begins, the number of activation values ​​generated from forward computation increases significantly, leading to excessive memory consumption and potentially reaching peak memory usage, resulting in insufficient memory. In other words, peak memory usage is generally reached during forward computation, and after backward computation begins, memory usage gradually decreases as backward computation progresses.

[0052] Recomputation refers to the process of training an AI model by not retaining the intermediate activation values ​​of each module during forward computation. Instead, the forward computation of the corresponding module is re-executed before the backward computation of each module to generate new intermediate activation values. After the backward computation of the module is completed, these intermediate activation values ​​are discarded, thereby alleviating the problem of excessively high memory peaks on the accelerator card.

[0053] Pipeline parallelism (PP) is a model parallel training method that distributes different layers of the model across different devices (such as accelerator cards), with each device responsible for training specific layers. This reduces the memory consumption of a single device, enabling large-scale model training. For example, a model might have 12 network layers. Accelerator card 0 executes layers 0-3, accelerator card 1 executes layers 4-7, and accelerator card 2 executes layers 8-11. The accelerator cards exchange data via a communication link. During forward computation, card 0 performs forward computation on layers 0-3 and sends the results to card 1. Card 1 performs forward computation on layers 4-7 and sends the results to card 2. Card 2 then performs forward computation on layers 8-11. The backward computation process is similar, starting from card 2 to obtain gradient information, which is then passed back to card 1 and card 0. In some scenarios, this pipeline parallelism is also called naive pipeline parallelism.

[0054] Virtual pipeline parallelism (VPP) is an extended training method of pipeline parallelism. In pipeline parallelism, while keeping the number of devices constant, it divides the network into more pipeline stages. Illustratively, taking a virtual pipeline parallelism scale of vpp as an example, all layers of the model are divided into vpp parts. After each part is completed, the result from the last device is sent to the first device, and the pipeline parallelism continues for the next part until all parts are completed. For example, based on the previous pipeline parallelism example, if the virtual pipeline scale vpp = 2, then the 12 network layers are divided into two parts: layers 0-5 in one part, and layers 6-11 in the other. Card 0 executes layers 0-1, card 1 executes layers 2-3, card 2 executes layers 4-5, card 0 executes layers 6-7, card 1 executes layers 8-9, and card 2 executes layers 10-11.

[0055] Micro-batch pipeline parallelism is an extended training method of pipeline parallelism. Typically, during model training, the samples used in one iteration form a batch (global batch). The gradients generated based on these samples are accumulated and used for weight updates. By dividing a batch of samples into multiple micro-batches, each containing at least one sample, and using micro-batches as the basic unit of parallel scheduling, the utilization rate of the equipment in pipeline parallelism can be improved, and the idle time of the equipment can be reduced. For example, in pipeline parallelism, the accelerator card executes forward or backward computations of a few layers based on a micro-batch each time.

[0056] A bubble is a phenomenon in parallel pipelines where a device may be idle, waiting for the result from another device before continuing its computation. This idle state is considered a waste of computing power.

[0057] The application scenarios and implementation environment of this application are described below.

[0058] This application is applied to scenarios where recomputation is used during AI model training to reduce memory usage on accelerator cards. This application does not limit the type of AI model or the type of accelerator card. See below for reference. Figure 1 This paper introduces the application scenarios of this application. Figure 1 This is a schematic diagram illustrating a related technology and its application scenario.

[0059] like Figure 1 As shown in Figure (a), in related technologies, taking an AI model including module A and module B as an example, the implementation method of recomputation during model training is as follows: Module A is taken as the module that needs to be recomputed. During the forward computation process, the accelerator card sequentially executes the forward computation of module A and module B. When the accelerator card executes the forward computation of module A, it discards the intermediate activation values ​​of module A and stores the output activation value of module A. A During the reverse computation process, the accelerator card performs the reverse computation of module B based on the output activation value of module A stored during the forward computation. Then, it re-executes the forward computation of module A to obtain the intermediate activation value and output activation value of module A. A (That is, recalculating module A), and then performing reverse computation of module A based on the intermediate activation value of module A. After performing reverse computation of module A, the intermediate activation value of module A is discarded to reduce the memory usage of the accelerator card. It is evident that since the recalculation and reverse computation of module A need to be executed together, and the reverse computation of module A is executed after the reverse computation of module B, it is necessary to ensure that the accelerator card can obtain the output activation value (Output) of module A when performing the reverse computation of module B. A When the accelerator card performs the forward computation of module A, it must retain the output activation value of module A. A This is then retained until the reverse calculation of module B, resulting in a waste of accelerator card memory resources.

[0060] like Figure 1As shown in Figure (b), this application provides an implementation method for recomputation during AI model training, which can effectively reduce the peak memory usage of the accelerator card and reduce memory resource waste. In this application, the recomputation and backward computation of module A are separated into two functions. That is, the recomputation is extracted and implemented as a separate function. In this way, the forward computation, backward computation, and recomputation involved in the AI ​​model training process can be scheduled independently. For example, in the forward computation process, the accelerator card executes the forward computation of module A and module B sequentially. When the accelerator card executes the forward computation of module A, it discards the intermediate activation values ​​of module A and stores the output activation value Output of module A. A Furthermore, after performing the forward computation of module B, the Output is discarded. A The actual storage, retaining the Output in the logical view. A This ensures the accuracy of the computational logic while freeing up memory space on the accelerator card. During the reverse computation process, the accelerator card first recomputes module A to obtain the intermediate activation value and output activation value (Output) of module A. A Furthermore, based on the intermediate activation value and output activation value Output of module A A The reverse computation of module B and module A are executed sequentially. That is, when the accelerator card executes the reverse computation of module B, it can rely on the output activation value of module A obtained through recomputation. A Furthermore, after performing the reverse computation of module A, the accelerator card discards the intermediate activation values ​​of module A, reducing the memory usage of the accelerator card. Therefore, the technical solution provided in this application increases the flexibility of recomputation, reduces the peak memory usage of the accelerator card, and improves model training performance.

[0061] It should be understood that the above Figure 1 This example uses two modules in the model. The same principle applies to models with more modules. For instance, if the model includes modules A, B, and C, and module A is the module to be recomputed, the recomputation of module A can be scheduled between the reverse computation of module C and the reverse computation of module B when using the implementation method provided in this application. In other words, since the reverse computation of module B depends on the output activation value of module A, scheduling the recomputation of module A before the reverse computation of module B, or scheduling the recomputation of module A after the accelerator card reaches its memory peak, can minimize the memory usage of the accelerator card by the output activation value of module A, thereby improving model training performance.

[0062] The following is for reference. Figure 2 The implementation environment of this application will be described. Figure 2This is a schematic diagram of an implementation environment provided in an embodiment of this application. For example... Figure 2 As shown, the implementation environment includes a computing system, which includes a host 100 and at least one accelerator card 200, and there is a communication connection between the host 100 and the at least one accelerator card 200.

[0063] The host 100 is used to provide AI services. In this embodiment, the host 100 can control the accelerator card 200 to execute computational tasks involved in the AI ​​model training process. For example, in response to the training request of the AI ​​model, the host 100 sends forward computation tasks, backward computation tasks, and recomputation tasks of each module in the AI ​​model to the accelerator card 200, thereby performing forward computation, backward computation, and recomputation on each module. This process can also be understood as loading the computational tasks involved in the AI ​​model training process into the accelerator card 200 for execution. Furthermore, the number of hosts 100 can be one or more, and this application does not limit this.

[0064] Accelerator card 200 provides computing power for the training process of AI models, thereby accelerating the training process. For example, accelerator card 200 can be a GPU, XPU, IPU, TPU, DSA chip, etc., and this application is not limited to these. Illustratively, accelerator card 200 receives AI model computation tasks (such as forward computation tasks, backward computation tasks, recomputation tasks, etc.) sent by host 100, calls the corresponding operator, processes the input data of the operator, and obtains the output data, that is, the execution result of the computation task. Furthermore, the number of accelerator cards 200 can be one or more, and this application does not limit this. When the number of accelerator cards 200 is N, the computing system has distributed computing capabilities and can execute AI model training requests in a pipelined parallel manner, where N is a positive integer. Pipeline parallelism includes, but is not limited to, naive pipelined parallelism, virtual pipelined parallelism, micro-batch pipelined parallelism, etc.

[0065] The aforementioned host 100 and accelerator card 200 can be integrated into a single server or configured separately; this application does not impose any limitations on this. Illustratively, taking the integration of host 100 and accelerator card 200 into a single server as an example, host 100 and accelerator card 200 are connected via a peripheral component interconnect express (PCIe) link, and data exchange between host 100 and accelerator card 200 occurs through the PCIe link. The aforementioned server can be a standalone physical server, a server cluster composed of multiple physical servers, a distributed file system, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Taking a computing device as an example of a cloud server, the computing device can also be referred to as a cloud platform (i.e., short for cloud computing platform), which refers to services based on hardware and software resources, providing computing, networking, and storage capabilities. Cloud computing platforms leverage the internet to remotely process and analyze massive amounts of data before returning it to users. This approach offers advantages such as large scale, distributed computing, virtualization, high availability, scalability, on-demand service, and security. Cloud platforms can rapidly deploy and distribute configurable computing resources with minimal management overhead and low interaction complexity between users and service providers.

[0066] Furthermore, the networks mentioned above include, but are not limited to, data center networks, storage area networks (SANs), local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), mobile, wired or wireless networks, private networks, or virtual private networks (VPNs). In some implementations, technologies and / or formats, including Hypertext Markup Language (HTML) and Extensible Markup Language (XML), are used to represent data exchanged over the network. Additionally, conventional encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPsec) can be used to encrypt all or part of the links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0067] The hardware structure of the host 100 in the above computing system is described below.

[0068] This application provides a computing device that can be configured as the aforementioned host 100, with reference to... Figure 3 , Figure 3 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 3 As shown, the computing device 300 includes a memory 301, a processor 302, a communication interface 303, and a bus 304. The memory 301, processor 302, and communication interface 303 are interconnected via the bus 304.

[0069] Memory 301 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Illustratively, memory 301 is used to store at least one piece of program code. When the program code stored in memory 301 is executed by processor 302, processor 302 performs the steps performed by the host in the following method embodiments.

[0070] The processor 302 may be a network processor (NP), a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or an integrated circuit used to control the execution of the program of the present application. The processor 302 may be a single-core processor or a multi-core processor. The number of processors 302 may be one or more.

[0071] The communication interface 303 uses a transceiver module, such as a transceiver, to enable communication between the computing device 300 and other devices or communication networks. For example, data can be acquired through the communication interface 303.

[0072] The memory 301 and the processor 302 can be set separately or integrated together.

[0073] Bus 304 may include a pathway for transmitting information between various components of computing device 300 (e.g., memory 301, processor 302, communication interface 303).

[0074] Based on the above Figures 1 to 3The content shown introduces the application scenarios and implementation environment provided in this application. The training method of the AI ​​model provided in this application is described below.

[0075] Figure 4 This is a flowchart illustrating a training method for an AI model provided in an embodiment of this application. For example... Figure 4 As shown, the method is applied to a host in a computing system as an example. Schematic, the method includes the following steps 401 to 403.

[0076] 401. In response to the training request of the AI ​​model, perform forward computation on multiple modules of the AI ​​model. The multiple modules include a first module and a second module. The first module is the module that needs to be recomputed during the backward computation process, and the second module is the module that needs to use the output data of the first module during the forward computation process.

[0077] In this embodiment, the computing system includes a host and accelerator cards. The host can access a wired or wireless network to provide AI services to users. Performing forward computation on multiple modules of the AI ​​model means that the host sends forward computation tasks for each module to the accelerator cards related to the training requests, and controls the accelerator cards to execute the forward computation of the corresponding modules according to the received tasks. The AI ​​model includes multiple modules. Any module in the AI ​​model can be one or more network layers, or a component or basic element within a network layer. Furthermore, any module can be a non-linear or linear module. This application does not limit the granularity or type of module division in the AI ​​model.

[0078] The multiple modules include a first module and a second module. The first module is the module that needs to be recalculated during the backward computation process. That is, during the forward computation of the first module, the intermediate activation values ​​of the first module can be discarded and re-executed during the subsequent backward computation to generate new intermediate activation values. This reduces the memory usage of the accelerator card. The second module is the module that needs to use the output data of the first module during the forward computation process. In other words, during the forward computation process, the forward computation of the second module depends on the output data of the first module, which is the output activation value of the first module. The same principle applies below. For example, the second module may be the next module after the first module among multiple modules. Or, in some embodiments, the AI ​​model includes multiple network layers, and at least one network layer is executed on each accelerator card related to the training request. This at least one network layer includes the aforementioned multiple modules executed sequentially, and the second module is the last module among multiple modules. For example, an AI model might be a Transformer model, and its network layers might be Transformer layers, also known as Transformer blocks. It should be understood that a Transformer block typically refers to the basic building block of a Transformer model, which may include components such as multi-head self-attention, a feedforward network, normalization, and residual connections. Multiple Transformer blocks are stacked sequentially to form a complete Transformer model, enabling effective feature extraction and representation learning from the input sequence. Illustratively, taking any Transformer block in the Transformer model as an example, if this network layer includes sequentially executed modules A, B, C, and D, then the first module can be module A, B, or C, and the second module is module D.

[0079] Indicatively, in response to a training request from an AI model, the host computer compiles operators based on the AI ​​model's framework, i.e., generates operators for the AI ​​model. For example, operator generation involves three processes: input tensor description, weight data transformation, and output tensor description. Specifically, in the input tensor description, information such as the input dimension and memory size of each operator is calculated, and the format of the operator's input data is defined. In the weight data transformation, the weight parameters used by the operator undergo data format, shape transformation, and data compression. In the output tensor description, information such as the output dimension and memory size of the operator is calculated. After compiling operators based on the AI ​​model's framework, the host computer adapts the operators to backend device interfaces (i.e., adapts the upper-layer operators to the lower-layer hardware devices), identifies the operator attributes, generates corresponding computation tasks based on the identified operators, and sends the computation tasks to the accelerator cards related to the training request.

[0080] 402. After the forward computation of the second module is completed, discard the output data of the first module.

[0081] In this embodiment, performing forward computation on the second module means that the host sends a forward computation task for the second module to the accelerator card, and controls the accelerator card to perform forward computation on the second module based on the output data of the first module according to the received forward computation task. After the forward computation of the second module is completed, discarding the output data of the first module means that the host sends a discard task to the accelerator card after the forward computation of the second module is completed, and controls the accelerator card to discard the output data of the first module based on the discard task. The discard task indicates that the actual storage of the output data of the first module in the accelerator card should be discarded, but the logical view of the output data should be retained. This ensures the accuracy of the computational logic while freeing up the accelerator card's memory space. It should be understood that since the output data of the first module is not used temporarily after the forward computation of the second module, the discard task can be executed to discard the output data and free up the accelerator card's memory space.

[0082] 403. Perform reverse computation on multiple modules of the AI ​​model. Before performing reverse computation on the second module, perform recomputation on the first module, and perform reverse computation on the second module based on the data obtained from recomputing the first module.

[0083] In this embodiment, performing reverse computation on multiple modules of the AI ​​model means that the host sends the reverse computation task for each module to the accelerator card related to the training request, and controls the accelerator card to execute the reverse computation of the corresponding module according to the received reverse computation task. In this process, before performing reverse computation on the second module, the first module is recomputed, and the second module is reverse computed based on the data obtained from the recomputation of the first module. That is, before sending the reverse computation task for the second module to the accelerator card, the recomputation task for the first module is sent to the accelerator card, so that the accelerator card executes the reverse computation task for the second module based on the data obtained from the recomputation of the first module. In other words, the recomputation of the first module is scheduled to be performed before the reverse computation of the second module. Thus, after the recomputation of the first module, the reverse computation of the second module can depend on the data obtained from the recomputation of the first module. That is, the output data of the first module does not need to be stored from the forward computation process all the way to the reverse computation process of the second module.

[0084] In this way, when performing forward computation on multiple modules of the AI ​​model, for the first module to be recomputed and the second module that depends on the output data of the first module, the output data of the first module is discarded after the forward computation of the second module is completed. When performing backward computation on multiple modules, the recomputation of the first module is scheduled to be performed before the backward computation of the second module. In this way, when performing the backward computation of the second module, it can rely on the data obtained by recomputing the first module. Through this flexible recomputation scheduling method, there is no need to store the output data of the first module from the forward computation process to the backward computation process, thus saving the memory resources of the accelerator card.

[0085] Furthermore, based on the foregoing, for the first module in the AI ​​model, by splitting the recomputation and backward computation of the first module into two functions, and extracting the recomputation separately as a single function, the forward computation, backward computation, and recomputation involved in the AI ​​model training process can be scheduled independently. This method of separately scheduling recomputation improves the flexibility of recomputation, allowing users to schedule recomputation to appropriate locations according to business needs, thereby reducing the memory footprint of the accelerator card during AI model training or improving model training performance. Moreover, the computing system can have one or multiple accelerator cards; that is, the implementation method of using recomputation during AI model training provided in this application is applicable to both single and multiple accelerator cards. Based on this, the following references... Figures 5 to 11 The illustrated embodiments introduce several other training methods for AI models provided in this application.

[0086] Figure 5This is a flowchart illustrating another AI model training method provided in an embodiment of this application. For example... Figure 5 As shown, the method is introduced using the application of the method to a computing system as an example. Schematic, taking the interaction between the host and the accelerator card as an example, the method includes the following steps 501 to 503.

[0087] 501. In response to a training request for the AI ​​model, the host identifies multiple modules of the AI ​​model, including a first module to be recomputed.

[0088] In this embodiment, the host can access a wired or wireless network to provide AI services to the user. The AI ​​model includes multiple modules. Any module in the AI ​​model can be one or more network layers, or a component or basic element within a network layer. Furthermore, any module can be a non-linear or linear module. This application does not limit the granularity or type of module division in the AI ​​model.

[0089] The multiple modules include a first module, which is the module to be recalculated. That is, during the forward computation of the first module, its intermediate activation values ​​are not retained. Instead, in the subsequent backward computation, the forward computation of the first module is re-executed to generate new intermediate activation values. This method reduces the memory usage of the accelerator card. Furthermore, this application does not limit the position of the first module in the AI ​​model. The first module can be determined according to business needs. For example, the first module may be the first module among multiple modules, or each module in the AI ​​model may be the first module, and so on. It should be understood that in this embodiment, the first module refers to the module that needs to be recalculated, and its number can be one or more; this application does not limit this.

[0090] 502. The host generates forward computation tasks for each module, backward computation tasks for each module, and recomputation tasks for the first module.

[0091] In this embodiment, for any one of the multiple modules, the host generates a forward computation task and a backward computation task for that module. Furthermore, for the first module, the host generates a recomputation task for that module. Specifically, the forward computation task for the first module instructs that forward computation be performed on the first module. After the forward computation, the intermediate activation values ​​of the first module are discarded, and the output data of the first module is stored. This process essentially separates the recomputation and backward computation of the first module in the AI ​​model into two functions, extracting the recomputation as a separate function. Thus, the forward computation, backward computation, and recomputation involved in the AI ​​model training process can be independently scheduled.

[0092] Furthermore, this application does not limit the timing of when the host generates the computing tasks corresponding to each module. The host can generate each computing task sequentially according to the training framework of the AI ​​model, and when any computing task is generated, it can send the computing task to the acceleration card related to the training request for execution. Alternatively, the host can generate the computing tasks corresponding to each module according to the training framework of the AI ​​model, and then send the computing tasks to the acceleration card related to the training request sequentially according to the execution order of each computing task, and so on. This process can be configured according to business needs.

[0093] 503. Before sending the first reverse computation task to the accelerator card related to the training request, the host sends the recomputation task of the first module to the accelerator card. There is a dependency between the first reverse computation task and the recomputation task.

[0094] In this embodiment, a communication connection is established between the host and the accelerator card, enabling the host to control the accelerator card to execute corresponding computational tasks. The first reverse computation task is a reverse computation task among multiple modules' reverse computation tasks. The dependency between the first reverse computation task and the recomputation task means that the execution of the first reverse computation task depends on the execution result of the recomputation task. Indicatively, the first reverse computation task can be determined based on the AI ​​model's training framework. Before sending the first reverse computation task to the accelerator card, the host sends the recomputation task of the first module to the accelerator card, allowing the accelerator card to depend on the execution result of the recomputation task when executing the first reverse computation task. This method of separately scheduling recomputation tasks improves the flexibility of recomputation, allowing users to schedule recomputation tasks to appropriate locations according to business needs, thereby reducing the memory usage of the accelerator card during AI model training or improving model training performance.

[0095] In summary, in the AI ​​model training method provided in this application embodiment, after determining multiple modules of the AI ​​model according to the training request of the AI ​​model, forward computation tasks and backward computation tasks corresponding to each module are generated. Furthermore, for the first module among the multiple modules to be recomputed, a recomputation task for the first module is generated separately. When distributing computation tasks to the acceleration card related to the training request, the recomputation task is scheduled before the first backward computation task that has a dependency on it. By scheduling the recomputation task separately, the flexibility of recomputation is effectively improved.

[0096] Based on the foregoing Figure 5 As shown in the embodiments, this application provides an AI model training method that schedules recomputation tasks separately. The first reverse computation task can be determined according to the training framework of the AI ​​model. The training method of the above AI model will be further explained below with different scenarios as examples.

[0097] Figure 6 This is a flowchart illustrating another AI model training method provided in an embodiment of this application. For example... Figure 6 As shown, the method is applied to a computing system including a host and an accelerator card. Schematic, the method includes the following steps 601 to 614.

[0098] 601. In response to a training request for an AI model, the host identifies multiple modules of the AI ​​model, including a first module to be recomputed.

[0099] Among them, multiple modules also include a second module, which is a module that needs to use the output data of the first module during the forward computation process. For example, the second module is the next module after the first module among multiple modules.

[0100] 602. The host generates forward computation tasks for each module, backward computation tasks for each module, and recomputation tasks for the first module.

[0101] 603. The host sends the forward computation task of the first module to the accelerator card related to the training request.

[0102] 604. The accelerator card performs the forward computation task of the first module.

[0103] The accelerator card performs forward calculations on the first module to obtain the intermediate activation value and output data of the first module, discards the intermediate activation value of the first module, and stores the output data of the first module in the memory of the accelerator card.

[0104] 605. The host sends the forward computation task of the second module to the accelerator card.

[0105] The forward computation task of the second module indicates that forward computation should be performed on the second module of the AI ​​model.

[0106] 606. The accelerator card performs the forward computation task of the second module.

[0107] In this process, the accelerator card performs forward computation on the second module based on the output data of the first module to obtain the intermediate activation value and output data of the second module, and stores the intermediate activation value and output data of the second module in the memory of the accelerator card. It should be noted that the second module can also be a module to be recalculated. For example, if the second module and the first module are both modules to be recalculated, then in this step, after obtaining the intermediate activation value and output data of the second module, the accelerator card discards the intermediate activation value of the second module and stores the output data of the second module in the memory of the accelerator card. This application does not limit this.

[0108] 607. The host sends a drop task to the accelerator card, which instructs the output data of the first module to be discarded.

[0109] In this embodiment of the application, the discard task instruction to discard the output data of the first module means discarding the actual storage of the output data of the first module in the accelerator card, while retaining the logical view of the output data. In this way, the accuracy of the calculation logic can be ensured while freeing up the memory space of the accelerator card.

[0110] 608. Accelerator card performs discard task.

[0111] It should be understood that since the accelerator card will not use the output data of the first module temporarily after executing the forward computation task of the second module, a discard task can be executed to discard the output data and free up the memory space of the accelerator card.

[0112] 609. The host sends a recomputation task for the first module to the accelerator card. The recomputation task instructs the first module to be recomputed.

[0113] 610. Accelerator cards perform recalculation tasks.

[0114] The accelerator card recalculates the first module to obtain the intermediate activation value and output data of the first module. In other words, the execution result of the recalculation task includes the output data of the first module.

[0115] 611. The host sends a first reverse computation task to the accelerator card. This first reverse computation task is the reverse computation task of the second module.

[0116] 612. The accelerator card performs the first reverse computation task.

[0117] In this process, the accelerator card performs reverse computation on the second module based on the output data of the first module stored in memory and the intermediate activation value of the second module. This means that the output data of the first module, which the accelerator card relies on for reverse computation of the second module, is obtained through the recomputation task of the first module, rather than being stored during the forward computation process. In other words, the output data of the first module does not need to be stored from the forward computation process all the way to the reverse computation process of the second module.

[0118] 613. The host sends the reverse computation task of the first module to the accelerator card.

[0119] 614. The accelerator card performs the reverse computation task of the first module.

[0120] The accelerator card performs reverse computation on the first module based on the intermediate activation values ​​of the first module stored in memory. It should be understood that if the first module is the first module of the AI ​​model, the accelerator card also needs to combine the samples input to the AI ​​model when performing the first reverse computation task; if there are other modules before the first module, the accelerator card also needs to combine the output data of the other modules when performing the first reverse computation task, which will not be elaborated here.

[0121] In steps 603 to 614 above, the first and second modules in the AI ​​model are used as examples for illustration. It should be understood that in some embodiments, if the AI ​​model includes more modules, between steps 608 and 609 above, there is also a step in which the host sequentially sends forward computing tasks of other modules and reverse computing tasks of other modules to the accelerator card.

[0122] By employing the above method, after determining multiple modules of the AI ​​model based on the training request, forward and backward computation tasks are generated for each module. Furthermore, for the first module requiring recomputation, a separate recomputation task is generated. When distributing computation tasks to the accelerator cards related to the training request, this recomputation task is scheduled before the backward computation task of the second module, which has a dependency on it. This method of separately scheduling recomputation tasks effectively improves the flexibility of recomputation. Moreover, when the accelerator card executes the backward computation task of the second module, it can rely on the output data of the first module obtained from the recomputation task. Thus, after executing the forward computation task of the second module, the accelerator card can promptly discard the output data of the first module, freeing up memory space and reducing memory usage.

[0123] In addition, in the above Figure 6In the illustrated embodiment, the second module is the next module after the first module. In other embodiments, the second module is a subsequent module of the first module. In some embodiments, the AI ​​model includes multiple network layers, and at least one network layer is executed on each accelerator card associated with the training request. This at least one network layer includes the aforementioned multiple modules executed sequentially, and the second module is the last module among the multiple modules. For example, the AI ​​model is a Transformer model, and the network layers are Transformer layers, also known as Transformer blocks. It should be understood that a Transformer block typically refers to the basic building block in a Transformer model, which may include components such as multi-head self-attention, feedforward networks, and normalization and residual connections. Multiple Transformer blocks are stacked sequentially to form a complete Transformer model, enabling effective feature extraction and representation learning of the input sequence. Schematic, taking any Transformer block in the Transformer model as an example, if the network layer executed on the accelerator card includes modules A, B, C, and D executed sequentially, then the first module can be module A, module B, or module C, and the second module is module D. Based on this, by using the above method, scheduling the recomputation of at least one of the modules A, B, and C before the reverse computation task of module D can minimize the memory usage of the accelerator card and thus improve the model training performance.

[0124] Figure 7 This is a flowchart illustrating another AI model training method provided in an embodiment of this application. For example... Figure 7 As shown, the method is applied to a computing system, which includes a host and N accelerator cards. Schematic, the method includes the following steps 701 to 703.

[0125] 701. In response to a training request for an AI model, the host identifies multiple modules of the AI ​​model, including a first module to be recomputed.

[0126] In this embodiment, the host computer is communicatively connected to N accelerator cards. The N accelerator cards execute the training requests of the AI ​​model in a pipelined parallel manner, where N is a positive integer. Pipeline parallelism includes, but is not limited to, naive pipelined parallelism, virtual pipelined parallelism, micro-batch pipelined parallelism, etc. The various modules of the AI ​​model are each executed with corresponding computational tasks by the N accelerator cards.

[0127] 702. The host generates forward computation tasks for each module, backward computation tasks for each module, and recomputation tasks for the first module.

[0128] To more clearly explain the process of N accelerator cards executing training requests in a pipelined parallel manner, in this embodiment, refer to Figure 8 This section will introduce the training methods for AI models. Figure 8 This is a schematic diagram of a pipelined parallel AI model training method provided in an embodiment of this application. Figure 8 As shown, the computing system includes N acceleration devices, N=4, represented as Card 0, Card 1, Card 2, and Card 3. The example described uses N acceleration cards in parallel operation based on micro-batches and virtual pipelines. The micro-batches b=8, the virtual pipeline vpp=2, and the AI ​​model consists of 8 layers. In the diagram, F represents forward computation, B represents backward computation, and R represents recomputation (assuming recomputation is performed on every layer). The 8 network layers are divided into two parts: layers 1 to 4 form one part, and layers 5 to 8 form the other. Card 0 executes layers 1 and 5, Card 1 executes layers 2 and 6, Card 2 executes layers 3 and 7, and Card 3 executes layers 4 and 8. Unfilled boxes represent the earlier network layers in the model (layers 1 to 4), and filled boxes represent the later network layers (layers 5 to 8). The numbers 1-8 in the boxes represent the 8 micro-batches b1 to b8, respectively. It should be noted that... Figure 8 The diagram is shown to illustrate the principle of the scheme. The dimensions of each box shown in the diagram are only for illustrative purposes and do not constitute a limitation on the execution time of each computing task. For example, in practical applications, the execution time of the reverse computing task is usually longer than that of the forward computing task. For instance, the execution time of the reverse computing task is twice that of the forward computing task.

[0129] like Figure 8 As shown, in the timing diagram corresponding to card 0, the unfilled box F1 indicates forward computation of layer 1 based on micro-batch b1, the filled box F1 indicates forward computation of layer 5 based on micro-batch b1, the unfilled box B1 indicates backward computation of layer 1 based on micro-batch b1, the filled box B1 indicates backward computation of layer 5 based on micro-batch b1, the unfilled box R1 indicates recomputation of layer 1 based on micro-batch b1, and the filled box R1 indicates recomputation of layer 5 based on micro-batch b1.

[0130] For example, taking micro-batch b1 as an example, if the first module is the first layer of the AI ​​model, then the forward computation task of the first module is the unfilled box F1 in the time sequence corresponding to card 0, and the recomputation task of the first module is the unfilled box R1 in the time sequence corresponding to card 0; if the first module is the fifth layer of the AI ​​model, then the forward computation task of the first module is the filled box F1 in the time sequence corresponding to card 0, and the recomputation task of the first module is the filled box R1 in the time sequence corresponding to card 0; if the first module is the second layer of the AI ​​model, then the forward computation task of the first module is the unfilled box F1 in the time sequence corresponding to card 1, and the recomputation task of the first module is the unfilled box R1 in the time sequence corresponding to card 1; if the first module is the sixth layer of the AI ​​model, then the forward computation task of the first module is the filled box F1 in the time sequence corresponding to card 1, and the recomputation task of the first module is the filled box R1 in the time sequence corresponding to card 1, and so on, without further details.

[0131] 703. Before sending the first reverse computation task to the first accelerator card among N accelerator cards, the host sends the recomputation task of the first module to the first accelerator card. The first reverse computation task is the reverse computation task of the first module.

[0132] The first accelerator card is used to perform computational tasks related to the first module.

[0133] by Figure 8 Taking the example shown, if the first module is the first layer of the AI ​​model, then the first reverse computation task is the unfilled box B1 in the time sequence corresponding to card 0, and the recomputation task of the first module is the unfilled box R1 in the time sequence corresponding to card 0. It should be understood that because the recomputation task is scheduled separately during the pipelined parallel execution of training requests by N accelerator cards, the recomputation task does not need to wait for the execution result of the previous accelerator card. Only the reverse computation task of the first module needs to wait for the execution result of the previous accelerator card. Illustratively, the host sends a second reverse computation task to the second accelerator card among the N accelerator cards. The second accelerator card is used to execute computation tasks related to the third module, which is the next module after the first module. The second reverse computation task is the reverse computation task of the third module. The host also sends a first reverse computation task to the first accelerator card, which instructs to perform reverse computation on the first module based on the execution result of the second reverse computation task. In this way, the pipeline's bubbles can be utilized for recomputation, thereby reducing the waste of computing power. For example, continue to refer to... Figure 8Taking the first accelerator card as card 0 and the second accelerator card as card 1 as an example, since the filled box R6 (recalculation of layer 5 based on micro-batch b6) and the filled box B6 (reverse calculation of layer 5 based on micro-batch b6) are scheduled separately, after card 0 executes the filled box B5 (reverse calculation of layer 5 based on micro-batch b5), it can immediately execute the filled box R6 (recalculation of layer 5 based on micro-batch b6). Then, after receiving the execution result of the filled box B6 (reverse calculation of layer 6 based on micro-batch b6 in the corresponding timing of card 1) sent by card 1, card 0 executes the filled box B6 (reverse calculation of layer 5 based on micro-batch b6) corresponding to card 0. Compared with related technologies, the bubble indicated by the arrow in the figure is filled, that is, in the related technologies, the filled box R6 and the filled box B6 corresponding to card 0 are bound together, and these two tasks need to wait together for card 1 to send the execution result of the filled box B6 corresponding to card 1 before they can be executed. Obviously, compared with related technologies, the technical solution provided in this application can effectively reduce the bubbles that occur during the parallel process of the pipeline, thereby improving the utilization rate of the accelerator card.

[0134] Figure 9 This is a flowchart illustrating another AI model training method provided in an embodiment of this application. For example... Figure 9 As shown, the method is applied to a computing system, which includes a host and M accelerator cards. Schematic, the method includes the following steps 901 to 903.

[0135] 901. In response to the training request of the AI ​​model, the host determines multiple modules of the AI ​​model, including a first module to be recomputed.

[0136] In this setup, M accelerator cards execute AI model training requests in parallel via virtual pipelines, where M is a positive integer. Illustratively, the host determines the first module based on the number of virtual pipelines, the number of pipelines, and the number of M accelerator cards. This process essentially means that the host treats a portion of the AI ​​model as modules to be recomputed, thereby freeing up memory space on the accelerator cards.

[0137] To more clearly explain the process of M accelerator cards executing training requests in a virtual pipelined parallel manner, in this embodiment, refer to Figure 10 This section will introduce the training methods for AI models. Figure 10 This is a schematic diagram illustrating a training method for an AI model based on virtual pipeline parallelism, provided in an embodiment of this application. For example... Figure 10As shown, the computing system includes M acceleration devices, where M=4. These M acceleration devices are represented as Card 0, Card 1, Card 2, and Card 3. The example described uses M acceleration cards parallelized based on micro-batch and virtual pipeline. The micro-batch size is b=8, the virtual pipeline size is vpp=2, and the AI ​​model consists of 8 layers. In the diagram, F represents forward computation, B represents backward computation, and R represents recomputation. The 8 network layers are divided into two parts: layers 1 to 4 form one part, and layers 5 to 8 form the other. Card 0 executes layers 1 and 5, Card 1 executes layers 2 and 6, Card 2 executes layers 3 and 7, and Card 3 executes layers 4 and 8. Unfilled boxes represent the earlier network layers in the model (layers 1 to 4), and filled boxes represent the later network layers (layers 5 to 8). The numbers 1-8 in the boxes represent the 8 micro-batches b1 to b8, respectively. It should be noted that... Figure 10 The diagram is shown to illustrate the principle of the scheme. The dimensions of each box shown in the diagram are only for illustrative purposes and do not constitute a limitation on the execution time of each computing unit. For example, in practical applications, the execution time of the reverse computing task is usually longer than that of the forward computing task. For instance, the execution time of the reverse computing task is twice that of the forward computing task.

[0138] like Figure 10 As shown, in the timing diagram corresponding to card 0, when card 0 performs the forward computation of layer 1 based on micro-batch b7 (corresponding to unfilled box F7), the available memory capacity reaches its peak. When card 0 performs the forward computation of layer 1 based on micro-batch b5 (corresponding to unfilled box F5), the available memory capacity reaches a certain threshold. Therefore, the modules corresponding to unfilled boxes F5, F6, and F7 can be identified as the first module, and the first module can be recomputed.

[0139] 902. The host generates forward computation tasks for each module, backward computation tasks for each module, and recomputation tasks for the first module.

[0140] For example, refer to Figure 10 , Figure 10 In the timing sequence corresponding to the middle card 0, unfilled boxes R5, R6, and R7 are all recalculation tasks of the first module.

[0141] 903. Before sending the first reverse computation task to the third accelerator card among the M accelerator cards, the host sends a recomputation task to the third accelerator card. The first reverse computation task is the reverse computation task of the first module, and the third accelerator card is used to execute the computation tasks related to the first module.

[0142] Among them, with Figure 10Taking the example shown, if the third accelerator card is card 0, the first reverse computation task is the unfilled boxes B5, B6, and B7 in the timing sequence corresponding to card 0. It should be understood that, since some modules are identified as modules to be recomputed during the parallel execution of training requests by the M accelerator cards in a virtual pipeline, the accelerator card can discard the intermediate activation values ​​of the first module when executing the forward computation task of the first module, thus freeing up memory space on the accelerator card. Illustratively, the host sends the forward computation task of the first module to the third accelerator card, instructing that after performing forward computation on the first module, the intermediate activation values ​​of the first module be discarded.

[0143] Furthermore, by employing the methods described above, when M accelerator cards execute training requests in parallel via a virtual pipeline, recomputing certain modules can be performed. This also allows for the utilization of pipeline bubbles for recomputation, thereby reducing wasted computing power. For example, continue to refer to... Figure 10 Taking the third accelerator card, card 0, as an example, for card 0, the unfilled box R5 (based on the recalculation of layer 1 based on micro-batch b5) can be scheduled to be executed at the bubble before unfilled box B3; the unfilled box R6 (based on the recalculation of layer 1 based on micro-batch b6) can be scheduled to be executed at the bubble before unfilled box B4; and the unfilled box R7 (based on the recalculation of layer 1 based on micro-batch b7) can be scheduled to be executed at the bubble before filled box B5. Thus, when card 0 executes unfilled box F5 (based on the forward calculation of layer 1 based on micro-batch b5), unfilled box F6 (based on the forward calculation of layer 1 based on micro-batch b6), and unfilled box F7 (based on the forward calculation of layer 1 based on micro-batch b7), the intermediate activation values ​​of layer 1 can be discarded, freeing up accelerator card memory space while maintaining performance. Of course, in practical applications, the recalculation scheduling strategy can be determined according to business needs, and this application does not limit this.

[0144] In some embodiments, for any one of the M accelerator cards, the host uses the following scheduling method to send tasks to that accelerator card, in order to achieve the above-described... Figure 10 The diagram illustrates the parallel execution effect of the pipeline. In a schematic way, let pp represent the number of pipelines, vpp represent the number of virtual pipelines, b represent the number of micro-batches, and the i-th accelerator card represent any single accelerator card (where i ranges from 0 to M-1). This scheduling method includes the following stages:

[0145] Phase 1: Based on the number of pipelines and virtual pipelines, the host sends X forward computation tasks to the accelerator card. These X forward computation tasks are not recomputed subsequently. Illustratively, X = min(pp × vpp + pp - 2 - 2i, pp × vpp). (Reference) Figure 11Taking the i-th accelerator card as card 0 as an example, the X forward computation tasks are the forward computations corresponding to the first 8 boxes.

[0146] Phase 2: Based on the number of pipelines, the host sends Y forward computation tasks to the accelerator card. These Y forward computation tasks require recomputation later. Schematic: Y = max(pp-2-2i, 0). (Reference) Figure 11 Taking the i-th accelerator card as card 0 as an example, the Y forward computation tasks are unfilled box F5 (forward computation of the first layer based on micro-batch b5) and unfilled box F6 (forward computation of the first layer based on micro-batch b6).

[0147] Phase 3: Based on the number of pipelines, micro-batch counts, and virtual pipeline counts, the host sends Z first computation task groups to the accelerator card. Each first computation task group includes one forward computation task and one backward computation task. Illustratively, Z = b × vpp - pp × vpp - pp + 2 + 2i. Wherein, if pp - 2 - 2i ≥ 0, the forward computation task of the first first computation task group in the Z first computation task groups needs to be recomputed. (Reference) Figure 11 Taking the i-th accelerator card as card 0 as an example, there are Z first computing task groups as indicated by the arrows in the figure, a total of 6 groups.

[0148] Phase 4: Based on the number of pipelines, the host sends W second computation task groups to the accelerator card. Each second computation task group includes one recomputation task and one reverse computation task. Schematic, W = max(pp⁻¹⁻²i, 0). (Reference) Figure 11 Taking the i-th accelerator card as card 0 as an example, there are W second computing task groups as indicated by the arrows in the figure, for a total of 3 groups.

[0149] Phase 5: The host sends K reverse computation tasks to the accelerator card based on the number of pipelines and the number of virtual pipelines. Schematic, K = min(pp × vpp - 1, pp × vpp + pp - 2 - 2i). (Reference) Figure 11 Taking the i-th accelerator card as card 0 as an example, there are K reverse calculation tasks as indicated by the arrows in the figure, a total of 7.

[0150] It should be noted that the above scheduling method is only an example. In some embodiments, the modules that need to be recalculated can be set according to business needs, and the execution time of the recalculation task can be set, etc. This application does not limit this.

[0151] Additionally, refer to Figure 12 This application also provides a training device for an AI model, which is used to implement the aforementioned training method for the AI ​​model. Figure 12 This is a schematic diagram of the structure of an AI model training device provided in an embodiment of this application. Figure 12As shown, the device includes a forward computation execution unit 1201, a discard unit 1202, and a reverse computation execution unit 1203.

[0152] The forward computation execution unit 1201 is used to perform forward computation on multiple modules of the AI ​​model in response to the training request of the AI ​​model. The multiple modules include a first module and a second module. The first module is the module that needs to be recomputed during the backward computation process, and the second module is the module that needs to use the output data of the first module during the forward computation process.

[0153] The discard unit 1202 is used to discard the output data of the first module after the forward computation of the second module is completed;

[0154] The reverse computation execution unit 1203 is used to perform reverse computation on multiple modules of the AI ​​model. Before performing reverse computation on the second module, it performs recomputation on the first module and performs reverse computation on the second module based on the data obtained from recomputing the first module.

[0155] In some embodiments, the second module is the module following the first module among a plurality of modules.

[0156] In some embodiments, the AI ​​model includes multiple network layers, with at least one network layer executed on each accelerator card. The at least one network layer includes multiple modules executed sequentially, with the second module being the last of the multiple modules.

[0157] In some embodiments, the AI ​​model is a Transformer model, and the network layer is a Transformer layer.

[0158] It should be noted that the AI ​​model training device provided in the above embodiments is only illustrated by the division of the above functional modules when training the AI ​​model. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the AI ​​model training device and the AI ​​model training method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0159] refer to Figure 13 This application also provides a training device for an AI model, which is used to implement the aforementioned training method for the AI ​​model. Figure 13 This is a schematic diagram of the structure of a training device for another AI model provided in an embodiment of this application. For example... Figure 13 As shown, the device includes a determining unit 1301, a generating unit 1302, and a sending unit 1303.

[0160] The determining unit 1301 is used to determine multiple modules of the AI ​​model in response to a training request of the AI ​​model, the multiple modules including a first module to be recomputed;

[0161] The generation unit 1302 is used to generate the forward computation task of each module, the backward computation task of each module, and the recomputation task of the first module in the multiple modules.

[0162] The sending unit 1303 is used to send a recalculation task to the accelerator card before sending the first reverse computation task to the accelerator card related to the training request. There is a dependency between the first reverse computation task and the recalculation task.

[0163] In some embodiments, training requests are executed by N accelerator cards in a pipelined parallel manner, where N is a positive integer;

[0164] The sending unit 1303 is used to send a recalculation task to the first accelerator card before sending the first reverse calculation task to the first accelerator card among N accelerator cards. The first reverse calculation task is the reverse calculation task of the first module, and the first accelerator card is used to execute the calculation tasks related to the first module.

[0165] In some embodiments, the transmitting unit 1303 is further configured to:

[0166] Send a second reverse computation task to the second accelerator card among N accelerator cards. The second accelerator card is used to execute computation tasks related to the third module. The third module is the next module after the first module. The second reverse computation task is the reverse computation task of the third module.

[0167] A first reverse computation task is sent to the first accelerator card. The first reverse computation task instructs the first module to be reverse computed based on the execution result of the second reverse computation task.

[0168] In some embodiments, training requests are executed by M accelerator cards in parallel via a virtual pipeline, where M is a positive integer.

[0169] Unit 1303 is used to determine the first module in response to a training request, based on the number of virtual pipelines, the number of pipelines, and the number of M accelerator cards.

[0170] The sending unit 1303 is used to send a recalculation task to the third accelerator card before sending the first reverse calculation task to the third accelerator card among the M accelerator cards. The first reverse calculation task is the reverse calculation task of the first module, and the third accelerator card is used to execute the calculation tasks related to the first module.

[0171] The sending unit 1303 is also used to send the forward computation task of the first module to the third accelerator card. The forward computation task of the first module indicates that after performing forward computation on the first module, the intermediate activation value of the first module is discarded.

[0172] It should be noted that the AI ​​model training device provided in the above embodiments is only illustrated by the division of the above functional modules when training the AI ​​model. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the AI ​​model training device and the AI ​​model training method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0173] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with substantially the same function and purpose. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the scope of the various examples described, a first accelerator card can be referred to as a second accelerator card, and similarly, a second accelerator card can be referred to as a first accelerator card. Both the first and second accelerator cards can be accelerator cards, and in some cases, they can be separate and different accelerator cards.

[0174] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple accelerator cards means two or more accelerator cards.

[0175] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0176] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of program structure information. This program structure information includes one or more program instructions. When these program instructions are loaded and executed on a computing device, the processes or functions according to the embodiments of this application are generated, in whole or in part.

[0177] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0178] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for training an artificial intelligence (AI) model, the method comprising: The method comprises: in response to a training request of an AI model, performing forward calculation on a plurality of modules of the AI model, the plurality of modules comprising a first module and a second module, the first module being a module that needs to perform re-calculation in a backward calculation process, and the second module being a module that needs to use output data of the first module in a forward calculation process; after the forward calculation of the second module is completed, discarding the output data of the first module; performing backward calculation on the plurality of modules of the AI model, performing re-calculation on the first module before performing backward calculation on the second module, and performing backward calculation on the second module based on data obtained by re-calculating the first module.

2. The method of claim 1, wherein, The second module is the next module of the first module in the plurality of modules.

3. The method of claim 1, wherein, The AI model comprises a plurality of network layers, at least one network layer is executed on each acceleration card, the at least one network layer comprises the plurality of modules executed in sequence, and the second module is the last module in the plurality of modules.

4. The method of claim 3, wherein, The AI model is a Transformer model, and the network layer is a Transformer layer. 5.A method for training an artificial intelligence (AI) model, comprising: The method comprises: in response to a training request of an AI model, determining a plurality of modules of the AI model, the plurality of modules comprising a first module to be re-calculated; generating a forward calculation task of each module in the plurality of modules, a backward calculation task of each module, and a re-calculation task of the first module; before sending a first backward calculation task to an acceleration card related to the training request, sending the re-calculation task to the acceleration card, and there is a dependency relationship between the first backward calculation task and the re-calculation task.

6. The method of claim 5, wherein, The training request is executed in a pipeline parallel manner by N acceleration cards, N being a positive integer; The method further comprises:

7. The method of claim 6, wherein, sending a second backward calculation task to a second acceleration card in the N acceleration cards, the second acceleration card being used to execute a calculation task related to a third module, the third module being a next module of the first module, and the second backward calculation task being a backward calculation task of the third module; sending the first backward calculation task to the first acceleration card, the first backward calculation task indicating that the first module is backward calculated based on an execution result of the second backward calculation task. The training request is executed in a virtual pipeline parallel manner by M acceleration cards, M being a positive integer; 8. The method of claim 5, wherein, ​ The determining the plurality of modules of the AI model in response to the training request of the artificial intelligence (AI) model comprises: in response to the training request, determining the first module based on a number of virtual pipelines, a number of pipelines, and a number of the M acceleration cards. The sending the re-computation task to the acceleration card before sending the first back-propagation computation task related to the training request to the acceleration card comprises: sending the re-computation task to a third acceleration card in the M acceleration cards before sending the first back-propagation computation task to the third acceleration card, the first back-propagation computation task being a back-propagation computation task of the first module, and the third acceleration card being configured to execute a computation task related to the first module. The method further comprises: sending a forward computation task of the first module to the third acceleration card, the forward computation task of the first module indicating discarding intermediate activation values of the first module after performing forward computation on the first module. 9.A training apparatus of an artificial intelligence (AI) model, comprising: The apparatus comprises: The forward computation execution unit is configured to perform forward computation on a plurality of modules of an AI model in response to a training request of the AI model, the plurality of modules comprising a first module and a second module, the first module being a module that needs to perform re-computation in a back-propagation computation process, and the second module being a module that needs to use output data of the first module in a forward computation process. The discarding unit is configured to discard the output data of the first module after the forward computation of the second module is performed. The back-propagation computation execution unit is configured to perform back-propagation computation on the plurality of modules of the AI model, perform re-computation on the first module before performing back-propagation computation on the second module, and perform back-propagation computation on the second module based on data obtained by performing re-computation on the first module.

10. The apparatus of claim 9, wherein, The second module is a next module of the first module in the plurality of modules.

11. The apparatus of claim 9, wherein, The AI model comprises a plurality of network layers, at least one network layer is executed on each acceleration card, the at least one network layer comprises the plurality of modules that are sequentially executed, and the second module is a last module in the plurality of modules.

12. The apparatus of claim 11, wherein, The AI model is a Transformer model, and the network layer is a Transformer layer.

13. A computing device, comprising: The computing device comprises a processor and a memory, the processor is configured to execute at least one piece of program code stored in the memory, so that the computing device implements the training method of the AI model according to any one of the preceding claims 1 to 8.

14. A computing system, comprising: The computing system comprises a host and at least one acceleration card, and the system is configured to implement the training method of the AI model according to any one of the preceding claims 1 to 8.

15. A computer program product, characterised in that, When the computer program product runs on the computing device, the computing device implements the training method of the AI model according to any one of the preceding claims 1 to 8.

16. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store at least one piece of program code, and the at least one piece of program code is configured to implement the training method of the AI model according to any one of the preceding claims 1 to 8.

Images