Storage method for model checkpoints, and related apparatus

Abstract

A storage method for model checkpoints, which method is applied to persistent storage of model checkpoints in the field of artificial intelligence (AI). In the storage method for model checkpoints, the operation of updating model parameters during model training is deployed on a central processing unit (CPU) for execution, such that the CPU can retain updated model parameters after executing the operation of updating the model parameters. In this way, while an accelerator continues executing model training on the basis of updated model parameters, a CPU can simultaneously store retained model parameters in a target storage medium, so that persistent storage of model parameters can be realized without the need to interrupt a training process, thereby avoiding affecting the normal training of AI models.

Description

A method and related apparatus for storing model checkpoints

[0001] This application claims priority to Chinese Patent Application No. 202410674450.2, filed on May 27, 2024, entitled "A Method for Storing Model Checkpoints and Related Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method and apparatus for storing model checkpoints. Background Technology

[0003] In the field of AI technology, the training process of an AI model often refers to the iterative updating of model parameters. Generally, to train a highly accurate AI model, the model often undergoes many rounds of iterative updates during training. Because AI model training involves numerous rounds of iterative updates, and the overall time for these updates is relatively long, it is often necessary to persistently store the model's training state at certain points in time. This persistently stored training state can serve as checkpoints, allowing the training process to be resumed based on these checkpoints if it is abnormally interrupted, thus avoiding having to restart the training process entirely.

[0004] In related technologies, to achieve persistent storage checkpoints, accelerators such as graphics processing units (GPUs) typically store the AI ​​model's parameters in a specific storage medium (such as a hard drive or a remote storage server) after completing a certain number of iterations of updates. This approach suffers from problems such as interrupting the training process and relatively low training efficiency. Summary of the Invention

[0005] This application provides a method for storing model checkpoints, which enables persistent storage of model parameters without interrupting the training process, thus avoiding impacting the normal training of the AI ​​model.

[0006] Firstly, a method for storing model checkpoints is provided, applicable to the persistent storage of checkpoints in the AI ​​field. This method is applied to a training and induction acceleration system, which includes an accelerator and a processor. Specifically, the method involves the accelerator performing the i-th round of training (where i is an integer greater than or equal to 1) of the first model (i.e., inputting the acquired training data into the first model and processing the training data by running the first model), obtaining the training result, and then passing the training result to the processor. The training result is used to update the parameters of the first model. Here, the accelerator is hardware specifically designed for accelerating AI computations, such as a graphics processing unit (GPU), tensor processing unit (TPU), or neural network processing unit (NPU).

[0007] Then, the processor updates the parameters of the first model based on the training results, obtaining the updated model parameters, and passes the updated model parameters to the accelerator. The processor can update the parameters of the first model by running an optimizer. The optimizer run by the processor can include, for example, a stochastic gradient descent (SGD) optimizer or an adaptive moment estimation (Adam) optimizer.

[0008] During the (i+1)th round of training of the first model by the accelerator based on the updated model parameters, the processor stores the updated model parameters in the target storage medium. That is, the accelerator's execution of the training iterations of the first model based on the updated model parameters and the processor's persistent storage of the updated model parameters in the target storage medium are performed synchronously. The target storage medium is the storage medium used to persistently store the model parameters, such as a hard drive connected to the execution device or a storage server connected to the execution device via a network.

[0009] In this solution, by deploying the operation of updating model parameters during model training to be executed on the processor, the processor can retain the updated model parameters after performing the update operation. Thus, while the accelerator continues model training based on the updated model parameters, the processor can simultaneously store the retained model parameters in the target storage medium, achieving persistent storage of model parameters without interrupting the training process and avoiding impact on the normal training of the AI ​​model.

[0010] Furthermore, since the communication speed between the accelerator and the processor is much faster than the speed at which the accelerator persists the model parameters to the target storage medium, the communication process between the accelerator and the processor caused by deploying the model parameter update operation on the processor will not have a significant impact on the overall training process. Moreover, having the processor perform the model parameter update operation eliminates the need for the accelerator to store the large amount of state data required for updating the model parameters (such as optimizer states), thus freeing up more storage space on the accelerator for the model training process. This effectively improves the efficiency of the accelerator in performing model training and shortens the overall model training time.

[0011] In one possible implementation, while the processor is storing the updated model parameters into the target storage medium, the processor can first retrieve the target model parameters that were previously stored into the target storage medium. For example, the processor can cache the target model parameters that needed to be stored into the target storage medium in memory instead of deleting them; thus, when model parameters need to be stored again, the target model parameters that were previously stored into the target storage medium can be quickly retrieved from memory.

[0012] Then, the processor calculates the difference between the updated model parameters and the target model parameters, and compresses the difference to obtain the compressed result. That is, it calculates the difference between the corresponding parameters in the updated model parameters and the target model parameters to determine the difference between each parameter in the updated model parameters and its corresponding parameter in the target model parameters, thus obtaining multiple difference values.

[0013] After obtaining the compression result, the processor stores the compression result to the target storage medium.

[0014] In this scheme, the amount of data obtained by compressing the difference between the model parameters is much smaller than that of the updated model parameters. Therefore, by storing the compressed result in the target storage medium, the storage space in the target storage medium can be effectively saved, thereby reducing storage costs.

[0015] In one possible implementation, the updated model parameters obtained based on the training results are stored in a first region of memory. That is, the processor stores the updated model parameters in the first region of memory. Then, during the process of storing the updated model parameters to the target storage medium, the processor first copies the updated model parameters from the first region to a second region of memory; then, the processor stores the updated model parameters located in the second region to the target storage medium. In other words, two identical copies of the updated model parameters are stored simultaneously in the same memory, one in the first region and the other in the second region. The processor actually performs operations on the updated model parameters in the second region, rather than on the model parameters in the first region.

[0016] In this solution, by using a dedicated memory area to store the model parameters that need to be persistently stored, it can be ensured that the model parameter persistence process performed by the processor will not affect the processor's normal update of model parameters, thereby ensuring the efficiency of model training.

[0017] In one possible implementation, during the process of the processor storing the updated model parameters to the target storage medium, if the processor determines that the conditions for performing the next round of updates to the parameters of the first model are met, the processor pauses the process of storing the updated model parameters to the target storage medium and begins to continue updating the parameters of the first model (i.e., updating the aforementioned updated model parameters). After the processor completes the next round of updates to the parameters of the first model, the processor then continues to execute the process of storing the updated model parameters to the target storage medium.

[0018] In this scheme, the processor prioritizes updating model parameters over persistently storing them. When an update is needed, the processor pauses the current persistent storage operation, allocating processing resources to the update process as much as possible to expedite it. The processor only persists model parameters when updates are not required, ensuring that persistent storage does not interfere with normal model training.

[0019] In one possible implementation, before the processor stores the updated model parameters to the target storage medium, the processor determines the first time period for performing the next round of updates to the parameters of the first model by monitoring the processor's utilization during operation. When the processor determines that the start time of the first time period has been reached, the processor can determine that the conditions for performing the next round of updates to the parameters of the first model are met.

[0020] In one possible implementation, by monitoring the processor's utilization during operation, the processor first determines the target time period during which the processor's utilization reaches a preset threshold; thus, based on the target time period, the processor can determine the time period during which the processor needs to update the parameters of the first model in the future (i.e., the first time period mentioned above).

[0021] In this scheme, since the time required for the accelerator to perform one training iteration on the first model is relatively fixed, by monitoring the changes in processor utilization when the model parameters are not persistently stored, the time period during which the processor needs to perform model parameter updates in the past can be effectively determined, and thus the time period during which the processor needs to perform model parameter updates in the future can be accurately predicted.

[0022] In one possible implementation, the processor determines that the conditions for performing the next round of parameter updates on the first model are met upon receiving the training results from the accelerator. That is, once the processor receives the training results from the accelerator, it means that the accelerator has completed one round of model training iterations, and therefore the processor needs to continue updating the parameters of the first model.

[0023] In this scheme, the need to update model parameters is determined by whether the processor receives the training results transmitted by the accelerator. This ensures that the processor can switch between updating model parameters and persistent storage operations in a simple and convenient way in practical applications, thereby improving the overall efficiency of model training.

[0024] In one possible implementation, the training results include the gradients of the parameters of the first model. During the accelerator's computation of the gradients of the first model's parameters, the accelerator passes the currently computed gradients to the processor. As the processor receives the gradients passed from the accelerator, it updates the parameters of the first model based on the currently acquired gradients.

[0025] Generally, the first model may contain millions or even hundreds of millions of parameters (the specific number of parameters depends on the actual structure of the first model). Each parameter has a corresponding gradient. Therefore, the accelerator will take a certain amount of time to calculate the gradient of each parameter in the first model, and the accelerator calculates the gradients of the parameters in an orderly manner. That is, the accelerator can calculate the gradients of the parameters in multiple batches. After calculating the gradient of each batch of parameters, the accelerator passes the calculated gradients to the processor, instead of calculating all the gradients and then passing them to the processor all at once. Each time the processor receives the gradients of a batch of parameters, it updates the parameters in the first model based on the received gradients, without waiting to receive all the gradients before starting the parameter update.

[0026] In this scheme, the accelerator sends the gradient of the first model's parameters to the processor asynchronously, and the processor also updates the first model's parameters asynchronously. This allows the processor to start updating the model parameters as early as possible, ensuring that the accelerator's calculation of parameter gradients and the processor's updating of model parameters can be performed in parallel as much as possible. This shortens the time it takes for the accelerator to obtain the updated model parameters and improves the overall training efficiency.

[0027] In one possible implementation, the training results include the gradients of the parameters of the first model. The processor also stores the updated optimizer state in the target storage medium. The updated optimizer state is gradient-based and includes the first and second moments of the gradient. This updated optimizer state is used to update the parameters of the first model incorporating the gradients.

[0028] In this scheme, by persistently storing the optimizer state used to update model parameters in the target storage medium, it can be ensured that when the model training progress needs to be restored later, the model training progress can be restored based on the gradient and optimizer state stored in the target storage medium, avoiding restarting the model training.

[0029] In one possible implementation, the processor can obtain the target optimizer state that was last stored in the target storage medium, which includes the first and second moments of the gradient of each parameter in the first model.

[0030] Then, the processor calculates the state difference between the updated optimizer state and the target optimizer state, and compresses the state difference to obtain the compressed optimizer state. That is, the updated optimizer state also includes the first and second moments of the gradient of each parameter in the first model. The difference between the first moments corresponding to the same parameter in the updated optimizer state and the target optimizer state can be calculated, resulting in multiple first-moment differences. Similarly, the difference between the second moments corresponding to the same parameter in the updated optimizer state and the target optimizer state can be calculated, resulting in multiple second-moment differences. This state difference includes both the aforementioned multiple first-moment differences and multiple second-moment differences.

[0031] Finally, the processor stores the compression results from the optimizer state to the target storage medium.

[0032] Secondly, a training and push acceleration system is provided, comprising: an accelerator and a processor; the accelerator is used to perform the i-th round of training of a first model to obtain training results and to pass the training results to the processor; the processor is used to update the parameters of the first model based on the training results to obtain updated model parameters and to pass the updated model parameters to the accelerator; during the process of the accelerator performing the (i+1)-th round of training of the first model based on the updated model parameters, the processor is also used to store the updated model parameters in a target storage medium.

[0033] In one possible implementation, the processor is specifically configured to: obtain the target model parameters previously stored in the target storage medium; calculate the difference between the updated model parameters and the target model parameters, and perform compression on the difference to obtain a compression result; and store the compression result in the target storage medium.

[0034] In one possible implementation, the processor is specifically configured to: store the updated model parameters in a first region of memory; copy the updated model parameters from the first region to a second region of memory; and store the updated model parameters located in the second region to a target storage medium.

[0035] In one possible implementation, the processor is specifically configured to: pause the process of storing the updated model parameters in the target storage medium if it is determined that the conditions for performing the next round of updates on the parameters of the first model are met during the process of storing the updated model parameters in the target storage medium; and continue the process of storing the updated model parameters in the target storage medium after the processor completes the next round of updates on the parameters of the first model.

[0036] In one possible implementation, the processor is further configured to: determine a first time period for the processor to perform the next round of updates to the parameters of the first model by monitoring the processor utilization during operation; the processor determines that the conditions for performing the next round of updates to the parameters of the first model are met, including: the processor determines that the first time period has arrived.

[0037] In one possible implementation, the processor is further configured to: determine a target time period during which the processor utilization reaches a preset threshold by monitoring the processor utilization during operation; and determine a first time period based on the target time period.

[0038] In one possible implementation, the processor determines that the conditions for performing the next round of updates to the parameters of the first model are met, including: the processor receiving the training results passed from the accelerator.

[0039] In one possible implementation, the training results include the gradients of the parameters of the first model, and the accelerator is used to pass the currently computed gradients to the processor during the process of the accelerator computing the gradients of the parameters of the first model.

[0040] During the process of the processor receiving gradients from the accelerator, the processor is used to update the parameters of the first model based on the currently acquired gradients.

[0041] In one possible implementation, the training results include gradients of the parameters of the first model. The processor is further configured to: store the updated optimizer state in a target storage medium, the updated optimizer state being determined based on the gradients and including the first and second moments of the gradients, the updated optimizer state being used to update the parameters of the first model in conjunction with the gradients. In one possible implementation, the accelerator includes an NPU, GPU, or TPU.

[0042] In one possible implementation, the processor is further configured to: obtain the target optimizer state previously stored in the target storage medium; calculate the state difference between the updated optimizer state and the target optimizer state, and perform compression on the state difference to obtain the compressed result of the optimizer state; and store the compressed result of the optimizer state in the target storage medium.

[0043] Thirdly, a training and acceleration system is provided, comprising: a processor, an accelerator, and a memory; the memory is used to store computer instructions, which, when executed by the processor, cause the training and acceleration system to perform any of the methods described above.

[0044] Fourthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, cause the computer to perform the methods of any of the above aspects.

[0045] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, enable the computer to perform the methods described above.

[0046] In a sixth aspect, a chip system is provided, the chip system including a processor and a communication interface for communicating with a module other than the chip shown, the processor for running computer programs or instructions such that an apparatus on which the chip system is mounted can perform the methods of any of the above aspects.

[0047] In a seventh aspect, a computing device is provided, the computing device including a training and acceleration system of the third aspect or a chip system of the sixth aspect, wherein the training and acceleration system or the chip system in the computing device is used to implement the operation steps of the method of any of the above aspects.

[0048] Eighthly, a computing device cluster is provided, comprising at least one computing device, wherein any one computing device is used to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects. Alternatively, some or all of the computing devices are used together to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects.

[0049] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0050] Figure 1 is a schematic diagram of persistent storage of checkpoints in related technologies;

[0051] Figure 2 is a schematic diagram of a checkpoint recovery model training method provided in this application;

[0052] Figure 3 is a schematic diagram of a system architecture provided in this application;

[0053] Figure 4 is a schematic diagram of a method for storing model checkpoints provided in this application;

[0054] Figure 5 is a schematic diagram of a method provided in this application for obtaining the difference of model parameters, performing compression, and then storing them;

[0055] Figure 6 is a schematic diagram of copying model parameters in memory according to this application;

[0056] Figure 7 is a schematic diagram of a processor alternately performing parameter updates and persistent storage according to this application;

[0057] Figure 8 is a schematic diagram of the process of training an execution model and persistent storage of checkpoints provided in this application;

[0058] Figure 9 is a schematic diagram of a checkpoint persistent storage provided in this application;

[0059] Figure 10 is a schematic diagram of the collaborative work process between a GPU and a processor provided in this application;

[0060] Figure 11 is a schematic diagram of a training acceleration system provided in this application;

[0061] Figure 12 is a schematic diagram of the structure of a computing device provided in this application;

[0062] Figure 13 is a schematic diagram of the structure of a computing device cluster provided in this application;

[0063] Figure 14 is a schematic diagram of another computing device cluster provided in this application;

[0064] Figure 15 is a schematic diagram of the structure of a chip provided in this application;

[0065] Figure 16 is a schematic diagram of the structure of a computer-readable storage medium provided in this application. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by this application are also applicable to similar technical problems.

[0067] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0068] To facilitate understanding, some technical terms used in this application will be introduced below.

[0069] (1) Neural Network

[0070] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0071] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0072] Generally speaking, an AI model refers to a model composed of neural networks.

[0073] (2) Loss Function

[0074] During neural network training, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the neural network becomes a process of minimizing this loss.

[0075] (3) Backpropagation algorithm

[0076] Neural networks can employ backpropagation (BP) to correct the parameters of the initial prediction model during training, thereby reducing the error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial prediction model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the prediction model, such as the weight matrix.

[0077] Specifically, during model training, the backpropagation algorithm is typically used to calculate the gradients of each node in the model. This allows for the adjustment of node weights based on the gradients of each node, thereby minimizing the model's loss function value. The gradient represents the rate of change of a function at a given point. Furthermore, the gradient of each node in the model can be determined by calculating its partial derivatives.

[0078] (4) Gradient descent method

[0079] Gradient descent is a first-order optimization algorithm commonly used in machine learning to recursively approximate a minimum-bias prediction model. To find a local minimum of a function using gradient descent, iterative searches must be performed at points a predetermined step distance away in the opposite direction of the gradient (or approximate gradient) at the current point on the function. Gradient descent is one of the most frequently used methods for solving the prediction model parameters of machine learning algorithms, i.e., unconstrained optimization problems.

[0080] Specifically, when finding the minimum value of the loss function, the gradient descent method can be used to iterate step by step to obtain the minimized loss function and the predicted model parameter values.

[0081] (5) Large Language Model (LLM)

[0082] Large language models are deep learning models trained on massive amounts of text data that can generate natural language text or understand the meaning of language text. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0083] Specifically, large language models are a technology that has emerged in recent years. Because large language models undergo meticulous data engineering and training processes, their parameters have learned a wealth of existing natural language processing knowledge. This knowledge can now replace humans in many language-related tasks, such as having large language models write code or perform text summarization.

[0084] (6) Optimizer

[0085] In the field of AI, an optimizer is an algorithm used to tune the parameters of a neural network model to minimize the loss function. The goal of an optimizer is to adjust the model's parameters based on the input data and the desired output label, thereby enabling the model to better fit the training data.

[0086] (7) Huffman coding

[0087] Huffman coding, also known as Huffman encoding, is an entropy coding (weighted coding) algorithm used for lossless data compression. In computer data processing, Huffman coding uses a variable-length encoding table to encode source symbols (such as a letter in a file). The variable-length encoding table is obtained by evaluating the probability of the source symbols occurring. Letters with a high probability of occurrence are given shorter codes, while those with a low probability of occurrence are given longer codes. This reduces the average length and expected value of the encoded string, thereby achieving lossless data compression.

[0088] (8) Parameter quantization

[0089] Parametric quantization refers to converting parameters represented with a higher number of bits into parameters represented with a lower number of bits, thereby saving storage space. However, because the number of bits representing the parameters is reduced, the precision of the parameters changes after quantization, making it a lossy data compression method.

[0090] In related technologies, to achieve persistent storage checkpoints, accelerators such as GPUs typically store the AI ​​model's parameters to specific storage media (e.g., hard drives or remote storage servers) after completing a certain number of iterations of the AI ​​model's updates. Since the write speed of the storage media used for persistently storing model parameters is usually slow, the persistent storage process often requires a considerable amount of time. Furthermore, with the widespread application of large-parameter models such as large language models, multimodal models, and hybrid expert models, the parameter scale of AI models is constantly increasing, thus requiring an increasing duration for persistent storage of model parameters. For example, widely used large language models have hundreds of millions of parameters, and the time required to persistently store these large numbers of model parameters even exceeds the time required for the AI ​​model to perform a single iteration of training.

[0091] In related technologies, the persistent storage of model parameters often requires interrupting the training process of the AI ​​model. The training process can only continue after the AI ​​model parameters have been stored, which affects the normal training of the AI ​​model and prolongs the training time.

[0092] For example, please refer to Figure 1, which is a schematic diagram of persistent storage of checkpoints in related technologies. As shown in Figure 1, during model training performed by the GPU, a persistent storage checkpoint is executed after a certain number of iterations of training on the model. For example, the model parameters are persistently stored once every 100 iterations of training on the model. While the GPU is storing checkpoints to the hard drive or a remote storage server, the GPU cannot perform model training again; that is, model training and persistent storage of checkpoints are sequential. Therefore, the duration of persistent storage of checkpoints becomes the duration of model training interruption. Thus, the more frequently checkpoints are persistently stored, the more times model training will be interrupted, ultimately resulting in a longer training time.

[0093] Therefore, to avoid the significant impact of persistent storage checkpoints on model training, related techniques typically reduce the frequency of persistent storage checkpoints, i.e., performing persistent storage checkpoints only at longer intervals, such as once every 500 iterations of training for each model pair. However, reducing the frequency of persistent storage checkpoints often brings other negative consequences.

[0094] For example, please refer to Figure 2, which is a schematic diagram of a checkpoint-based model training recovery method provided in this application. As shown in Figure 2, if the model training is interrupted due to an anomaly (e.g., a power outage or system crash), it is necessary to recover the model training based on the last persistently stored checkpoint. When recovering the model training, the training process that was executed before the training interruption needs to be re-executed based on the model parameters corresponding to the checkpoint. Therefore, if the frequency of persistently storing checkpoints is low, the training process that needs to be re-executed when recovering the model training will be longer, resulting in a significant impact on the overall training time of the model.

[0095] In view of this, this application provides a method for storing model checkpoints. By deploying the operation of updating model parameters during model training to be executed on the processor, the processor can retain the updated model parameters after performing the update operation. Thus, while the accelerator continues model training based on the updated model parameters, the processor can simultaneously store the retained model parameters in the target storage medium, achieving persistent storage of model parameters without interrupting the training process and avoiding impact on the normal training of the AI ​​model.

[0096] Furthermore, having the processor perform the model parameter update operation eliminates the need for the accelerator to store the large amount of state data (such as optimizer state) required for updating model parameters. This frees up more storage space on the accelerator to perform the model training process, effectively improving the efficiency of the accelerator in performing the model training process and shortening the overall model training time.

[0097] It should be noted that although shifting the operation of updating model parameters to the processor will involve communication between the processor and the accelerator, the communication speed between the accelerator and the processor is much faster than the speed at which the accelerator persists the model parameters to the target storage medium. Furthermore, a portion of the gradients calculated by the accelerator can be passed to the processor to perform the model parameter update (i.e., the processor does not need to wait for the accelerator to finish calculating the gradients of all parameters before it can start performing the model parameter update). Therefore, the communication process between the accelerator and the processor caused by deploying the operation of updating model parameters on the processor will not have a significant impact on the overall training process.

[0098] Please refer to Figure 3, which is a schematic diagram of a system architecture provided in this application. As shown in Figure 3, in this system architecture, the execution device 30 can be implemented by a single physical host (computing device) or multiple physical hosts (computing device cluster). The execution device 30 includes an accelerator 301 and a processor 302. The accelerator 301 can be hardware or software used for training or inference of artificial intelligence models, such as a graphics processing unit (GPU), a tensor processing unit (TPU), or a neural network processing unit (NPU). The processor 302 can include hardware with computing capabilities, such as a CPU.

[0099] The accelerator 301 executes the forward inference process and the backward calculation of model parameter gradients during model training, while the processor 302 executes the step of updating model parameters during training and returns the updated model parameters to the accelerator 301. Furthermore, the processor 302 also performs persistent storage of the model parameters obtained during training, thereby persistently storing the model parameters in the storage medium 31. The storage medium 31 is used for persistent storage of model parameters, meaning it can still store model parameters even in the event of power failure.

[0100] Optionally, the storage medium 31 can be located inside the execution device 30. For example, the storage medium 31 can be a hard disk inside the execution device 30. The storage medium 31 can also be located outside the execution device 30. For example, the storage medium 31 can be a storage medium in a storage server outside the execution device 30. In this case, the execution device 30 can transmit model parameters to the storage server where the storage medium 31 is located via a communication network or connection cable, enabling the storage server to persistently store the model parameters in the storage medium 31. The storage server can be a storage server located in a data center or a storage server located in the cloud for providing cloud storage services.

[0101] Please refer to Figure 4, which is a schematic diagram of a method for storing model checkpoints provided in this application. As shown in Figure 4, the method for storing model checkpoints includes the following steps 401-405.

[0102] Step 401: The accelerator performs the i-th round of training of the first model to obtain the training result.

[0103] In this application, the accelerator can be hardware specifically designed for performing AI calculations, such as a GPU, TPU, or NPU; this application does not impose any specific limitations on this. For ease of description, the following will use a GPU as an example of an accelerator. The first model is an AI model, such as a large language model, a deep neural network model, a multimodal model, a hybrid expert model, etc.; this application also does not impose any specific limitations on this.

[0104] Specifically, when the accelerator performs the i-th round of training on the first model (where i is an integer greater than or equal to 1), the accelerator inputs the acquired training data into the first model and processes the training data by running the first model to obtain the training result. The training result obtained by the accelerator during the training iteration process is used to update the parameters of the first model.

[0105] Optionally, the training result may include the gradients of the parameters of the first model. That is, after the accelerator inputs the training data into the first model and obtains the output of the first model, it calculates the gradients of the parameters in the first model through the backpropagation algorithm based on the difference between the output of the first model and the true result (i.e., the loss function value), thereby obtaining the training result.

[0106] In addition, the training result can also be the output of the first model or the loss function value, which is not specifically limited in this application.

[0107] Step 402: The accelerator passes the training results to the processor.

[0108] After the accelerator obtains the training results by executing the training iterations of the first model, the training results are stored in the storage medium used by the accelerator. For example, in the case of a GPU, the training results obtained by the GPU executing the training iterations of the first model are stored in the GPU's video memory. Therefore, the accelerator can send the training results to the memory used by the processor so that the processor can retrieve the training results passed by the accelerator from the memory.

[0109] In this application, the accelerator and the processor are deployed in the same execution device, such as on the same physical host. Therefore, the accelerator and the processor have a high communication speed (e.g., the accelerator and the processor communicate via the system bus), and the accelerator can transmit the training results to the processor in a short time.

[0110] Step 403: The processor updates the parameters of the first model based on the training results to obtain the updated model parameters.

[0111] After receiving the training results from the accelerator, the processor can update the parameters of the first model based on the training results, thereby obtaining the updated model parameters in the first model. For example, if the training results include the gradients of the parameters of the first model, the processor updates each parameter based on the gradient of each parameter in the first model, thereby obtaining each updated parameter.

[0112] Specifically, the processor can update the parameters of the first model by running an optimizer. The optimizer run by the processor can include, for example, a stochastic gradient descent (SGD) optimizer, an adaptive moment estimation (Adam) optimizer, etc. This application does not specifically limit the optimizer used to update the model parameters.

[0113] Optionally, if the training results include the gradients of the parameters of the first model, the accelerator may send the gradients of the parameters of the first model to the processor asynchronously, and the processor may update the parameters of the first model asynchronously.

[0114] Specifically, as the accelerator continuously calculates the gradients of the parameters of the first model, it passes the calculated gradients to the processor. Generally, the first model may contain millions or even hundreds of millions of parameters, and each parameter requires a corresponding gradient calculation. Therefore, the number of gradients to be calculated in the first model will be millions or even hundreds of millions. That is, the accelerator divides the parameters of the first model into multiple batches, calculates the gradients of the parameters of the first model in batches, and passes the calculated gradients to the processor after each batch of parameters has been calculated, instead of calculating all gradients and then passing them to the processor all at once. Similarly, as the processor continuously receives gradients from the accelerator, the processor can update the parameters of the first model based on the acquired gradients. That is, the processor updates the parameters in the first model based on the received gradients after receiving each batch of parameter gradients, without waiting to receive all gradients before starting the parameter update.

[0115] In this scheme, the accelerator sends the gradient of the first model's parameters to the processor asynchronously, and the processor also updates the first model's parameters asynchronously. This allows the processor to start updating the model parameters as early as possible, ensuring that the accelerator's calculation of parameter gradients and the processor's updating of model parameters can be performed in parallel as much as possible. This shortens the time it takes for the accelerator to obtain the updated model parameters and improves the overall training efficiency.

[0116] Of course, in some possible implementations, the accelerator can also use a synchronous transmission method to send the gradients of the first model's parameters to the processor, that is, wait until all gradients have been calculated before sending them to the processor. The processor can also use a synchronous update method to update the parameters of the first model, that is, the processor starts executing the model parameter update after receiving all gradients.

[0117] In step 404, the processor passes the updated model parameters to the accelerator.

[0118] After updating the model parameters, in order for the accelerator to continue training iterations of the first model, the processor needs to return the updated model parameters to the accelerator.

[0119] Similarly, when the processor updates the parameters of the first model asynchronously, it can pass the updated parameters to the accelerator as soon as it calculates a portion of the updated parameters, without having to wait for all parameters to be updated before passing them to the accelerator, thus shortening the time required to pass the updated parameters to the accelerator.

[0120] Step 405: During the (i+1)th round of training of the first model by the accelerator based on the updated model parameters, the processor stores the updated model parameters in the target storage medium.

[0121] After the processor calculates the updated model parameters, it no longer needs to participate in the training process of the first model (until the accelerator delivers the next training result). Therefore, the processor can perform the persistent storage process of the model parameters, that is, store the updated model parameters in the target storage medium. Since the processor has already passed the updated model parameters to the accelerator, and the accelerator is no longer responsible for the persistent storage of model parameters, when the accelerator continues to execute the training iteration process of the first model based on the updated model parameters, the processor simultaneously persists the updated model parameters in the target storage medium. That is, the accelerator's execution of the training iteration process of the first model based on the updated model parameters and the processor's persistent storage of the updated model parameters in the target storage medium are performed synchronously.

[0122] The target storage medium is a storage medium used to persist storage model parameters, such as a hard disk connected to the execution device, or a storage medium in a storage server that the execution device connects to via a network. This application does not make any specific limitations on this.

[0123] In this solution, by deploying the operation of updating model parameters during model training to be executed on the processor, the processor can retain the updated model parameters after performing the update operation. Thus, while the accelerator continues model training based on the updated model parameters, the processor can simultaneously store the retained model parameters in the target storage medium, achieving persistent storage of model parameters without interrupting the training process and avoiding impact on the normal training of the AI ​​model.

[0124] Furthermore, since the communication speed between the accelerator and the processor is much faster than the speed at which the accelerator persists the model parameters to the target storage medium, the communication process between the accelerator and the processor caused by deploying the model parameter update operation on the processor will not have a significant impact on the overall training process. Moreover, having the processor perform the model parameter update operation eliminates the need for the accelerator to store the large amount of state data required for updating the model parameters (such as optimizer states), thus freeing up more storage space on the accelerator for the model training process. This effectively improves the efficiency of the accelerator in performing model training and shortens the overall model training time.

[0125] Optionally, when the processor uses the optimizer to update the parameters of the first model, in addition to obtaining the gradient of the first model's parameters, it often also needs to obtain the optimizer's state in order to update the parameters of the first model. That is, the processor uses the optimizer to perform the parameter update of the first model based on the gradient of the first model's parameters and the optimizer's state.

[0126] Therefore, to ensure that the model's training progress can be recovered based on the data persistently stored in the target storage medium, the processor can also store the optimizer state in the target storage medium. The optimizer state is determined based on the gradients of the parameters of the first model. For example, if the processor uses the Adam optimizer, the optimizer state specifically includes the first and second moments of the gradients of the first model's parameters. The optimizer state is specifically used to update the parameters of the first model by incorporating the gradients of the first model's parameters. Furthermore, if the processor uses other optimizers, the optimizer state can also consist only of the first moments of the gradients of the first model's parameters or other state data; this application does not specifically limit the optimizer state.

[0127] For example, in any training iteration of the first model, the first and second moments of the gradient can be represented by the following Equations 1 and 2. n = a*m n-1 +(1-a)*g Formula 1 v n = b*v n-1 +(1-b)*g 2 Formula 2

[0128] Where, m n The first moment of the gradient of the first model during the nth training iteration is represented by 'a'; 'a' represents the coefficient; 'm' represents the first moment of the gradient of the first model during the nth training iteration. n-1 The first moment of the gradient of the first model during the (n-1)th training iteration is represented by g; g represents the gradient; v n The second moment of the gradient of the first model during the nth training iteration is represented by b; the coefficient is represented by v. n-1 It represents the second moment of the gradient of the first model during the (n-1)th training iteration.

[0129] From Equations 1 and 2 above, we can see that the first moment of the gradient can be understood as the average value used to track the gradient, and the second moment of the gradient can be understood as the average value used to track the square of the gradient. By comprehensively considering both the first and second moments of the gradient, the Adam optimizer can calculate the corresponding update step size to update the model parameters. Specifically, based on the first moment of the gradient (i.e., the average of all previous gradients and the current gradient), the Adam optimizer can ensure that the gradient change is not too large when updating model parameters, ensuring a smooth and stable gradient transition. Based on the second moment of the gradient, the Adam optimizer can generate adaptive learning rates for different parameters.

[0130] In this scheme, by persistently storing the optimizer state used to update model parameters in the target storage medium, it can be ensured that when the model training progress needs to be restored later, the model training progress can be restored based on the gradient and optimizer state stored in the target storage medium, avoiding restarting the model training.

[0131] Understandably, when the first model has a large number of parameters, the amount of data that needs to be persistently stored on the target storage medium will also be substantial, thus occupying a significant amount of storage space. Therefore, to save storage space and reduce storage costs, this application proposes a scheme to compress the updated model parameters.

[0132] Specifically, please refer to Figure 5, which is a schematic diagram of calculating the difference of model parameters, performing compression, and then storing them according to this application. As shown in Figure 5, in step 405 above, in order to efficiently store the updated model parameters into the target storage medium, the processor can first retrieve the target model parameters that were previously stored into the target storage medium. For example, the processor caches the target model parameters that needed to be stored into the target storage medium in memory without deleting them; in this way, when the model parameters need to be stored again, the target model parameters that were previously stored into the target storage medium can be quickly retrieved from memory.

[0133] Then, the processor calculates the difference between the updated model parameters and the target model parameters, and compresses the difference to obtain the compressed result. Specifically, since both the updated and target model parameters are multiple parameters of the same model (i.e., the first model), the differences between the corresponding parameters in the updated and target models can be calculated to determine the difference between each parameter in the updated model parameters and its corresponding parameter in the target model, resulting in multiple differences. By compressing these multiple differences, a compressed result with reduced data volume can be obtained. The compression method for the model parameter differences can be lossless compression (e.g., Huffman coding) or lossy compression (e.g., parameter quantization). The specific compression method can be selected based on the actual situation, and this application does not impose specific limitations here.

[0134] Generally speaking, lossy compression can produce smaller data volumes, but it can also alter the precision of the data. Lossless compression, on the other hand, can maintain the precision of the data, but the resulting data volume is usually larger than that produced by lossy compression.

[0135] Secondly, after obtaining the compression result, the processor can store the compression result to the target storage medium. Compared to the updated model parameters, the amount of data obtained by compressing the difference between the model parameters is much smaller than that of the updated model parameters. Therefore, by storing the compression result to the target storage medium, storage space in the target storage medium can be effectively saved, thereby reducing storage costs.

[0136] Specifically, regarding the multiple model parameters in the first model, although the updated model parameters may be different and vary considerably, as training progresses, the changes in most model parameters tend to stabilize, meaning that the changes in most model parameters are very similar. Therefore, in this case, by calculating the difference between the updated model parameters and the model parameters that were previously stored on the target storage medium, we can obtain the differences among multiple model parameters, and most of these differences are very close. Thus, by compressing these closely related differences, effective data compression can be achieved, resulting in a compressed data volume.

[0137] It should be noted that during the training iteration of the first model, when the processor needs to store the model parameters to the target storage medium for the first time, the processor cannot obtain the model parameters that were previously stored to the target storage medium. Therefore, when storing the model parameters for the first time, the processor stores all the model parameters to the target storage medium, rather than storing the compressed result of the difference between the model parameters.

[0138] In addition, when the processor needs to store the optimizer state to the target storage medium, the processor can also calculate the difference between the currently updated optimizer state and the previous optimizer state that needs to be stored to the target storage medium, obtain the optimizer state difference, compress the optimizer state difference, and then store the compressed result in the target storage medium.

[0139] For example, the processor can obtain the target optimizer state last stored in the target storage medium, which includes the first and second moments of the gradient of each parameter in the first model.

[0140] Then, the processor calculates the state difference between the updated optimizer state and the target optimizer state, and compresses the state difference to obtain the compressed optimizer state. That is, the updated optimizer state also includes the first and second moments of the gradient of each parameter in the first model. The difference between the first moments corresponding to the same parameter in the updated optimizer state and the target optimizer state can be calculated, resulting in multiple first-moment differences. Similarly, the difference between the second moments corresponding to the same parameter in the updated optimizer state and the target optimizer state can be calculated, resulting in multiple second-moment differences. This state difference includes both the aforementioned multiple first-moment differences and multiple second-moment differences.

[0141] Finally, the processor stores the compression results from the optimizer state to the target storage medium.

[0142] Understandably, when the number of model parameters is large, persisting the model parameters to the target storage medium may take a long time, especially when the processor also needs to calculate the differences between the model parameters and perform compression on those differences. Since the process of persisting the model parameters is executed in parallel with the accelerator's training of the first model, after the accelerator completes one round of training iterations for the first model, it will request the processor to continue updating the model parameters. Therefore, to avoid the processor-executed model parameter persistence process affecting the model parameter update process, this application proposes using a dedicated memory area to store the model parameters that need to be persisted.

[0143] For example, please refer to Figure 6, which is a schematic diagram of copying model parameters in memory according to this application. As shown in Figure 6, after the processor updates the model parameters based on the training results, it stores the updated model parameters obtained based on the training results in a first region of memory.

[0144] The processor then copies the updated model parameters from the first region of memory to the second region within the same memory. In this way, two identical copies of the updated model parameters are stored simultaneously in the same memory location, one in the first region and the other in the second region.

[0145] Secondly, the processor stores the updated model parameters (e.g., the model parameters updated in the first round) located in the second region into the target storage medium. That is, when the processor performs the persistent storage of the updated model parameters, it actually performs operations on the updated model parameters in the second region, not on the model parameters in the first region. In this way, if another round of model parameter updates is needed (e.g., a second round of model parameter updates is required), the model parameters stored in the first region are updated in the second round, while the model parameters stored in the second region remain unchanged. Thus, the process of performing the second round of model parameter updates does not affect the persistent storage of the model parameters updated in the first round, thereby achieving parallel execution of the processor's persistent storage of the first model's model parameters and the accelerator's training of the first model.

[0146] For example, if the processor obtains the training results for the next training iteration from the accelerator while it is persistently storing the updated model parameters in the second region, the processor can then continue updating the model parameters in the first region based on these results, so that the updated model parameters can be returned to the accelerator. Since the model parameters to be persistently stored on the target storage medium have already been copied to the second region, even if the processor updates the model parameters in the first region, it will not affect the persistent storage of the model parameters. This ensures that the persistent storage of the model parameters does not affect the processor's normal updating of model parameters, thereby guaranteeing the efficiency of model training.

[0147] It should be noted that when the processor needs to calculate the differences between model parameters, perform compression, and then store them to the target storage medium, the processor can also store the model parameters that were previously stored to the target storage medium in the second region. Furthermore, the differences between the model parameters calculated by the processor and the compression results can also be stored in the second region. This ensures that the operations performed by the processor in the first and second regions are independent and do not affect each other. In other words, the first region in memory can be understood as a dedicated region for the processor and accelerator to perform model training, specifically for updating and storing the updated model parameters; the second region in memory can be understood as a dedicated region for the processor to persist the model parameters to the target storage medium.

[0148] When dealing with a large number of model parameters or poor write performance of the target storage medium, persisting the model parameters to the target storage medium may take a long time. This can easily lead to a situation where the time required for persistent storage exceeds the time it takes for the accelerator to perform one training iteration of the first model. Consequently, when the accelerator requests the processor to perform the next model parameter update, the processor is still processing the model parameter persistence. Therefore, to ensure that the persistence of model parameters does not affect the training of the first model and thus guarantee its training efficiency, this application proposes to increase the priority of the processor in performing model parameter updates. This ensures that the processor can utilize more computing resources to perform model parameter updates each time, thereby maximizing the speed of model parameter updates.

[0149] For example, if the processor needs to continue updating the parameters of the first model in the next round when it is storing the updated model parameters to the target storage medium during the process of the processor storing the updated model parameters to the target storage medium, the processor will pause storing the updated model parameters to the target storage medium and start continuing to update the parameters of the first model (i.e., updating the updated model parameters mentioned above).

[0150] After the processor completes the next round of parameter updates for the first model, the processor continues to store the updated model parameters into the target storage medium.

[0151] In other words, when the processor is persistently storing the updated model parameters, if it finds that it needs to continue updating the parameters of the first model after persistently storing a portion of the updated model parameters to the target storage medium, the processor can pause the persistence of the remaining model parameters to the target storage medium. Then, the processor will switch to updating the parameters of the first model. Once the processor has completed updating the parameters of the first model, it will then continue persisting the remaining model parameters to the target storage medium.

[0152] In short, this scheme prioritizes updating model parameters over persistently storing them. When an update is needed, the processor pauses the current persistent storage operation, allocating processing resources to the update process to expedite it. Only when updating parameters is not required will the processor persist them, ensuring that persistent storage does not interfere with normal model training.

[0153] For example, please refer to Figure 7, which is a schematic diagram of a processor alternately performing parameter updates and persistent storage according to this application. As shown in Figure 7, after the GPU performs one round of iterative training of the model, the GPU passes the gradient of the model parameters to the processor, thereby enabling the processor to perform model parameter updates. After the processor returns the updated model parameters to the GPU, the GPU continues to perform the next round of iterative training of the model, while the processor synchronously performs checkpoint compression (i.e., calculates the difference between model parameters and compresses the difference). During the process of the processor performing checkpoint compression, the GPU has completed another round of iterative training of the model and passed the corresponding gradient of the model parameters to the processor. Therefore, at this time, the processor pauses the execution of checkpoint compression and begins to perform model parameter updates again.

[0154] After the processor completes the model parameter update again, it continues to perform checkpoint compression. Following checkpoint compression, the processor begins persistent storage of the checkpoints, storing the compression results to the target storage medium. Since persistent storage of checkpoints primarily consumes data transfer channel bandwidth and does not consume excessive processor computational resources, the persistent storage operation can be paused during model parameter updates, allowing the process of storing checkpoints to the target storage medium to run concurrently with the model parameter update process. Alternatively, in some implementations, the processor can pause persistent storage of checkpoints while updating model parameters; that is, the process of storing checkpoints to the target storage medium also needs to be paused, thus freeing up more processing resources for model parameter updates.

[0155] In this application, the processor can determine when to perform model parameter updates in a variety of ways.

[0156] In one possible implementation, during the process of the processor storing the updated model parameters to the target storage medium, if the processor determines that the conditions for performing the next round of updates to the parameters of the first model are met, the processor pauses the process of storing the updated model parameters to the target storage medium and begins to continue updating the parameters of the first model (i.e., updating the aforementioned updated model parameters). After the processor completes the next round of updates to the parameters of the first model, the processor then continues to execute the process of storing the updated model parameters to the target storage medium.

[0157] For example, before the processor stores the updated model parameters to the target storage medium, the target time period during which the processor utilization reaches a preset threshold can be determined by monitoring the processor's utilization during operation. That is, the processor utilization is monitored when the processor is not performing persistent storage of model parameters. If the processor utilization reaches the preset threshold, it indicates that the processor is performing model parameter updates, and thus the target time period can be considered as the time period during which the processor performs model parameter updates. The preset threshold can be a threshold set according to actual conditions, such as 85% or 90%, and this application does not specifically limit it.

[0158] In this way, based on the target time period, the processor can determine the time period (i.e., the first time period) during which it needs to perform the next round of parameter updates for the first model. Therefore, it can pause the persistent storage of model parameters during this time period. That is, when the processor determines that it has reached the start time of the first time period, it can determine that the conditions for performing the next round of parameter updates for the first model are met. Since the time required for the GPU to perform one training iteration on the first model is relatively fixed, by monitoring the changes in processor utilization when persistent storage of model parameters is not performed, the time periods during which the processor needs to perform model parameter updates in historical time periods can be effectively determined, thereby accurately predicting the time periods during which the processor needs to perform model parameter updates in future time periods.

[0159] For example, suppose the following pattern is observed during processor monitoring: In a 10-second cycle, the processor utilization reaches a preset threshold for the first 2 seconds of each cycle. This indicates that the processor needs to update model parameters within the first 2 seconds of each 10-second cycle. Therefore, a 10-second cycle can be set, and the persistent storage of model parameters can be paused for the first 2 seconds of each cycle, allowing the processor to concentrate its resources on updating the model parameters. Furthermore, in some cases, to ensure sufficient time for model parameter updates, the persistent storage of model parameters can be paused for the first 3 or 4 seconds of each cycle, ensuring the successful completion of the update.

[0160] In another possible implementation, the processor determines that the conditions for performing the next round of parameter updates on the first model are met when it receives the training results from the accelerator; that is, it needs to continue updating the parameters of the first model. In other words, once the processor receives the training results from the accelerator, it means that the accelerator has completed one round of model training iterations, and therefore the processor needs to continue updating the parameters of the first model.

[0161] In this scheme, the need to update model parameters is determined by whether the processor receives the training results transmitted by the accelerator. This ensures that the processor can switch between updating model parameters and persistent storage operations in a simple and convenient way in practical applications, thereby improving the overall efficiency of model training.

[0162] To facilitate understanding, the following will use GPU as the accelerator and hard disk as the target storage medium as an example to explain in detail the process of performing model training and persistent storage of checkpoints in practical applications.

[0163] For example, please refer to Figure 8, which is a schematic diagram of a process for performing model training and persistent storage of checkpoints provided in this application. As shown in Figure 8, the process for performing model training and persistent storage of checkpoints includes the following steps 801-810.

[0164] Step 801: The GPU performs one training iteration of the first model to obtain the gradient of the model parameters.

[0165] When the GPU executes the first training iteration of the first model for the first time, it does so based on the initialized model parameters. When the GPU is not executing the first training iteration for the first model for the first time, it executes the training iteration based on the updated model parameters obtained from the processor.

[0166] Furthermore, the GPU executes one training iteration of the first model by inputting the training input into the first model after the model parameters have been updated, and constructing a loss function based on the output of the first model and the true result; then, it executes the backpropagation algorithm based on the value of the loss function to obtain the gradient of each model parameter in the first model.

[0167] Step 802: The GPU passes the gradients of the model parameters to the processor.

[0168] Step 803: The processor updates the parameters of the first model based on the gradient of the model parameters and the optimizer state to obtain the updated model parameters.

[0169] After obtaining the gradients of the model parameters passed by the GPU, the processor first updates the optimizer state based on these gradients. The specific method for updating the optimizer state can be found in Equations 1 and 2 above. Then, the processor updates the parameters of the first model based on the gradients of the model parameters and the updated optimizer state, thus obtaining the updated model parameters.

[0170] Step 804: The processor passes the updated model parameters to the GPU.

[0171] After obtaining the updated model parameters, the processor can pass the updated model parameters to the GPU so that the GPU can continue to perform the training iteration process on the first model based on the updated model parameters, that is, 801-804 are executed in a loop.

[0172] In addition, to improve the speed at which the GPU obtains updated model parameters, in steps 801-804 above, the GPU may pass the gradient of the calculated part of the model parameters to the processor as soon as it calculates the gradient of that part of the model parameters; similarly, the processor may pass the updated model parameters to the GPU as soon as it calculates the updated model parameters.

[0173] Step 805: The processor determines whether the number of times the model parameters have been updated is a multiple of 100.

[0174] After each update of the model parameters, the processor can determine whether the number of model parameter updates is a multiple of 100, that is, whether the number of model parameter updates reaches 100, 200, 300, 400, etc.

[0175] In other words, in this application, the model parameters need to be persistently stored once every 100 model parameter updates. It should be noted that this application uses the example of persistent storage every 100 model parameter updates to illustrate the frequency of persistent storage of model parameters. In practical applications, the frequency of persistent storage of model parameters can also be set to other values, and this application does not impose any specific limitations on this.

[0176] Step 806: The processor determines whether the model parameters and optimizer status have been stored on the hard disk.

[0177] If the processor determines that the number of times the model parameters have been updated has reached an integer multiple of 100, it means that the processor needs to perform a checkpoint persistence. Therefore, the processor first determines whether the model parameters and optimizer state have already been stored on the hard disk.

[0178] Step 807: If the model parameters and optimizer state are not stored on the hard disk, the processor stores the updated model parameters and optimizer state on the hard disk.

[0179] If the model parameters are not stored on the hard disk, it means that the processor cannot calculate the differences between the model parameters and the differences between the optimizer states. The processor then stores the updated model parameters and optimizer states on the hard disk.

[0180] Step 808: If the model parameters and optimizer state are already stored on the hard disk, the processor calculates the difference between the updated model parameters and the target model parameters previously stored on the hard disk, as well as the difference between the updated optimizer state and the optimizer state previously stored on the hard disk.

[0181] If the model parameters and optimizer states are already stored on the hard drive, it means that the processor can obtain the differences between the model parameters and the differences between the optimizer states. Therefore, the processor calculates the difference between the updated model parameters and the target model parameters that were previously stored on the hard drive, as well as the difference between the updated optimizer states and the optimizer states that were previously stored on the hard drive.

[0182] It should be noted that in steps 807-808, in order to ensure that the persistent storage of checkpoints does not affect the processor's normal execution of model parameter updates, the model parameters and optimizer state that need to be persistently stored can be copied from the first region of memory to the second region of memory, so as to ensure that the first region of memory is dedicated to executing model parameter updates and the second region of memory is dedicated to executing persistent storage of checkpoints.

[0183] Step 809: The processor compresses the differences between model parameters and the differences between optimizer states to obtain the compression result.

[0184] The processor can compress the differences between model parameters and the differences between optimizer states using either lossy or lossless compression methods. The specific compression method used can be determined based on the actual situation and is not specifically limited here.

[0185] Optionally, when the optimizer state includes the first and second moments of the gradient, the differences between the first moments of the gradients of different parameters are relatively small. Therefore, it is possible to effectively compress the differences between the first moments of the gradients of different parameters, i.e., to obtain results with a smaller data volume. However, since the differences between the second moments of the gradients of different parameters may not be small (because the second moment is obtained based on the square of the gradient), this application may not directly compress the differences between the second moments of the gradient. Instead, it may set another parameter based on the first and second moments and compress the difference of this parameter.

[0186] Specifically, assuming the first moment of the gradient is m and the second moment of the gradient is v, then we can set the parameter c = vm. 2 When compressing the difference between optimizer states, one can first calculate the first moment m of the current gradient. t The first moment m of the gradient stored previously t-1 The difference Δm between them, and the calculation of the current parameter c. tCompared to the previously stored parameter c t-1 The difference Δc between them is calculated, and then the difference Δm and the difference Δc are compressed respectively to obtain the compressed result of the optimizer state.

[0187] Step 810: The processor stores the compression result to the hard disk.

[0188] After calculating the compression result, the processor can store the compression result on the hard disk.

[0189] In other words, the processor only stores the full set of model parameters and optimizer state to the hard disk the first time it persists the checkpoint to the hard disk; after the processor has persisted the checkpoint to the hard disk, it only persists the compressed result of the difference between the model parameters and the compressed result of the difference between the optimizer state to the hard disk.

[0190] For example, please refer to Figure 9, which is a schematic diagram of a checkpoint persistent storage provided in this application. As shown in Figure 9, the update results of model parameters and optimizer state are stored in a certain area of ​​memory used by the processor. When checkpoint persistence needs to be performed, the update results of model parameters and optimizer state are copied to another area of ​​memory. In this way, when checkpoint persistent storage is performed for the first time, the update results of model parameters and optimizer state copied to another area of ​​memory can be stored on the hard disk, thereby obtaining checkpoint 1, which includes all model parameters and all optimizer states.

[0191] In any subsequent execution of checkpoint persistence, the updated model parameters and optimizer state copied to another region of memory are compared with the previously stored model parameters and optimizer state on disk. The differences are then calculated and compressed to obtain the compressed results of the model parameter differences and optimizer state differences. By combining these compressed results of model parameter differences and optimizer state differences at each execution, checkpoint 2, checkpoint 3, ..., checkpoint N can be obtained.

[0192] In addition, during the process of updating model parameters and performing checkpoint compression and persistent storage, the processor will be subject to a computation scheduling process to ensure that the processor prioritizes the execution of model parameters and performs checkpoint compression and persistent storage only when idle.

[0193] For example, please refer to Figure 10, which is a schematic diagram of the collaborative work between a GPU and a processor provided in this application. As shown in Figure 10, when the GPU executes the training iteration process of the first model, it mainly performs the forward computation and backward propagation of the model. The forward computation of the model refers to inputting the input data into the first model and calculating the loss function value based on the output result of the first model; the backward propagation of the model refers to calculating the gradient of each model parameter in the first model based on the loss function value.

[0194] The processor is responsible for updating model parameters and performing checkpoint compression and persistent storage, which are executed by the processor in different memory regions. By monitoring the processor's utilization when checkpoint compression and persistent storage are not performed, the time periods during which the processor periodically performs model parameter updates can be determined. This allows for processor scheduling, ensuring that the processor prioritizes model parameter updates when they are needed, pausing checkpoint compression and persistent storage. Checkpoint compression and persistent storage are only performed when the processor does not need to perform model parameter updates.

[0195] The method provided in this application has been described in detail above. Next, the device provided in this application for performing the above method will be described.

[0196] Please refer to Figure 11, which is a schematic diagram of the structure of a training and pushing acceleration system provided in this application. As shown in Figure 11, the training and pushing acceleration system includes: an accelerator 1101 and a processor 1102; the accelerator 1101 is used to perform the i-th round of training of the first model to obtain the training result, and transmit the training result to the processor 1102; the processor 1102 is used to update the parameters of the first model based on the training result to obtain the updated model parameters, and transmit the updated model parameters to the accelerator 1101; during the process of the accelerator 1101 performing the (i+1)-th round of training of the first model based on the updated model parameters, the processor 1102 is also used to store the updated model parameters in the target storage medium.

[0197] In one possible implementation, the processor 1102 is specifically used to: obtain the target model parameters that were previously stored in the target storage medium; calculate the difference between the updated model parameters and the target model parameters, and perform compression on the difference to obtain a compression result; and store the compression result in the target storage medium.

[0198] In one possible implementation, the processor 1102 is specifically used to: store the updated model parameters in a first region of memory; copy the updated model parameters from the first region to a second region of memory; and store the updated model parameters located in the second region to a target storage medium.

[0199] In one possible implementation, the processor 1102 is specifically configured to: during the process of storing the updated model parameters to the target storage medium, if it is determined that the conditions for performing the next round of updates to the parameters of the first model are met, then pause the process of storing the updated model parameters to the target storage medium; after the processor 1102 completes the next round of updates to the parameters of the first model, then continue the process of storing the updated model parameters to the target storage medium.

[0200] In one possible implementation, the processor 1102 is further configured to: determine the first time period for the processor 1102 to perform the next round of updates to the parameters of the first model by monitoring the utilization rate of the processor 1102 during operation; the processor 1102 determines the conditions for performing the next round of updates to the parameters of the first model, including: the processor 1102 determines the start time point of the current first time period.

[0201] In one possible implementation, the processor 1102 is further configured to: determine a target time period during which the utilization of the processor 1102 reaches a preset threshold by monitoring the utilization of the processor 1102 during operation; and determine a first time period based on the target time period.

[0202] In one possible implementation, processor 1102 determines that the conditions for performing the next round of updates to the parameters of the first model are met, including: processor 1102 receiving the training results passed by accelerator 1101.

[0203] In one possible implementation, the training result includes the gradient of the parameters of the first model, and in the process of calculating the gradient of the parameters of the first model, the accelerator 1101 is used to pass the currently calculated gradient to the processor 1102.

[0204] During the process of receiving gradients from accelerator 1101, processor 1102 updates the parameters of the first model based on the currently acquired gradients.

[0205] In one possible implementation, the training result includes the gradient of the parameters of the first model, and the processor 1102 is further configured to: store the updated optimizer state in the target storage medium, the updated optimizer state being updated based on the gradient, and the updated optimizer state including the first moment and the second moment of the gradient, the updated optimizer state being used to update the parameters of the first model in conjunction with the gradient.

[0206] In one possible implementation, the processor 1102 is further configured to: obtain the target optimizer state previously stored in the target storage medium; calculate the state difference between the updated optimizer state and the target optimizer state, and perform compression on the state difference to obtain the compressed result of the optimizer state; and store the compressed result of the optimizer state in the target storage medium.

[0207] In one possible implementation, the accelerator 1101 includes an NPU, a GPU, or a TPU.

[0208] Please refer to Figure 12, which is a schematic diagram of a computing device provided in this application. The computing device 1200 shown in Figure 12 can be used to execute the model checkpoint storage method provided in this embodiment. As shown in Figure 12, the computing device 1200 includes: a bus 1202, a processor 1204, an accelerator 1205, a memory 1206, and a communication interface 1208. The processor 1204, the memory 1206, and the communication interface 1208 communicate with each other via the bus 1202. The computing device 1200 can be a server or a terminal device. It should be understood that this application does not limit the number of processors, accelerators, and memories in the computing device 1200.

[0209] Bus 1202 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 12, but this does not imply that there is only one bus or one type of bus. Bus 1202 can include pathways for transmitting information between various components of computing device 1200 (e.g., memory 1206, processor 1204, communication interface 1208).

[0210] The processor 1204 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0211] Accelerator 1205 may be, for example, a neural network processing unit (NPU), a graphics processing unit (GPU), or a tensor processing unit (TPU).

[0212] The memory 1206 may include volatile memory, such as random access memory (RAM). The processor 1204 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0213] The memory 1206 stores executable program code, and the processor 1204 executes the executable program code to implement the above-described model checkpoint storage method. That is, the memory 1206 stores instructions for executing the model checkpoint storage method.

[0214] The communication interface 1208 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1200 and other devices or communication networks.

[0215] It should be understood that the computing device 1200 according to this application is used to execute the model checkpoint storage method as shown in Figures 4 to 10, and can correspond to the execution of the corresponding subject in the method according to this application, which will not be described in detail here for the sake of brevity.

[0216] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0217] Please refer to Figure 13, which is a schematic diagram of a computing device cluster provided in this application. As shown in Figure 13, the computing device cluster includes at least one computing device 1200. The memory 1206 in one or more computing devices 1200 in the computing device cluster may store the same instructions for executing the model checkpointing storage method.

[0218] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster may also store partial instructions for executing the stored method of model checkpointing. In other words, a combination of one or more computing devices 1200 can jointly execute the instructions for executing the stored method of model checkpointing.

[0219] It should be noted that the memory 1206 in different computing devices 1200 in the computing device cluster can store different instructions, which are used to execute some functions of the training and acceleration system.

[0220] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 14 illustrates one possible implementation. Figure 14 is also a schematic diagram of another computing device cluster structure provided in this application. As shown in Figure 14, in computing device cluster 1400, two computing devices 1200A and 1200B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0221] It should be understood that the functions of computing device 1200A shown in Figure 14 can also be performed by multiple computing devices 1200. Similarly, the functions of computing device 1200B can also be performed by multiple computing devices 1200.

[0222] It should be understood that the computing device 1200 or computing device cluster 1400 in this application can correspond to the training and acceleration system in Figure 11 of this application, and can correspond to the corresponding entity executing the model checkpoint storage method in Figure 4 of this application. Furthermore, the above and other operations and / or functions of each module in the computing device 1200 or computing device cluster 1400 are respectively for implementing the corresponding flow of the model checkpoint storage method in Figure 4, and for the sake of brevity, will not be elaborated further here.

[0223] Specifically, please refer to Figure 15, which is a schematic diagram of the structure of a chip provided in this application. The chip can be represented as a neural processing unit (NPU), a graphics processing unit (GPU), or a tensor processing unit (TPU). The following description will use an NPU 1500 as an example. The NPU 1500 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 1503, which is controlled by a controller 1504 to retrieve matrix data from memory and perform multiplication operations.

[0224] In some implementations, the arithmetic circuit 1503 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1503 is a two-dimensional pulsating array. The arithmetic circuit 1503 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1503 is a general-purpose matrix processor.

[0225] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1508.

[0226] Unified memory 1506 is used to store input and output data. Weight data is directly transferred to weight memory 1502 via Direct Memory Access Controller (DMAC) 1505. Input data is also transferred to unified memory 1506 via DMAC.

[0227] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1509.

[0228] The Bus Interface Unit (BIU) 1510 is used by the instruction fetch memory 1509 to fetch instructions from external memory, and also by the memory access controller 1505 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0229] The DMAC is mainly used to move input data from external memory DDR to unified memory 1506, or to weight data to weight memory 1502, or to input data to input memory 1501.

[0230] The vector computation unit 1507 includes multiple processing units that further process the output of the computation circuit 1503 when needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0231] In some implementations, the vector computation unit 1507 can store the processed output vector in the unified memory 1506. For example, the vector computation unit 1507 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1503, such as linear interpolation of feature planes extracted by a convolutional layer, or, for example, a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 1507 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1503, for example, for use in subsequent layers of the neural network.

[0232] The instruction fetch buffer 1509 connected to the controller 1504 is used to store the instructions used by the controller 1504;

[0233] Unified memory 1506, input memory 1501, weighted memory 1502, and instruction fetch memory 1509 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0234] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0235] It should be understood that the chip in Figure 15 of this application may correspond to the accelerator mentioned in this application, or be deployed on the computing device 1200 or computing device cluster 1400 of this application. Furthermore, the chip in Figure 15 of this application may correspond to the corresponding entity executing the model checkpoint storage method in Figure 4 of this application. The above and other operations and / or functions of each module in the chip respectively implement the corresponding flow of the model checkpoint storage method in Figure 4, which will not be elaborated here for the sake of brevity. In addition, in this application, the chip structure is not limited to the chip structure shown in Figure 15, and may include more or fewer hardware structures to implement the functions of the method shown in Figure 4.

[0236] Referring to Figure 16, which is a schematic diagram of the structure of a computer-readable storage medium provided in this application. This application also provides a computer-readable storage medium in which, in some embodiments, the method disclosed in Figure 4 above can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0237] Figure 16 schematically illustrates a conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein, the example computer-readable storage medium including a computer program for executing computer processes on a computing device.

[0238] In one embodiment, the computer-readable storage medium 1600 is provided using a signal bearer medium 1601. The signal bearer medium 1601 may include one or more program instructions 1602, which, when executed by one or more processors, can provide the functions or parts thereof described above with reference to FIG4.

[0239] In some examples, the signal carrying medium 1601 may include a computer-readable medium 1603, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video disc (DVD), a digital magnetic tape, a memory, ROM, or RAM, etc.

[0240] In some embodiments, the signal-bearing medium 1601 may comprise a computer-recordable medium 1604, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, and so on. In some embodiments, the signal-bearing medium 1601 may comprise a communication medium 1605, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, and so on). Therefore, for example, the signal-bearing medium 1601 may be transmitted by a wireless communication medium 1605 (e.g., a wireless communication medium conforming to the IEEE 1202.X standard or other transmission protocols).

[0241] One or more program instructions 1602 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1602 conveyed to the computing device via a computer-readable medium 1603, a computer-recordable medium 1604, and / or a communication medium 1605.

[0242] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0243] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0244] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0245] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions can be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0246] The above are merely specific embodiments of this application. Those skilled in the art can conceive of variations or substitutions based on the specific embodiments provided in this application, and all such variations or substitutions should be covered within the protection scope of this application.

Claims

1. A model checkpoint storage method, characterized by, The method is applied to a training acceleration system, the system comprising a processor and an accelerator, and the method comprising: the accelerator performing i-th round training of a first model to obtain a training result and passing the training result to the processor; the processor updating parameters of the first model based on the training result to obtain updated model parameters and passing the updated model parameters to the accelerator; in a process in which the accelerator performs i+1-th round training of the first model based on the updated model parameters, the processor stores the updated model parameters into a target storage medium.

2. The method of claim 1, wherein, The processor stores the updated model parameters into a target storage medium, comprising: the processor obtaining target model parameters stored into the target storage medium last time; the processor calculating a difference between the updated model parameters and the target model parameters and performing compression on the difference to obtain a compression result; the processor storing the compression result into the target storage medium.

3. The method of claim 1 or 2, wherein the processor stores the updated model parameters into a target storage medium, comprising: the processor storing the updated model parameters in a first area in a memory; the processor copying the updated model parameters from the first area to a second area in the memory; the processor storing the updated model parameters in the second area into the target storage medium.

4. The method according to any one of claims 1 to 3, characterized in that, The processor stores the updated model parameters into a target storage medium, comprising: in a process in which the processor stores the updated model parameters into a target storage medium, if the processor determines that a condition for performing a next round of update on the parameters of the first model is met, the processor suspends the process of storing the updated model parameters into the target storage medium; after the processor completes the next round of update performed on the parameters of the first model, the processor continues to perform the process of storing the updated model parameters into the target storage medium.

5. The method of claim 4, wherein, The method further comprises: the processor determining a first time period in which the processor performs a next round of update on the parameters of the first model by monitoring utilization of the processor during running; the processor determining that the condition for performing the next round of update on the parameters of the first model is met, comprising: the processor determining that a starting time point of the first time period is currently reached.

6. The method of claim 5, wherein, The processor determines a first time period in which the processor performs a next round of update on the parameters of the first model by monitoring utilization of the processor during running, comprising: the processor determines a target time period in which utilization of the processor reaches a preset threshold by monitoring the utilization of the processor during running; the processor determines the first time period based on the target time period.

7. The method of claim 4, wherein, The processor determines that the condition for performing the next round of update on the parameters of the first model is met, comprising: the processor receiving the training result passed by the accelerator.

8. The method according to any one of claims 1 to 7, characterized in that, The training result comprises gradients of parameters of the first model, and the accelerator delivers the training result to the processor, comprising: In the process of calculating the gradients of the parameters of the first model, the accelerator delivers the currently calculated gradients to the processor; The processor updates the parameters of the first model based on the training result, comprising: In the process of receiving the gradients delivered by the accelerator, the processor updates the parameters of the first model based on the currently obtained gradients.

9. The method according to any one of claims 1 to 8, characterized in that, The training result comprises gradients of parameters of the first model, and the method further comprises: The processor stores the updated optimizer state into the target storage medium, the optimizer state being updated based on the gradients, and the updated optimizer state comprising first and second moments of the gradients, the updated optimizer state being used to update the parameters of the first model in combination with the gradients.

10. The method of claim 9, wherein, The processor stores the optimizer state into the target storage medium, comprising: The processor obtains a target optimizer state stored into the target storage medium last time; The processor calculates a state difference between the updated optimizer state and the target optimizer state, and performs compression on the state difference to obtain a compression result of the optimizer state; The processor stores the compression result of the optimizer state into the target storage medium.

11. A training acceleration system, comprising: Comprise: An accelerator and a processor; The accelerator is configured to perform i-th round training of a first model to obtain a training result, and deliver the training result to the processor; The processor is configured to update parameters of the first model based on the training result to obtain updated model parameters, and deliver the updated model parameters to the accelerator; In the process that the accelerator performs i+1-th round training of the first model based on the updated model parameters, the processor is further configured to store the updated model parameters into a target storage medium.

12. The system of claim 11, wherein, The processor, specifically configured to: Obtain a target model parameter stored into the target storage medium last time; Calculate a difference between the updated model parameters and the target model parameter, and perform compression on the difference to obtain a compression result; Store the compression result into the target storage medium.

13. The system of claim 11 or 12, wherein The processor, specifically configured to: Store the updated model parameters in a first area in a memory; Copy the updated model parameters from the first area to a second area in the memory; Store the updated model parameters in the second area into the target storage medium.

14. The system of any of claims 11-13, wherein, The processor, specifically configured to: In the process that the processor stores the updated model parameters into the target storage medium, if it is determined that a condition for performing next round update on the parameters of the first model is met, then the process of storing the updated model parameters into the target storage medium is paused. After the processor completes the next round of update performed on the parameters of the first model, the process of storing the updated model parameters into the target storage medium is continued.

15. The system of claim 14, wherein, The processor is further configured to: determine a first time period for the processor to perform the next round of update on the parameters of the first model by monitoring the utilization of the processor during operation; the determination that the condition for performing the next round of update on the parameters of the first model is satisfied comprises determining that a starting time point of the first time period is currently reached.

16. The system of claim 15, wherein, The processor is further configured to: determine a target time period in which the utilization of the processor reaches a preset threshold by monitoring the utilization of the processor during operation; determine the first time period based on the target time period.

17. The system of claim 14, wherein the determination by the processor that the condition for performing the next round of update on the parameters of the first model is satisfied comprises: receiving, by the processor, the training result delivered by the accelerator.

18. The system of any of claims 11-17, wherein, The training result comprises gradients of the parameters of the first model, and in the process of calculating the gradients of the parameters of the first model, the accelerator is configured to deliver the currently calculated gradients to the processor. In the process of receiving the gradients delivered by the accelerator, the processor is configured to update the parameters of the first model based on the currently obtained gradients.

19. The system of any of claims 11-18, wherein, The training result comprises gradients of the parameters of the first model, and the processor is further configured to: store an optimizer state into the target storage medium, the optimizer state being determined based on the gradients, and the optimizer state comprising first and second moments of the gradients, the optimizer state being used in combination with the gradients to update the parameters of the first model.

20. The system of any of claims 11-19, wherein, The processor is further configured to: obtain a target optimizer state stored into the target storage medium last time; calculate a state difference between the updated optimizer state and the target optimizer state, and perform compression on the state difference to obtain a compression result of the optimizer state; store the compression result of the optimizer state into the target storage medium.

21. A computing device, comprising: The computing device comprises a memory, an accelerator and a processor; the memory stores a code, and the processor is configured to execute the code, and when the code is executed, the computing device performs the method of any one of claims 1 to 10.

22. A cluster of computing devices, characterized in that, The computing device comprises at least one computing device, and each computing device comprises a processor, an accelerator and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the operation steps of the method of any one of claims 1 to 10.

23. A computer storage medium, comprising, The computer storage medium stores instructions, and the instructions cause the computer to implement the method of any one of claims 1 to 10 when executed by the computer.

24. A computer program product, characterised in that, The computer program product stores instructions, and the instructions cause the computer to implement the method of any one of claims 1 to 10 when executed by the computer.

Images