Lightweight checkpoint device and method for learning time series data

Abstract

Disclosed are a lightweight checkpoint device and method for learning time series data. The checkpoint method is performed by a checkpoint device and comprises the steps of: training a learning model; generating full checkpoints by performing a checkpointing operation on the learning model when a first epoch is completed during the training of the learning model; performing the checkpointing operation at each checkpointing interval to generate partial checkpoints including a weight tensor including only elements having values different from those of elements of a weight tensor generated as a result of a previous checkpointing operation; and generating new full checkpoints by merging the generated partial checkpoints when an epoch corresponding to a merge point is completed during the training.

Description

A lightweight checkpointing device and method for learning time series data

[0001] The present invention relates to a lightweight checkpoint technique for stably performing learning of time series data, and more particularly, to a checkpoint technique for stably and without interruption learning of time series data generated in various environments such as mobile networks.

[0002] Next-generation mobile networks, 5G / 6G, are rapidly increasing the number of devices and considering diverse applications, increasing the diversity and complexity of end-to-end mobile networks. Consequently, managing the vast amount of data generated by multiple devices, at multiple levels, and on multiple timescales is crucial. To address this, mobile network operators are using deep learning technology to learn from data and extract diverse information to manage and operate their networks.

[0003] Most of this data is time-series data with temporal dependencies and relationships within the sequential data. Data is collected at various time intervals (e.g., 1 ms to 1 sec), and a significant amount of data is collected after a certain period of time. Training large data sets requires a long training time, ranging from several hours to several days. This can lead to various failures during training, such as network issues, hardware failures, and system failures. Furthermore, failures (or failures) during training can increase training runtime by up to two times because they cause GPU work to be lost. Therefore, checkpointing techniques have been proposed, which snapshot and save the model at each epoch during training.

[0004] Checkpointing is a crucial technique for rapid failure recovery to ensure smooth training progress. Checkpoints are snapshotted at regular intervals and stored in (remote) storage. To recover the model from failures and retrain it, the most recent checkpoint is loaded. However, applying existing checkpointing techniques to time-series data training poses the following challenges. First, checkpointing can increase the overall training runtime depending on the frequency. That is, while frequent checkpointing reduces training loss due to failures (or failures), it also increases the overall training execution time. Conversely, failures can increase training loss but reduce training time. Ultimately, increased training time makes it difficult to manage mobile networks in a timely manner, and increased training loss can lead to inaccurate network analysis information for mobile network operators. Therefore, determining an appropriate checkpointing cycle must consider these tradeoffs.

[0005] Meanwhile, weight tensors generated through time-series data training (e.g., LSTM) consist of floating-point data and can reach hundreds of megabytes per epoch. Weight tensors naturally account for a significant portion of the model, significantly impacting checkpoint size. Saving full-size checkpoints incurs significant overhead. Therefore, efficient checkpoint size management is crucial to minimize the storage requirements for checkpoints.

[0006] Checkpointing is a crucial feature for quickly recovering from failures (or errors) to ensure smooth training progress. Checkpoints are snapshotted at set intervals and stored in (remote) storage. To recover and retrain a model after a failure, the most recent checkpoint is loaded. However, checkpointing not only impacts the overall training runtime depending on the cycle, but also causes storage capacity shortages as the model size increases, leading to a concurrent increase in checkpoint size. Therefore, a new checkpointing system is required for stable training of time-series data.

[0007] Therefore, the present invention proposes a lightweight checkpointing system (named CheckBullet in this specification) that minimizes storage requirements and ensures fast recovery while hardly increasing learning runtime in learning time series data.

[0008] The technical task of the present invention is to provide a lightweight checkpoint device and method for learning time series data.

[0009] According to one embodiment of the present invention, a checkpoint method performed by a checkpoint device includes the steps of performing learning on a learning model, when a first epoch is completed during learning on the learning model, performing a checkpointing operation on the learning model to generate a full checkpoint, performing a checkpointing operation at each checkpointing interval, but generating a partial checkpoint including a weight tensor that includes only elements having different values ​​from elements of a weight tensor generated as a result of a previous checkpointing operation, and when an epoch corresponding to a merging point during learning is completed, merging the generated partial checkpoints to generate a new full checkpoint.

[0010] According to an embodiment of the present invention, a lightweight checkpoint device and method for learning time series data have the effect of enabling stable and uninterrupted learning of time series data generated in various environments such as mobile networks.

[0011] In order to more fully understand the drawings cited in the detailed description of the present invention, a detailed description of each drawing is provided.

[0012] FIG. 1 illustrates a checkpoint device according to one embodiment of the present invention.

[0013] Figure 2 is a drawing for explaining the operation of the checkpoint device illustrated in Figure 1.

[0014] Figure 3 illustrates an example of flexible checkpointing when the epoch times of each model are different.

[0015] Figure 4 illustrates an example of weight tensor deduplication.

[0016] Figure 5 shows the average L2 loss according to the quantization level.

[0017] Figure 6 shows the average L2 loss during model retraining using checkpoints stored in remote storage after training failure.

[0018] Figure 7 shows the average L2 loss overhead according to the number of failure occurrences.

[0019] Figure 8 shows the average runtime overhead according to the checkpoint technique.

[0020] Figure 9 shows the total recovery time according to the checkpoint method.

[0021] Figure 10 shows the change in cumulative recovery time according to the number of failures.

[0022] Figure 11 shows the total recovery time according to β.

[0023] Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed in this specification are merely illustrative for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described in this specification.

[0024] Embodiments according to the concept of the present invention may have various modifications and take various forms, and thus, embodiments are illustrated in the drawings and described in detail herein. However, this is not intended to limit embodiments according to the concept of the present invention to specific disclosed forms, but rather includes all modifications, equivalents, or alternatives falling within the spirit and technical scope of the present invention.

[0025] While terms such as "first" or "second" may be used to describe various components, these components should not be limited by these terms. These terms are only intended to distinguish one component from another. For example, a first component may be referred to as a "second component," and similarly, a second component may be referred to as a "first component," without departing from the scope of the present invention.

[0026] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

[0027] The terminology used herein is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, it should be understood that the terms "comprises" or "has" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in this specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0028] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0029] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings attached to this specification. However, the scope of the patent application is not limited or restricted by these embodiments. The same reference numerals in each drawing represent the same elements.

[0030] First, we introduce recent research on fault-tolerant mobile networks and various examples of using checkpointing.

[0031] Research is continuously being conducted to ensure high availability in mobile networks.

[0032] REINFORCE (SG Kulkarni, G. Liu, KK Ramakrishnan, M. Arumaithurai, T. Wood, and X. Fu, "REINFORCE: Achieving Efficient Failure Resiliency for Network Function Virtualization-Based Services," IEEE / ACM Transactions on Networking, vol. 28, no. 2, pp. 695-708, Apr. 2020.) presents a framework that ensures high availability of software-based Network Functions (NFs) by providing high resilience for both NFs and NF chains.

[0033] L25GC (V. Jain, H. Chu, S. Qi, C. Lee, H. Chang, C. Hsieh, K. Ramakrishnan, and J. Chen, "L25GC: A Low Latency 5G Core Network based on High-Performance NFV Platforms," ​​in Proc. ACM Special Interest Group on Data Communication (SIGCOMM), pp. 143-157, Aug. 2022.) focuses on reducing the control plane latency of 5G core networks and node / link failures of high-performance NFV platforms, thereby improving the performance of the data plane.

[0034] SAFE (R. Xia, H. Dai, J. Zheng, R. Gu, X. Wang, W. Wang, and G. Chen, "SAFE: Service Availability via Failure Elimination Through VNF Scaling," IEEE / ACM Transactions on Networking, early access, Jan. 2023, doi:10.1109 / TNET.2022.3233488.) utilized VNF scaling technology to build a failure recovery system for VNF instances running on a virtualization platform. However, these studies mainly focused on providing stable software-based virtualized network functions and did not consider the issues of DL (Deep Learning) training systems for mobile communication networks.

[0035] Recently, there is a growing trend of building fault-tolerant systems for mobile communication networks using DL.

[0036] TrustFTSR (K. Haseeb, T. Saba, A. Rehman, Z. Ahmed, H. H. Song, and H. H. Wang, "Trust Management With Fault-Tolerant Supervised Routing for Smart Cities Using Internet of Things," IEEE Internet of Things Journal, no. 9, no. 22, pp. 22608-22617, Nov. 2022.) proposed a fault-tolerant relay system that utilizes supervised learning techniques for trust management in IoT networks.

[0037] Sharma (N. Sharma, U. Agarwal, S. Shaurya, S. Mishra, and O. J. Pandey, "Energy-Efficient and QoS-Aware Data Routing in Node Fault Prediction Based IoT Networks," IEEE Transactions on Network and Service Management, early access, Apr. 2023, doi:10.1109 / TNSM.2023.3268676.) presented an optimal data routing method based on a novel joint node fault prediction in IoT networks. This approach utilizes a novel unsupervised learning-based Local Outlier Factor (LOF) to predict future faults and adopts a Q-learning framework for multi-hop data routing.

[0038] STAD (A. Dridi, C. Boucetta, SE Hammami, H. Afifi, and H. Moungla, "STAD: Spatio-Temporal Anomaly Detection Mechanism for Mobile Network Management," IEEE Transactions on Network and Service Management, vol. 18, no. 1, pp. 894-906, Mar. 2021.) is an automated framework for detecting temporal anomalies, enabling network operators to immediately analyze anomalies to prevent failures caused by network performance degradation.

[0039] Elsayed (MS Elsayed, N. Le-Khac, S. Dev, and AD Jurcut, "Network Anomaly Detection Using LSTM Based Autoencode," inProc. ACM Symposium on QoS and Security for Wireless and Mobile Networks (Q2SWinet), pp. 37-45, Nov. 2020.) proposed a study to detect anomaly-based attacks using LSTM autoencoders and one-class support vector machines (OC-SVMs).

[0040] Most recent research on fault-tolerant systems utilizes DL to recognize and assess network conditions. However, for this research to be successful, a technology capable of reliably training large-scale data must first be considered.

[0041] Checkpointing has been widely adopted to recover execution state after a system failure.

[0042] CHORE (P. Sigdel, X. Yuan, and N. Tzeng, "Realizing Best Checkpointing Control in Computing Systems," IEEE Transactions on Parallel and Distributed Systems, no. 32, vol. 2, pp. 315-329, Feb. 2021.) estimated real-time MTBF (Mean Time Between Failures) to adaptively derive optimal checkpointing strategies in real systems. This enables high-performance computing (HPC) systems to track actual failure instances and perform optimal checkpointing to save failure recovery costs.

[0043] ULSC 2 -RADIC (A. Wong, E. Heymann, D. Rexachs, and E. Luque, "Middleware to Manage Fault Tolerance Using Semi-Coordinated Checkpoints," IEEE Transactions on Parallel and Distributed Systems, no. 32, vol. 2, pp. 254-268, Feb. 2021.) proposed semi-coordinated checkpointing that only stores interactions between process groups to reduce the cost of computational work due to compute node failures in message-passing interface (MPI) applications.

[0044] In addition, the studies of Y. Sharma et al. (Y. Sharma, J. Taheri, W. Si, D. Sun, and B. Javadi, "Dynamic Resource Provisioning for Sustainable Cloud Computing Systems in the Presence of Correlated Failures," IEEE Transactions on Sustainable Computing, no. 6, vol. 4, pp. 641-654, Oct. / Dec. 2021.) and J. Liu et al. (J. Liu, P. Wang, J. Zhou, and K. Li, "McTAR: A Multi-Trigger Checkpointing Tactic for Fast Task Recovery in MapReduce," IEEE Transactions on Services Computing, no. 14, vol. 6, pp. 1824-1836, Nov. / Dec. 2021.) introduced a checkpointing technique for fault tolerance based on task execution in computing systems.

[0045] However, DNN training consumes significant time for a single task, and appropriate checkpointing must be performed mid-training. Therefore, checkpointing techniques tailored to the characteristics of DNN training must be developed.

[0046] Recently, several studies have been introduced that utilize checkpointing in DNNs for robust model learning.

[0047] Check-N-Run (A. Eisenman, K. K. Matam, S. Ingram, D. Mudigere, R. Krishnamoorthi, K. Nair, M. Smelyanskiy, and M. Annavaram, "Check-N-Run: a Checkpointing System for Training Deep Learning Recommendation Models," in Proc. USENIX Symposium on Networked Systems Design and Implementation (NSDI), pp. 929-943, Apr. 2022.) applies incremental checkpointing and quantization techniques for recommendation model-based training.

[0048] CheckFreq (J. Mohan, A. Phanishayee, and V. Chidambaram, "CheckFreq: Frequent, Fine-Grained DNN Checkpointing," in Proc. USENIX Conference on File and Storage Technologies (FAST), pp. 203-216. Feb. 2021.) is an automatic fine-grained checkpointing framework that determines and dynamically adjusts the checkpointing frequency.

[0049] These operations can maintain training accuracy while reducing training execution time overhead. However, in time-series data generated by mobile networks, the weight tensors, which are composed of floating-point numbers, change significantly at each epoch. Therefore, directly utilizing incremental and fixed checkpointing for data generated by mobile networks at each epoch, which significantly impacts training execution time and storage capacity, is not a trivial task. In particular, frequent checkpointing by CheckFreq can increase checkpoint latency in mobile networks, leading to training bottlenecks. In terms of disaster recovery, Check-N-Run requires a long recovery time because it must read all previous checkpoints for recovery. Consequently, a lightweight checkpointing technique suitable for training and recovery based on time-series data generated by mobile networks is needed.

[0050] Below, the checkpoint device, method, and system proposed in the present invention are described.

[0051] FIG. 1 illustrates a checkpoint device according to one embodiment of the present invention.

[0052] Referring to FIG. 1, a checkpoint device, which may also be referred to as a checkpoint system, includes four main components: 1) a controller, 2) a GPU cluster, 3) a CPU cluster, and 4) storage. Depending on the embodiment, the checkpoint device may further include storage. In other words, the checkpoint device may refer to a computing device that includes at least one CPU, at least one GPU, a storage device, and a controller.

[0053] The controller, which may also be referred to as a controller, a control unit, or a control section, includes at least one of a checkpointing interval optimizer, a merging point optimizer, a network data collector, and a training failure manager.

[0054] A checkpointing interval optimizer, which may also be named a checkpointing interval decision unit, a checkpointing interval decision unit, etc., determines the checkpointing frequency for each model. A merge point optimizer, which may also be named a merge point decision unit, a merge point decision unit, etc., combines (or merges) multiple checkpoints during training. A network data collector, which may also be named a data collector, a data collection unit, etc., collects and / or preprocesses the generated time series data. A learning failure manager, which may also be named a learning failure detection unit, a learning failure detection unit, etc., detects failures during training.

[0055] The controller plays a crucial role in providing training parameters to the GPU cluster and specifying checkpointing frequency and merging points (which may refer to merging intervals) to the CPU cluster. In the present invention, it is assumed that the controller is aware of data and training information generated by mobile networks, etc.

[0056] A GPU cluster consisting of multiple workers handles model training and inference, and calls a CPU cluster consisting of multiple workers based on checkpointing frequency and merge points. The CPU cluster snapshots the model and performs checkpointing at the epoch level.

[0057] Storage, which may also be referred to as a storage device or storage unit, stores checkpoints and models generated by GPU and CPU clusters. Specifically, checkpoints generated by the CPU cluster are stored in the checkpoint storage included in the storage, and models generated after training (or during training) on ​​the GPU cluster are stored in the model storage included in the storage.

[0058] The proposed checkpointing device (CheckBullet) must copy model parameters for further processing and storage. However, if snapshots and checkpointing are performed during training, this affects the training runtime. Therefore, the proposed checkpointing device (CheckBullet) utilizes a structure that separates a GPU cluster for training and a CPU cluster for snapshots and checkpointing, allowing for parallel execution (i.e., simultaneous training and checkpointing).

[0059] Specifically, the CPU cluster operates at the end of an epoch (i.e., after backward propagation) and snapshots (or creates snapshots of) the model for checkpointing. Simultaneously, the GPU cluster can perform training for the next epoch in parallel without interrupting training. This concurrent training and checkpointing can provide the following benefits: Since the CPU cluster generates checkpoints without affecting the GPU cluster, the model training execution time is barely increased. Furthermore, the additional computational cost caused by checkpointing on the GPU cluster can be reduced. Typically, model training takes a long time, resulting in significant resource overhead on the GPU cluster, while the CPU cluster experiences relatively low overhead. The proposed checkpointing device (CheckBullet) utilizes the available resources of the CPU cluster during training to achieve robust and stable model training.

[0060] Below, the specific operation of the proposed checkpoint device is described.

[0061] The checkpointing device can perform 1) flexible determination of checkpointing intervals, 2) deduplication of weight tensors, and 3) merging of partial checkpoints. The operation process is described in detail below.

[0062] The operation of the checkpoint device (CheckBullet) includes both checkpointing and model recovery, as illustrated in Figure 2. First, the controller collects time-series data and sends a training command to the GPU cluster. Then, the GPU cluster starts training, and upon completion of the first epoch, it sends a checkpointing interval request (or a checkpointing interval calculation request) to the controller (① in Figure 2(a)). After receiving the checkpointing interval from the controller, the GPU cluster calls the CPU cluster for checkpointing and continues training. In other words, the GPU cluster performs the next epoch without interrupting training. Meanwhile, in response to the call from the GPU cluster, the CPU cluster creates a model snapshot and quantizes the weight tensors contained in the model. Afterwards, the full checkpoint, including the model structure (e.g., layers and hyperparameters) and the weight tensors, is saved to the checkpoint repository (② in Figure 2(a)).

[0063] When the first epoch corresponding to the checkpointing interval is completed, the GPU cluster requests a merge point from the controller and receives the merge point from the controller (③ in Figure 2(b)). The GPU cluster then calls the CPU cluster and continues training. Meanwhile, the CPU cluster can perform checkpointing in response to the GPU cluster's call to obtain a new quantized weight tensor. The CPU cluster compares the newly quantized weight tensor with the previously stored weight tensor and stores only the changed weights in storage (i.e., partial checkpointing) (④ in Figure 2(b)).

[0064] During model training, when the merging point determined in ③ of Fig. 2(b) is reached, the CPU cluster compares two consecutive checkpoints with the minimum training loss among the stored checkpoints and generates a merged checkpoint (⑤ of Fig. 2(c)). Thereafter, the CPU cluster stores the merged checkpoint and deletes the previously stored checkpoint (⑥ of Fig. 2(c)). At this time, the merging operation of the CPU cluster may be performed in response to a call from the GPU cluster. Furthermore, the merging of two consecutive checkpoints may not generate a complete checkpoint. In this case, at least one previous checkpoint may be additionally merged.

[0065] If a failure occurs during training, the controller detects this and sends a retraining command to the corresponding GPU cluster (⑦ in Figure 2(d)). The GPU cluster requests checkpoint information for retraining from the CPU cluster. Consequently, the CPU cluster retrieves the saved checkpoint based on the minimum training loss, performs dequantization, and transfers the restored weight tensor to the GPU cluster (⑧ in Figure 2(d)). Finally, the GPU cluster completes all training and stores the trained model in model storage (⑨ in Figure 2(d)).

[0066] Below, we describe the details of the proposed checkpointing mechanism (CheckBullet). First, we describe how to set checkpoint intervals by considering model characteristics and the timing of failures. Furthermore, we introduce a weight tensor deduplication technique that removes duplicate weights between two consecutive checkpoints. Finally, we introduce a technique for merging partial checkpoints for fast recovery.

[0067] Determine checkpointing interval

[0068] In Deep Learning (DL), models have different epoch times due to various factors, such as hyperparameters and batch sizes. For example, if a model uses a large batch size to process more data at once, memory requirements may increase and epoch times may become longer. In such situations, a fixed checkpointing interval can lead to the following problems: a short checkpointing interval can lead to more frequent and unnecessary checkpoints in case of failure, while a long checkpointing interval can lead to longer recovery times to recover existing training loss after a failure. Therefore, in this invention, we propose a flexible checkpointing interval determination method that adjusts the checkpointing interval by considering potential failures.

[0069] I represents the time it takes for the model to perform one epoch, and E represents the total number of epochs required to complete the entire training. I and E are known at the end of the first epoch of model training. Therefore, the total training time required, T, is given by I × E.

[0070] Intuitively, optimal checkpointing is performed just before a failure occurs. Therefore, the proposed checkpointing device (CheckBullet) utilizes the concept of Mean Time Between Failures (MTBF) as a metric to determine the checkpointing interval of the model. T F Mean Time Between Failure (MTBF), denoted by , represents the average time interval between two failures and can be used to evaluate the operational reliability of a system. In particular, a high MTBF indicates that the system operates stably for a long period of time and has fewer failures. According to the study of S. Di et al. (S. Di, M.S. Bouguerra, L. Bautista-gomez, and F. Cappello, "Optimization of Multi-level Checkpoint Model for Large Scale HPC Applications," in Proc. IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 19-24, May 2014.), the checkpoint device (CheckBullet) can calculate the MTBF at the end of each epoch. Here, the epoch time I is T F Assuming that the failure intervals follow an exponential distribution, the MTBF T F is expressed as mathematical expression 1.

[0071] [Mathematical Formula 1]

[0072]

[0073] In mathematical expression 1, B FC is the cost of a full checkpoint in terms of the time required for checkpointing, and can mean the time taken to perform a full checkpoint.

[0074] After completing the first epoch, the proposed checkpoint device (CheckBullet) calculates the checkpointing interval H from the controller and performs checkpointing in the background before a failure occurs (i.e., T F It may be desirable to perform checkpointing beforehand. Therefore, based on the total number of epochs E, the checkpoint device (CheckBullet) determines the checkpointing interval in units of epochs. Specifically, T F If is greater than T, checkpointing can be skipped because there is less chance of failure during execution. Otherwise, the checkpointing interval H during model training is can be set (or calculated) as .

[0075] Figure 3 illustrates an example of flexible checkpointing when the epoch times of each model are different. The models in Figure 3 go through a total of 10 epochs, and the I of the models is calculated as 0.8, 1.5, and 2 at the end of the first epoch. In addition, B FC The values ​​are measured as 0.038, 0.17, and 0.254 hours. At the end of the first epoch, the controller, using Equation 1, T F are calculated as 8.4, 6.6, and 7.8 hours. Model 1 is, due to its relatively short epoch time (i.e., T F > T), model training is completed before a failure occurs. Therefore, for model 1, no checkpointing interval H may be determined. On the other hand, for models 2 and 3, T is T FBecause it is larger, the checkpointing interval H must be calculated. Furthermore, since Model 2 has a shorter epoch time than Model 3, H is longer. Specifically, Models 2 and 3 can perform partial checkpointing every 4 and 3 epochs, respectively. Furthermore, Model 3 can generate a single merged checkpoint by collecting three partial checkpoints based on the merge points calculated by the controller.

[0076] Below, we describe weight tensor deduplication.

[0077] Time-series data is primarily analyzed using LSTM models due to the need to capture long-term relationships for time-domain prediction. The size of the LSTM model is primarily affected by the weight tensor, which consists of floating-point numbers. This complexity can be reduced by compressing the weights into fewer bits through quantization. The proposed checkpointing device (CheckBullet) utilizes this quantization technique to reduce the checkpoint size while maintaining the training loss.

[0078] The proposed checkpoint device (CheckBullet) can quantize the weights of each layer using linear quantization. Specifically, the weights of each layer are [0, 2 n - 1] Within the range, it can be converted from floating-point values ​​to integers through n-bit quantization. The set of indices of the checkpointing interval is K = {1, 2, … , K}, and the set of layers at k ∈ K is L = {1, 2, … , L}. The weight tensor W is generated at the end of the epoch, is given as . Here, m is the dimension of the input data, and n represents the dimension of the hidden state (or cell). For uniform distribution in a narrower scale, the layer of checkpoint index k Linear scale for is calculated as in mathematical formula 2.

[0079] [Equation 2]

[0080]

[0081] In mathematical expression 2, is k within is the weight tensor. and is k within It represents the maximum and minimum values ​​among the elements of the weight tensor. Based on, in k The quantized value within is calculated by element-wise multiplication of the weight tensor and the scaling factor, as shown in Equation 3.

[0082] [Equation 3]

[0083]

[0084] While quantization significantly reduces spatial complexity, the quantized weight tensors of a layer still retain a significant size. To address this, weight tensor deduplication techniques can be utilized to remove redundant values ​​from the quantized weight tensors. Because values ​​within a weight tensor are unlikely to repeat across multiple epochs, the checkpointing device (CheckBullet) can capture short-term weight changes and remove redundant weights. Weight tensor deduplication techniques work by quantizing the k-th weight tensor and comparing it to the (k-1)-th weight tensor, storing only the weights with different values ​​for each layer (i.e., partial checkpointing).

[0085] A detailed description of weight tensor deduplication is provided in Algorithm 1. At the end of the first epoch, the CPU cluster quantizes the weight tensor and performs a full checkpointing to save all parameters required for model training. This includes the quantized weight tensor (see lines 3–8). Then, at each interval H, the kth weight tensor is quantized (see lines 9–14), and only the changed weights are saved by comparing them with the (k-1)th weight tensor (see lines 15–17). This process is repeated until model training is complete.

[0086] [Algorithm 1]

[0087]

[0088] Figure 4 illustrates an example of weight tensor deduplication. In the H-th epoch, the CPU cluster quantizes the weight tensor of each layer and compares it with the weight tensor saved in the first epoch. Then, only the changed elements are stored in the checkpoint storage (see (a) of Figure 4). In the 2H-th epoch, the CPU cluster stores only the changed elements by comparing them with the partial checkpoint saved in the H-th epoch (see (b) of Figure 4). However, in the 2H-th epoch, the checkpoint saved in the H-th epoch is partial, so a complete comparison is not performed. As mentioned earlier, since the weight tensor changes with each epoch, it is rare for the same element values ​​to appear across multiple epochs. Since the checkpoint device (CheckBullet) only captures changes in the weight tensor within a short period of time, checkpointing takes less time and does not cause training interruption. Additionally, the checkpoint device (CheckBullet) can significantly reduce the checkpoint size by removing identical weight values ​​between two consecutive weight tensors.

[0089] Below, we describe the merging of partial checkpoints.

[0090] Deduplication of weight tensors can reduce storage requirements. However, increasing the number of layers within the deduplicated weight tensor linearly increases recovery costs, potentially leading to longer failure recovery times. In such cases, recovery costs can be mitigated by merging partial checkpoints saved during training. Therefore, the checkpointing device (CheckBullet) can adopt a partial checkpoint merging strategy that enables fast recovery while preserving the original model training loss.

[0091] For a merge operation, the point at which partial checkpoints are merged must first be determined. Specifically, the merge point can be determined by considering recovery time and storage requirements. R TH is the maximum allowable recovery time after a failure occurs, and can have a value preset by the administrator, etc. R FC represents the time it takes to recover a full checkpoint, and R PC represents the time taken to find the partial checkpoint with the minimum learning loss. Also, R M is the time taken to merge the selected partial checkpoints. The checkpoint device (CheckBullet) creates a partial checkpoint every H epochs. X R Let be the number of saved partial checkpoints. Then, since the time required to restore a checkpoint must be less than or equal to the maximum allowable recovery time, we can obtain a constraint as in Equation 4.

[0092] [Equation 4]

[0093]

[0094] According to mathematical formula 4, X R must satisfy mathematical expression 5.

[0095] [Equation 5]

[0096]

[0097] After the failure, the model was R TH It should be recovered using a checkpoint within X R can be set as in mathematical formula 6.

[0098] [Equation 6]

[0099]

[0100] Likewise, the limited storage capacity of checkpoints must also be considered. Total Let and M be the total capacity of the checkpoint storage and the number of models being trained, respectively. Then, the maximum checkpoint storage capacity of a single model, S TH is given as in mathematical formula 7.

[0101] [Equation 7]

[0102]

[0103] The checkpointing device (CheckBullet) generates a full checkpoint and several partial checkpoints for each model during training. S FC Wow S PC Let X be the storage space required for full checkpoints and partial checkpoints, respectively. S Let be the number of partial checkpoints stored considering the maximum checkpoint storage capacity. The checkpoint device (CheckBullet) calls the full checkpoint to verify the model structure and searches the partial checkpoints to select the checkpoint with the minimum learning loss. Since the sum of the storage space occupied by the checkpoints of a single model must be less than or equal to the given maximum checkpoint storage capacity, we can obtain Equation 8.

[0104] [Equation 8]

[0105]

[0106] By mathematical formula 8, X S must satisfy mathematical expression 9.

[0107] [Equation 9]

[0108]

[0109] To preserve checkpoint storage capacity, X S can be determined as in mathematical formula 10.

[0110] [Equation 10]

[0111]

[0112] If either of the conditions for maximum allowable recovery time and maximum checkpoint storage capacity is not met, model recovery may be affected. Considering both Equations 6 and 10, the merge point X can be expressed as Equation 11.

[0113] [Equation 11]

[0114]

[0115] As a result, the checkpoint device (CheckBullet) merges X partial checkpoints into one weight tensor (i.e., merged checkpoint), taking into account both recovery time and storage requirements.

[0116] The checkpoint device (CheckBullet) must merge two consecutive partial checkpoints among the checkpoints saved at merge point X. However, during model training, the most recently saved checkpoint does not always have the minimum training loss. The temporary increase in training loss during model training can be explained by the following reasons. During training, the model adjusts the learning rate to converge to the optimal solution, which may result in a temporary increase in the training loss (i.e., learning rate adjustment). In addition, the training loss may increase if the model gets stuck in a local minimum within the loss landscape while searching for the global minimum (i.e., local minimum). Therefore, an approach is needed to select the optimal checkpoint, rather than the most recent checkpoint. Partial checkpoints should be merged based on the minimum training loss.

[0117] [Algorithm 2]

[0118]

[0119] In the checkpoint device (CheckBullet), the merging technique of partial checkpoints is utilized to select the minimum learning loss. In Algorithm 2, when X partial checkpoints are collected, the checkpoint device (CheckBullet) selects the index k corresponding to the minimum learning loss from the learning loss list A, thereby selecting k best Defines (see lines 3-5). k best If k is 1, the epoch r with the oldest checkpoint becomes the full checkpoint or merged checkpoint (see lines 6-7). Otherwise, k best Since is a partial checkpoint, the checkpoint device (CheckBullet) merges the rth and (r-1)th partial checkpoints (see lines 8 to 12). The merged checkpoint is a complete weight tensor. are stored in the storage. The checkpoint device (CheckBullet) also removes existing partial checkpoints and the training loss list A (see lines 13-14). This process is repeated until model training is complete. Consequently, merging partial checkpoints enables rapid recovery after training failure, as checkpoints are merged based on the minimum training loss.

[0120] Performance Evaluation

[0121] The proposed technique is implemented using TensorFlow and Keras libraries. Each model was trained on an Intel Core i7-10700 with 128 GB of CPU memory and an NVIDIA GEFORCE RTX 2070 with 8 GB of GPU memory. The experiments use CUDA 9.0 and CUDNN 7.0 libraries. In all experiments, each model was trained to the target accuracy with a specific number of samples. The proposed technique trains with fixed-time epochs and adopts an LSTM model, which is suitable for long-term relationship training of time-series data. The loss function is the Mean Squared Error (MSE), and Adam is used as the optimizer. The proposed technique uses 8-bit quantization as a metric for future experiments.

[0122] The performance of the proposed technique is evaluated in terms of 1) training loss, 2) training execution time, 3) storage requirements during training, and 4) recovery time after training failure. For performance comparison, the following techniques are considered: 1) Full ckpt: Full checkpointing is performed including both the model structure and weight tensors, 2) Weight ckpt: A checkpointing method that includes only the weight tensors, 3) Q-weight ckpt: A checkpointing method that includes only the quantized weight tensors, and 4) Old ckpt: A method that saves a checkpoint long before a training failure occurs. If H = 10, Old ckpt performs checkpointing at the 7th and 17th epochs, which are checkpointing points that are 30% shorter than H. Therefore, Old ckpt has a different checkpointing interval from the proposed technique. Weight ckpt and Q-weight ckpt have the same checkpointing interval as the proposed technique, but differ in how they store checkpoints.

[0123] We also consider three datasets obtained from mobile networks, as shown in Table 1. ICARUS (D. Roy, V. Chaudhury, C. Tassie, C. Spooner, and K. Chowdhury, "ICARUS: Learning on IQ and Cycle Frequencies for Detecting Anomalous RF Underlay Signals," in Proc. IEEE International Conference on Computer Communications (INFOCOM), pp. 1-10, May 2023.) is a dataset collected from the NSF POWDER testbed, which contains data on LTE signals and Direct Sequence Spread Spectrum (DSSS) signals. Power_dlul (JA Ayala-Romero, A. Garcia-Saavedra, X. Costa-Perez, and G. Iosifidis, "Bayesian Online Learning for Energy-Aware Resource Orchestration in Virtualized RANs," in Proc. IEEE Conference on Computer Communications (INFOCOM), pp. 1-10, May 2021.) is a dataset designed for performance evaluation of virtualized base stations (vBSs), and includes information on vBS status such as bandwidth of LTE interface, uplink / downlink traffic load, etc. Monroe (AS Khatouni, M. Trevisan, D. Giordano, M. Rajiullah, S. Alfredsson, A. Brunstrom, C. Midoglu, and O. Alay, "An Open Dataset of Operational Mobile Networks," in Proc.ACM Symposium on Mobility Management and Wireless Access (MobiWac), pp. 83-90, Nov. 2020.) is a mobile network dataset collected from 27 mobile network operators and contains general contextual information about node status (e.g., cell ID, signal strength, frequency).

[0124] DatasetSamplesModel size (M)Batch sizeParamsICARUS1,2500.147648,821Power_dlul32,7414.5128380,881Monroe2,682,9821112569,623,201

[0125] To model realistic failure events, the proposed technique uses the Weibull distribution to establish failure rates. Unlike the exponential distribution, the Weibull distribution exhibits constant, increasing, or decreasing failure rates over time, depending on the scale parameter α > 0 and the shape parameter β > 0. Specifically, when β = 1, the Weibull distribution is identical to the exponential distribution, and thus the failure rate is constant. On the other hand, when β > 1, the failure rate increases over time, and when β < 1, the failure rate decreases.

[0126] Training Loss

[0127] Figure 5 shows the average L2 loss according to the quantization level, showing that the average L2 loss decreases as model training progresses. The initial learning rate is set to 0.01, and the LSTM model is trained until convergence. The L2 loss is measured based on checkpoints generated during model training. The results are calculated as the average L2 loss from 20 iterations of the same model training. Since all three datasets produced similar results, the Power_dlul dataset was used as a representative dataset. Using this dataset, a model was developed to predict the power consumption of the vBS.

[0128] Compared to the unquantized weight tensor (i.e., the baseline), the average L2 loss overhead increases by 290% and 46% for 2-bit and 4-bit quantization, reaching 0.03 and 0.003, respectively. However, the increases are only 0.0007 and 0.0001 for 6-bit and 8-bit quantization, remaining within 7% and 1%, respectively. In particular, the 8-bit quantization results in a training loss increase of less than 1% compared to the baseline. Therefore, we can confirm that the training loss occurs within 1% even when quantization is used in the proposed technique.

[0129] Figure 6 shows the average L2 loss during model retraining using checkpoints stored in remote storage after training failure. Given that the three datasets mentioned above yielded similar results in Figure 6, only the retraining results using the Power_dlul dataset are presented as a representative example. "No failure" represents a model training scenario without training failure and serves as the baseline in Figure 6. The proposed technique retains the checkpoint with the minimum training loss among previously stored checkpoints. Therefore, when retraining begins after a training failure, the average L2 loss is equal to the minimum training loss. Q-weight ckpt reflects the weight tensor stored immediately before the training failure during the retraining process. In other words, it always retains the most recently stored checkpoint, so it tends to have a slightly higher average L2 loss than the proposed technique. On the other hand, "Old ckpt" tends to have a higher average L2 loss because it uses a checkpoint stored long before the training failure for retraining.

[0130] Figure 7 illustrates the average L2 loss overhead according to the number of failures. The L2 loss overhead represents the percentage increase in L2 loss compared to the L2 loss when model training is performed without training failures. Typically, retraining is performed immediately after a failure occurs, so frequent failures inevitably lead to an increase in L2 loss. Consequently, as the number of failures increases, Old ckpt increases by up to 200% of the average L2 loss overhead. Q-weight ckpt experiences increased retraining loss due to the influence of previous failures due to multiple failures, with a loss overhead ranging from 15% to 47%. The proposed technique considers both the mean-time-between-failure (MTBF) and the minimum training loss to save optimal checkpoints. Therefore, retraining using checkpoints obtained by the proposed technique maintains a stable training loss regardless of the number of failures.

[0131] Table 2 shows the average L2 loss overhead according to β. When β is 0.5, failures occur more frequently than expected (i.e., early in training). Based on the rapid loss decay during initial model training, the most recent checkpoint is likely to have the minimum training loss. Therefore, at the 100th epoch, the loss overheads of the proposed method and Q-weight ckpt are similar, with values ​​of 11.23% and 12.42%, respectively, and the loss overhead of the proposed method decreases over time. Since Q-weight ckpt and Old ckpt are retrained using the most recently saved checkpoint at the 100th epoch, their loss overheads are relatively high, at 4.35% and 22.84%, respectively. When β is 1.0, the proposed method has a maximum loss overhead of 0.46% because it saves the optimal checkpoint by considering both MTBF and minimum training loss. In contrast, Q-weight ckpt and Old ckpt have relatively high loss overheads, averaging 8% and 19%, respectively. When β is 2.0, the failure rate increases after sufficient model training. Therefore, the overall loss overhead increases slightly at the 300th epoch, but the proposed technique only increases it by 0.47%. Because the loss changes at each epoch, Q-weight ckpt and Old ckpt, which reflect the most recent checkpoint during retraining, show somewhat higher loss overheads, 6.23% and 15.15%, respectively.

[0132] Epock 30Epoch 100Epoch 300β=0.51.02.00.51.02.00.51.02.0Mean L2loss overhead (%)CheckBullet11.230.460.540.240.180.370.140.120.47Q-weight ckpt12.4212.22.144.356.191.963.64.956.23Old ckpt48.3528.89.7422.8416.447.1812.712.4615.15

[0133] Training Runtime

[0134] Figure 8(a) shows the average runtime overhead. The runtime overhead represents the additional time incurred compared to the runtime of pure model training without checkpointing. The proposed technique performs both snapshotting and checkpointing on the CPU cluster. In contrast, GPU checkpointing performs both snapshotting and checkpointing on the GPU cluster, while GPU snapshot performs snapshotting on the GPU cluster and checkpointing on the CPU cluster.

[0135] During the early epochs, full checkpoints (i.e., 0.03 in Fig. 8(a)) and partial checkpoints (i.e., 0.15 in Fig. 8(a)) are performed. On the other hand, after the intermediate epochs (i.e., after 0.3 in Fig. 8(a)), a merge for the partial checkpointing is performed, which increases the runtime overhead. GPU checkpointing increases the runtime overhead by up to 63% as training progresses, and GPU snapshot increases the runtime overhead by 3.4% due to the time required to copy the model to the GPU cluster. As shown in Fig. 8(b), CheckBullet induces the minimum runtime overhead of 0.2% during model training, because it only includes the time when the GPU cluster calls the CPU cluster.

[0136] Storage Requirements during training

[0137] Table 3 shows the normalized checkpoint sizes compared to Full ckpt. For Weight ckpt, the reduction in checkpoint size compared to Full ckpt is intuitive. However, the proposed technique, due to its quantization technique to remove floating-point errors, shows a significant reduction in checkpoint size, on average 3.5x and 2.9x, respectively, compared to Full ckpt and Weight ckpt. Furthermore, the proposed technique, by employing a technique to remove redundant weight tensors, shows an average 30% reduction in checkpoint size compared to Q-weight ckpt. Since CheckBullet stores only weight tensors to reduce checkpoint size, it reduces checkpoint size regardless of the dataset or model structure (e.g., layers and parameters).

[0138] CheckpointingmethodNormalized checkpoint sizeICARUS Power_dlul MonroeFull ckpt (Baseline)111Weight ckpt0.780.910.85Q-weight ckpt0.350.350.42CheckBullet0.250.250.30

[0139] Recovery Time after Training Failure

[0140] While it is important to have a stable lightweight checkpoint before a failure, follow-up management using checkpoints after a failure is also important for rapid recovery. Figure 9 shows the total recovery time according to the checkpointing method used by Power_dlul. The total recovery time is defined as the sum of the initialization time and the training recovery time. The initialization time is the time from the occurrence of a training failure to the start of retraining, and the training recovery time is the time it takes to reach the training loss just before the training failure during retraining. The proposed technique requires initialization time because it includes a dequantization process during initial recovery, but it saves the checkpoint with the minimum training loss, eliminating the training recovery time for additional recovery. Because Q-weight ckpt saves quantized partial checkpoints, it incurs additional overhead in terms of the training recovery time required to catch up with the previous training loss, as well as the initialization time for merging checkpoints and dequantization. In the case of old ckpt, because there is no checkpoint saved immediately before the training failure, repeated training is required to catch up with the previous training loss, requiring more time for retraining. Therefore, as can be seen in Figure 9, Q-weight ckpt and Old ckpt require an additional 16 and 48 seconds, respectively, compared to the proposed technique.

[0141] Figure 10 illustrates the variation in cumulative recovery time according to the number of failures. As shown in Figure 9, the proposed technique only requires initial recovery time, so even in the case of frequent failures, the overall recovery time does not increase significantly. However, for Q-weight ckpt and Old ckpt, a significant amount of time is spent on training recovery time, along with the initial time. It is clear that the recovery time increases exponentially as the failure frequency increases.

[0142] Figure 11 shows the total recovery time according to β. The total recovery time is calculated as the average recovery time after performing the same model training 10 times. When β is 0.5, the proposed technique fails earlier than expected, but because it uses a checkpoint with the minimum training loss, the total recovery time is reduced by approximately 15 seconds compared to Q-weight ckpt. Q-weight ckpt takes additional time to reach the original training loss. When β is 1.0, the total recovery time is relatively low because both the proposed technique and Q-weight ckpt use checkpoints saved immediately before failure. When β is 1.0, the proposed technique saves approximately 35 seconds compared to Q-weight ckpt. Overall, because the proposed technique saves checkpoints with the minimum training loss, varying β does not significantly affect the total recovery time. Old ckpt, because it uses checkpoints saved long ago, has a high average total recovery time of 364 seconds regardless of β.

[0143] The devices described above may be implemented as hardware components, software components, and / or a collection of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor (DSP), a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0144] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or command the processing device, independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave to be interpreted by the processing device or to provide instructions or data to the processing device. Software may be distributed on network-connected computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0145] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known to and usable by those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0146] Although the present invention has been described with reference to the embodiments illustrated in the drawings, these are merely exemplary, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible from the described techniques. For example, appropriate results can be achieved even if the described techniques are performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different form than the described method, or are replaced or substituted with other components or equivalents. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

Claims

1. In a checkpoint method performed by a checkpoint device, A step of performing training on a learning model; When the first epoch is completed during learning for the above learning model, a step of performing a checkpointing operation for the above learning model to generate a full checkpoint; At each checkpointing interval, performing a checkpointing operation, generating a partial checkpoint including a weight tensor that includes only elements having different values from the elements of the weight tensor generated as a result of a previous checkpointing operation; and A checkpoint method comprising the step of generating a new full checkpoint by merging the generated partial checkpoints when an epoch corresponding to a merge point during learning is completed.

2. In paragraph 1, The checkpoint device comprises a controller, a GPU cluster including a plurality of workers, a CPU cluster including a plurality of workers, and a storage device. The step of performing the above learning is performed by the GPU cluster, The step of generating the above full checkpoint, the step of generating the above partial checkpoint, and the step of generating the above new full checkpoint are performed by the CPU cluster. Checkpoint method.

3. In paragraph 2, Upon completion of said first epoch, the controller further comprises a step of calculating said checkpointing interval and transmitting said checkpointing interval to said GPU cluster in response to a request from said GPU cluster. Checkpoint method.

4. In paragraph 3, The CPU cluster performs the steps of generating the full checkpoint, the step of generating the partial checkpoint, and the step of generating the new full checkpoint in response to a call from the GPU cluster, and stores the full checkpoint, the partial checkpoint, and the new full checkpoint in the storage device. Checkpoint method.

5. In paragraph 4, If a learning failure occurs during learning, re-training is performed based on the weight tensor included in the new checkpoint. Checkpoint method.

6. In paragraph 5, If a learning failure occurs during learning, The above controller sends a retraining command to the GPU cluster, The CPU cluster responds to a call from the GPU cluster and transmits the weight tensor for retraining to the GPU cluster. Checkpoint method.

7. In paragraph 6, The weight tensor stored in the above storage device is a weight tensor in which each element is quantized by the CPU cluster, The weight tensor for the retraining transmitted to the GPU cluster is a weight tensor in which each element is semi-quantized by the CPU cluster. Checkpoint method.

8. In paragraph 7, The above controller Calculate the above checkpointing interval, The above T is the total learning time required for learning the above learning model, The above E is the total number of epochs required for training the above learning model, T above F is the MTBF (Mean Time Between Failures) of the above learning model. Checkpoint method.

9. In paragraph 8, T above F is calculated by mathematical formula 1, The above mathematical expression 1 is And, B above FC is the time taken to perform the entire checkpoint operation, The above I is the time taken to perform one epoch for the above learning model. Checkpoint method.

10. In paragraph 9, The above merging point (X) is defined by mathematical expression 2, The above mathematical expression 2 is And, Above X R is the number of partial checkpoints stored in the above storage device, Above X S is the number of partial checkpoints that can be stored in the storage space allocated to the above learning model. Checkpoint method.

11. In paragraph 10, Above X R is defined by mathematical formula 3, The above mathematical formula 3 is And, Above R TH has a predefined value as the maximum allowable recovery time after a learning failure occurs, Above R FC is the time taken to restore the entire checkpoint above, Above R M is the time taken to merge the selected partial checkpoints, Above R PC is the time taken to find a partial checkpoint with the minimum learning loss. Checkpoint method.

12. In paragraph 11, X S is defined by mathematical formula 4, The above mathematical expression 4 is And, S above TH is the maximum checkpoint storage capacity of the above learning model, S above FC is the storage space required to store the entire checkpoint above, S above PC is the storage space required to store the above partial checkpoints. Checkpoint method.

13. In paragraph 13, The above new full checkpoint is generated based on the partial checkpoint with the minimum learning loss. Checkpoint method.

Images