Checkpoint management method and device, storage medium and product
By managing intervals based on the temporal attribute information of checkpoints during deep learning model training, using a rapidly growing sequence to divide intervals and selecting and retaining key checkpoints, the problem of linear growth in checkpoint storage is solved, achieving sublinear growth and improved stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- VIPSHOP (GUANGZHOU) SOFTWARE CO LTD
- Filing Date
- 2026-02-10
- Publication Date
- 2026-05-26
Smart Images
Figure CN122086327A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of storage resource management technology, and in particular to a checkpoint management method, device, storage medium and product. Background Technology
[0002] In long-term, continuous training scenarios for deep learning models, checkpoints need to be generated periodically to save the training state. To balance storage costs and data availability, automated management of historical checkpoints is required to adapt to the storage resource demands of long-term, large-scale training scenarios. Currently, common checkpoint management strategies mainly involve retaining the latest checkpoint or retaining checkpoints at fixed time intervals.
[0003] However, retaining the latest checkpoints or retaining checkpoints at fixed intervals both lead to a linear increase in the system's storage space requirements over training time. For long-running training tasks, the physical capacity of the storage system becomes a significant constraint on the model's training results. Furthermore, the increasing number of checkpoints forces the system to rely on periodic external manual intervention to adjust management strategies or clean up data. This manual intervention not only increases the operational burden but is also prone to errors such as accidental deletion of critical checkpoints or corruption of training states, significantly reducing the stability and reliability of checkpoint management.
[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention
[0005] The main purpose of this application is to provide a checkpoint management method, device, storage medium and product, which aims to solve the technical problem that the stability and reliability of checkpoint management are greatly reduced due to the need for periodic manual intervention caused by the linear growth of checkpoint storage.
[0006] To achieve the above objectives, this application proposes a checkpoint management method, which includes: Obtain checkpoints generated during model training, wherein the checkpoints are used to record training data during model training; Based on the time attribute information of the checkpoints, the checkpoints are assigned to the corresponding checkpoint retention intervals, wherein the time span of the checkpoint retention interval of any later stage is greater than the time span of the checkpoint retention interval of the immediately preceding stage. After the checkpoint retention interval is filled with checkpoints, at least one checkpoint is selected from the checkpoints in the checkpoint retention interval for retention processing, and the unselected checkpoints in the checkpoint retention interval are deleted.
[0007] In one embodiment, the step of assigning the checkpoint to the corresponding checkpoint retention interval based on the checkpoint's time attribute information includes: Generate a rapidly growing sequence based on initial seed parameters and a preset sequence length; The checkpoint retention interval is obtained by dividing the time dimension range of the checkpoint using the rapidly growing sequence. Based on the time attribute information of the checkpoints, an interval matching mapping relationship is constructed, and the checkpoints are assigned to the corresponding checkpoint retention intervals through the interval matching mapping relationship.
[0008] In one embodiment, the step of dividing the time dimension range of the checkpoint using the rapidly growing sequence to obtain the checkpoint retention interval includes: Verify the fit between the numerical range of the rapidly growing sequence and the time dimension range, and adjust the numerical range based on the fit between the numerical range and the time dimension range so that the rapidly growing sequence fits the time dimension range. The checkpoint retention interval is obtained by taking two consecutive adjacent terms in the adapted fast-growing sequence as the start and end time boundaries, respectively, so that the time span of the checkpoint retention interval matches the numerical span of two adjacent terms in the fast-growing sequence.
[0009] In one embodiment, the step of verifying the fit between the numerical range of the rapidly growing sequence and the time dimension range, and adjusting the numerical range based on the fit between the numerical range and the time dimension range, includes: Determine whether the numerical range can cover the time dimension range; If the numerical range cannot cover the time dimension range, a new term is generated according to the recursive mechanism of the rapidly growing sequence. If the numerical range covers the time dimension range, the corresponding items are selected from the rapidly growing sequence for partitioning based on the start and end times of the time dimension range.
[0010] In one embodiment, the step of constructing an interval matching mapping relationship based on the time attribute information of the checkpoints, and assigning the checkpoints to corresponding checkpoint retention intervals through the interval matching mapping relationship includes: The time attribute information is standardized to convert it into initial time data; The initial time data is verified to obtain the target time data; Based on the target time data, an interval matching mapping relationship is constructed, and the checkpoint is matched to the corresponding checkpoint retention interval through the interval matching mapping relationship.
[0011] In one embodiment, the step of selecting at least one checkpoint from the checkpoint retention interval for retention processing and deleting the unselected checkpoints in the checkpoint retention interval includes: Extract the time attribute association information of checkpoints in the checkpoint retention interval, and create checkpoint filtering conditions based on the time attribute association information; The checkpoints in the checkpoint retention interval are prioritized according to the checkpoint filtering criteria. At least one checkpoint is selected to be retained based on the sorting result, and the checkpoints that are not selected in the checkpoint retention interval are determined as checkpoints to be deleted. Determine the invocation status of the checkpoint to be deleted. If the invocation status of the checkpoint to be deleted is not invoked, delete the checkpoint directly.
[0012] In one embodiment, after determining the invocation status of the checkpoint to be deleted, and deleting the checkpoint directly when the invocation status is not invoked, the method further includes: When the call status of the checkpoint to be deleted is "called", record the calling task associated with the checkpoint to be deleted and monitor the execution status of the calling task in real time. If the execution status of the invoked task is "execution completed", then the checkpoint to be deleted is deleted.
[0013] In addition, to achieve the above objectives, this application also proposes an electronic device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the checkpoint management method as described above.
[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the checkpoint management method described above.
[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the checkpoint management method described above.
[0016] One or more technical solutions proposed in this application have at least the following technical effects: The method involves acquiring checkpoints generated during model training, whereby these checkpoints record training data. Based on the temporal attribute information of each checkpoint, the checkpoints are assigned to corresponding checkpoint retention intervals. The time span of any subsequent checkpoint retention interval is greater than the time span of its immediately preceding checkpoint retention interval. After a checkpoint retention interval is filled with checkpoints, at least one checkpoint is selected from the checkpoints within that interval for retention, and any unselected checkpoints are deleted. In this embodiment, checkpoints are assigned to checkpoint retention intervals based on their temporal attribute information. Each interval retains only key checkpoints, ensuring that large-span intervals do not lose critical nodes while significantly reducing checkpoint storage requirements. Combined with the characteristic that the time span of each checkpoint retention interval is greater than that of the preceding checkpoint retention interval, the number of checkpoints retained per unit time in the early stages of model training meets the high-frequency requirements of model training, while the number of checkpoints retained per unit time in the later stages of model training is significantly reduced, thus decreasing checkpoint storage requirements. At least one checkpoint is selected from the checkpoints in the checkpoint retention interval for retention, and unselected checkpoints in the interval are deleted. Within each interval, a small number of critical checkpoints are retained, while redundant checkpoints are removed. This process gradually reduces the number of checkpoints retained per unit time as training progresses, achieving a sublinear increase in checkpoint storage over time. This eliminates the need for periodic manual intervention to adjust the management strategy, significantly improving the stability and reliability of checkpoint management. Attached Figure Description
[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart illustrating an embodiment of the checkpoint management method of this application. Figure 2 This is a schematic diagram of the checkpoint allocation process provided in Embodiment 1 of the checkpoint management method of this application; Figure 3 A flowchart illustrating the process of dividing checkpoint retention intervals, provided in Embodiment 1 of the checkpoint management method of this application; Figure 4A simplified flowchart illustrating the checkpoint management method provided in Embodiment 2 of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the checkpoint management method in this application embodiment.
[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.
[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.
[0023] The main solution of this application embodiment is: to obtain checkpoints generated during model training, wherein the checkpoints are used to record training data during model training; Based on the time attribute information of checkpoints, checkpoints are assigned to corresponding checkpoint retention intervals. The time span of the checkpoint retention interval of any later stage is greater than the time span of the checkpoint retention interval of its immediate preceding stage. After the checkpoint retention interval is filled with checkpoints, at least one checkpoint is selected from the checkpoints in the checkpoint retention interval for retention, and the unselected checkpoints in the checkpoint retention interval are deleted.
[0024] In long-term, continuous training scenarios for deep learning models, checkpoints need to be generated periodically to save the training state. To balance storage costs and data availability, automated management of historical checkpoints is required to adapt to the storage resource demands of long-term, large-scale training scenarios. Currently, common checkpoint management strategies mainly involve retaining the latest checkpoint or retaining checkpoints at fixed time intervals.
[0025] However, retaining the latest checkpoints or retaining checkpoints at fixed intervals both lead to a linear increase in the system's storage space requirements over training time. For long-running training tasks, the physical capacity of the storage system becomes a significant constraint on the model's training results. Furthermore, the increasing number of checkpoints forces the system to rely on periodic external manual intervention to adjust management strategies or clean up data. This manual intervention not only increases the operational burden but is also prone to errors such as accidental deletion of critical checkpoints or corruption of training states, significantly reducing the stability and reliability of checkpoint management.
[0026] This application provides a solution for acquiring checkpoints generated during model training. These checkpoints record training data during the training process. Based on the temporal attribute information of the checkpoints, they are assigned to corresponding checkpoint retention intervals. The time span of any subsequent checkpoint retention interval is greater than the time span of its immediately preceding checkpoint retention interval. After a checkpoint retention interval is filled, at least one checkpoint is selected from the checkpoints in that interval for retention, and unselected checkpoints are deleted. In this embodiment, checkpoints are assigned to retention intervals based on their temporal attribute information. Each retention interval retains only key checkpoints, ensuring that large-span intervals do not lose critical nodes while significantly reducing checkpoint storage requirements. Combined with the characteristic that the time span of each retention interval is greater than that of the preceding interval, the number of checkpoints retained per unit time in the early stages of model training can meet the high-frequency requirements of model training, while the number of checkpoints retained per unit time in the later stages of model training decreases significantly, slowing down the growth rate of checkpoint storage requirements. At least one checkpoint is selected from the checkpoints in the checkpoint retention interval for retention, and unselected checkpoints in the retention interval are deleted. Within each retention interval, a small number of critical checkpoints are retained, while redundant checkpoints are deleted. This process gradually reduces the number of checkpoints retained per unit time as training progresses, achieving a sublinear increase in checkpoint storage over time. This eliminates the need for periodic manual intervention to adjust the management strategy, significantly improving the stability and reliability of checkpoint management.
[0027] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a computer server and a personal computer, or an electronic device capable of performing the above functions.
[0028] Based on this, embodiments of this application provide a checkpoint management method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the checkpoint management method of this application.
[0029] In this embodiment, the checkpoint management method includes steps S10 to S30: Step S10: Obtain checkpoints generated during model training, wherein checkpoints are used to record training data during model training. It should be noted that in deep learning model training scenarios, checkpoints are snapshots of the training state generated during model training. The training data recorded in checkpoints includes learnable parameters of each layer of the model, the current state of the optimizer, training progress information, and performance evaluation data. As the deep learning model training iterates and optimizes, checkpoints are continuously generated over time. Existing technologies typically employ uniform or full retention modes, resulting in a linear increase in checkpoint storage over training time, consuming a large amount of storage resources in the long term, and requiring periodic manual intervention to clean up redundant data. This application obtains checkpoints during the model training process and divides the checkpoint retention intervals based on the time attribute information of the checkpoints, where the time span of the later stage is longer than that of the previous stage. Checkpoints are then assigned to the corresponding intervals, retaining key checkpoints and deleting redundant data within each interval. This reduces the storage requirements for checkpoints and significantly improves the stability and reliability of checkpoint management. Checkpoints are obtained by real-time monitoring of the checkpoint generation directory of the model training framework, receiving generation notifications actively pushed by the framework, or periodically querying the storage path of checkpoints.
[0030] Understandably, the acquired checkpoints are not only the core data carriers for fault recovery and training backtracking during model training, but also the basis for subsequent selection of key checkpoints, such as selecting key checkpoints based on performance evaluation data.
[0031] Step S20: Based on the time attribute information of the checkpoints, the checkpoints are assigned to the corresponding checkpoint retention intervals, wherein the time span of the checkpoint retention interval of any later stage is greater than the time span of the checkpoint retention interval of the immediately preceding stage. It should be noted that the time attribute information of a checkpoint refers to the core information that uniquely identifies the timing of the checkpoint's generation, including the timestamp, the standard date and time string, and the timestamp corresponding to the training iteration round. A single checkpoint can only match one checkpoint retention interval. The time span refers to the difference between the end time and the start time of a single interval.
[0032] Understandably, categorizing scattered checkpoints into corresponding intervals according to their generation time provides an orderly management foundation for subsequent interval-based filtering, retention, and deletion of redundant data. Through standardized units and anomaly handling, the accuracy of the allocation results is ensured, avoiding issues such as the omission of recent checkpoints or the over-retention of distant checkpoints due to incorrect attribution. Time-series categorization enables refined management of checkpoints.
[0033] In one feasible implementation, refer to Figure 2 , Figure 2 This is a schematic diagram of the checkpoint allocation process provided in Embodiment 1 of the checkpoint management method of this application. The steps of allocating checkpoints to their corresponding checkpoint retention intervals based on their time attribute information include steps S21-S23: Step S21: Generate a rapidly growing sequence based on the initial seed parameters and the preset sequence length; Step S22: Divide the time dimension range of the checkpoints by a rapidly growing sequence to obtain the checkpoint retention interval; Step S23: Construct an interval matching mapping relationship based on the time attribute information of the checkpoints, and assign the checkpoints to the corresponding checkpoint retention intervals through the interval matching mapping relationship.
[0034] It should be noted that a rapidly growing sequence refers to a sequence in which the value of subsequent terms increases at a rate higher than that of a linearly growing sequence. Its inherent characteristic is that the numerical span between adjacent terms increases with each term. The initial seed parameters are the first two base values of the rapidly growing sequence, which are non-negative integers and not both 0. The preset sequence length is the number of terms to be generated in the rapidly growing sequence, and this preset length can be customized according to the model training cycle. The generated rapidly growing sequence is not used directly for calculation, but rather for establishing checkpoint retention intervals in the time dimension. Each value in the rapidly growing sequence is used as a time length value in practical applications, and its unit is consistent with the unit of the time attribute information of the checkpoint, such as days and hours. The unit of the time attribute information must be consistent with the unit of the rapidly growing sequence and the time dimension range of the checkpoint, such as both being hours or days. If there is a unit difference, unit conversion must be performed first, such as converting minutes to hours, before performing the allocation. For abnormal checkpoints with missing time attribute information, incorrect format, or exceeding the time dimension range, they can be directly marked as data to be cleaned and will not participate in interval allocation to avoid affecting the accuracy of the overall allocation logic.
[0035] Understandably, due to the recursive nature of the generalized Fibonacci sequence, which can form interval distributions with small recent time spans and large distant time spans, no additional complex algorithm adjustments are needed. Therefore, the generalized Fibonacci sequence is chosen as the preferred solution in this embodiment. By generating the sequence and dividing the time dimension range according to the generated sequence, the sequence characteristics can be directly transformed into a checkpoint management strategy, ensuring the effectiveness of interval division and avoiding the linear growth problem of storage volume caused by uniform division. By rapidly growing the sequence to determine the checkpoint retention interval, the difference between adjacent terms in the checkpoint retention interval will increase rapidly as the term order increases. When this difference is mapped to the time span, a distribution pattern of dense recent time intervals and sparse distant time intervals is formed. In practical application scenarios, in order to avoid model anomalies and need to retain all checkpoints within a preset number of days to roll back the model, the first term of the initial seed parameter can be set according to the number of days to be retained. For example, if checkpoints within 7 days to be retained are needed to complete the model rollback, the first term in the initial seed parameter can be set to 7. The checkpoint retention intervals divided based on the rapidly growing sequence generated from this initial seed parameter will not assign checkpoints generated within the last 7 days.
[0036] Preferably, the generalized Fibonacci sequence is selected as the rapidly growing sequence, and the initial seed parameter is set to [1, 2], with a preset sequence length L of 8. The sequence S generated according to the recursive relationship of the generalized Fibonacci sequence is [1, 2, 3, 5, 8, 13, 21, 34]. Each value of the generated sequence S is interpreted as a time length in days. Taking a common reference time point as the origin T, adjacent terms of the sequence S are mapped sequentially onto the time axis, thus dividing L-1 checkpoint retention intervals. That is, the time boundary for generating the first checkpoint retention interval is (T-1 days, T-2 days), and this first checkpoint retention interval covers the checkpoint for the entire day from the second day. The time boundaries for the second checkpoint retention interval are (T-2 days, T-3 days), the third checkpoint retention interval is (T-3 days, T-5 days), the fourth checkpoint retention interval is (T-5 days, T-8 days), the fifth checkpoint retention interval is (T-8 days, T-13 days), the sixth checkpoint retention interval is (T-13 days, T-21 days), and the seventh checkpoint retention interval is (T-21 days, T-34 days). The time span of the checkpoint retention interval is determined by the numerical difference between two adjacent terms in the sequence, and is 1 day, 1 day, 2 days, 3 days, 5 days, 8 days, and 13 days respectively. The interval span increases significantly with the interval number, reflecting a distribution pattern of denser intervals closer to the end and sparser intervals further away. The preset sequence length L can be set according to the total duration of the model training plan, such as planning to retain key checkpoints within approximately 100 days, so that the end boundary of the last interval can cover the time range of the model training cycle.
[0037] In one feasible implementation, see Figure 3 , Figure 3 This is a flowchart illustrating the process of dividing the checkpoint retention interval according to Embodiment 1 of this application. The step of dividing the time dimension range of checkpoints using a rapidly growing sequence to obtain the checkpoint retention interval includes steps S221~S222: Step S221: Verify the fit between the numerical range and the time dimension range of the rapidly growing sequence. Adjust the numerical range based on the fit between the numerical range and the time dimension range so that the rapidly growing sequence fits the time dimension range. Step S222: Take two consecutive adjacent terms in the adapted fast-growing sequence as the start and end time boundaries of the checkpoint retention interval, so that the time span of the checkpoint retention interval matches the numerical span of two adjacent terms in the fast-growing sequence, thus obtaining the checkpoint retention interval.
[0038] It should be noted that the numerical range refers to the numerical interval formed by the first and last terms of the sequence. For example, the numerical range of the sequence {2,1,3,4,7} is 2-7. Its unit is completely consistent with the unit of the time dimension range of the checkpoint, both being days, hours, etc., ensuring the effectiveness of the adaptability check. Adaptability refers to whether the numerical range of the sequence can completely cover the time dimension range of the checkpoint, i.e., the smallest term of the sequence is less than or equal to the earliest generation time of the checkpoint, and the largest term of the sequence is greater than or equal to the latest generation time of the checkpoint, avoiding time gaps where the checkpoint has no corresponding interval. When the numerical range of the sequence is less than the time dimension range, a new term is generated according to the sequence recursion rule, i.e., extending the numerical range. When the numerical range of the sequence is greater than the time dimension range, a sub-sequence corresponding to the time dimension range is extracted. Two adjacent consecutive terms serve as the start and end time boundaries, respectively. The preceding term is used as the start time boundary, and the following term as the end time boundary, ensuring the consistency of interval division. Furthermore, the time span of the interval retained by the checkpoint is equal to the numerical difference between the corresponding two adjacent terms. For example, if the sequence terms 3 and 5 are used as boundaries, the interval span is 2, achieving precise span matching.
[0039] Understandably, the adaptation verification and adjustment steps are crucial to avoiding invalid interval divisions. This operation ensures precise alignment between the rapidly growing sequence and the checkpoint time dimension range, resolving the problem of mismatched sequence and time range leading to chaotic divisions. Using adjacent terms of the adapted sequence as interval boundaries ensures continuous connection of all retained intervals while strictly inheriting the characteristic of increasing span between adjacent terms, further reinforcing the distribution pattern of denser recent intervals and sparser long-term intervals.
[0040] For example, the time dimension of the model training checkpoints ranges from 3 hours to 47 hours, with the time unit being hours. Based on a rapidly growing sequence with initial seed parameters F0=2 and F1=1, and a preset sequence length L=8, the generated rapidly growing sequence is {2,1,3,4,7,11,18,29}, whose numerical range is from 1 to 29 hours. This is less than the upper limit of the time dimension range of 47 hours, and the fit check result is unsuitable.
[0041] Based on the adaptation results, the numerical range was adjusted. Following the recursive rule of the rapidly growing sequence, subsequent terms are the sum of the previous two terms, resulting in the generation of a new term 47. The adjusted sequence is {2,1,3,4,7,11,18,29,47}, with the numerical range expanded to 1 to 47 hours, which is fully adapted to the checkpoint time dimension range of 3 to 47 hours.
[0042] Subsequently, checkpoint retention intervals were divided, using adjacent terms of the adapted, rapidly growing sequence as boundaries, and extracting sub-sequences 3 to 47 corresponding to the time dimension range, resulting in retention intervals (3,4], spanning 1 hour; (4,7], spanning 3 hours; (7,11], spanning 4 hours; (11,18], spanning 7 hours; (18,29], spanning 11 hours; and (29,47], spanning 18 hours. All intervals are continuous and non-overlapping, with the time span gradually increasing with the item order, providing a reliable checkpoint retention interval basis for subsequent storage optimization.
[0043] In one feasible implementation, the step of verifying the fit between the numerical range and the time dimension range of a rapidly growing sequence, and adjusting the numerical range based on the fit between the numerical range and the time dimension range, includes: Determine whether the numerical range can cover the time dimension range; When the numerical range cannot cover the time dimension range, new terms are generated based on the recursive mechanism of the rapidly growing sequence; When the numerical range can cover the time dimension range, the corresponding terms are selected from the rapidly growing sequence for partitioning based on the start and end times of the time dimension range.
[0044] It should be noted that the last term S[L-1] of the rapidly growing sequence is calculated, representing the maximum time length it can represent, and compared with the time dimension range of the checkpoint set, i.e., the time difference between the latest checkpoint and the earliest checkpoint. If S[L-1] is greater than or equal to the time dimension range of the checkpoint set, it is determined that it can cover the data; otherwise, it cannot cover the data.
[0045] Understandably, when the numerical range of the rapidly growing sequence cannot cover the time dimension of the checkpoint set, generating new terms from the rapidly growing sequence can adapt to scenarios where the training cycle exceeds the estimate, without needing to reset the initial parameters, thus improving the flexibility of the solution. When it can cover the time dimension, truncating the corresponding terms from the rapidly growing sequence can avoid redundancy in the checkpoint retention interval division caused by the excessively large numerical range of the rapidly growing sequence, ensuring that the checkpoint retention interval is precisely aligned with the time dimension range of the checkpoint. Furthermore, whether generating new terms from the rapidly growing sequence or selecting corresponding terms, the recursive rules of the rapidly growing sequence are strictly followed, ensuring that the core characteristic of increasing span between adjacent terms is retained after adjustment. This provides crucial support for the subsequent near-dense and far-sparse interval distribution and sublinear growth in storage.
[0046] In one feasible implementation, the step of constructing an interval matching mapping relationship based on the time attribute information of checkpoints, and assigning checkpoints to corresponding checkpoint retention intervals through the interval matching mapping relationship includes: The time attribute information is standardized to convert it into initial time data; The initial time data is validated to obtain the target time data; An interval matching mapping relationship is constructed based on the target time data, and checkpoints are matched to the corresponding checkpoint retention intervals through the interval matching mapping relationship.
[0047] It should be noted that standardization is used to unify the format and units of time attribute information. Validation includes validity, completeness, and uniqueness checks. Validity check requires the initial time data to be non-negative and fall within the time dimension range of the checkpoint. Completeness check requires the initial time data to have no missing fields and no format errors. Uniqueness check requires the initial time data to have no duplicate time identifiers. The interval matching mapping relationship is the correspondence between the target time data and the checkpoint reserved interval.
[0048] Understandably, different training frameworks generate checkpoint time information in different formats; for example, some frameworks output timestamps, while others output date strings. Converting this data uniformly to numerical data avoids matching misalignments caused by format incompatibility, laying the foundation for accurate matching later. Filtering invalid and abnormal data ensures that all target time data participating in the matching process is reliable, preventing issues like mismatched checkpoint intervals or lack of interval assignment after abnormal data enters the matching stage. This ensures consistency in the matching logic, improves matching accuracy, and reduces the complexity of subsequent filtering operations.
[0049] For example, the time dimension of the checkpoints ranges from 4 hours to 60 hours. The adapted checkpoint retention intervals are (4,7], (7,11], (11,18], (18,29], (29,47], and (47,76], all in hours. The time attribute information of three checkpoints is obtained. Checkpoint 1 is a timestamp of 1735696800, corresponding to 2025-01-01, 02:00:00. After standardization, it is converted to the distance from the start of the time dimension. The cumulative duration of 4 hours is 5 hours. During verification, 5.0 is a non-negative number within the range of 4-60 hours, with no missing or duplicate values, thus yielding the target time data of 5.0. Checkpoint 2 is the standard date and time string 2025-01-01, 09:30:00. After standardization, it is parsed into a timestamp and then converted to a cumulative duration of 12.5 hours. During verification, 12.5 is within the range of 4-60 hours, thus yielding the target time data of 12.5. Checkpoint 3 is the training iteration round. The corresponding cumulative duration is 35.2 hours. During the verification process, 35.2 falls within the range of 4-60 hours, resulting in the target time data of 35.2. Subsequently, interval matching mapping relationships are constructed: (4,7] maps to 5.0≤x≤7.0, (7,11] maps to 7.0<x≤11.0, (11,18] maps to 11.0<x≤18.0, (18,29] maps to 18.0<x≤29.0, and (29,47] maps to 29.0<x≤47.0. The time data (47,76) is mapped to 47.0 < x ≤ 76.0. Through matching based on this mapping, the target time data 5.0 for checkpoint 1 falls within the interval (4,7], 12.5 for checkpoint 2 falls within the interval (11,18], and 35.2 for checkpoint 3 falls within the interval (29,47]. If checkpoint 4 exists, its time attribute information is 2025 / 01 / 01. After standardization, it cannot be converted into a valid value, the validation fails, it is marked as data to be cleaned, and it does not participate in the matching.
[0050] Step S30: After the checkpoint retention interval is filled with checkpoints, select at least one checkpoint from the checkpoints in the checkpoint retention interval for retention processing, and delete the checkpoints that have not been selected in the checkpoint retention interval.
[0051] It should be noted that when at least one checkpoint has been assigned within a checkpoint retention interval, that interval is considered ready to perform a filtering operation. In practical applications, the triggering conditions for checkpoint retention operations can be customized according to user needs. For example, it can be executed immediately when the first checkpoint is assigned to each retention interval, or after a specific number of checkpoints have been assigned, or it can be executed periodically in batches on checkpoint retention intervals that have been assigned checkpoints. Each retention interval retains at least one checkpoint, which can be customized according to the backtracking needs of the actual training scenario, such as one or two, ensuring that each model training phase has valid checkpoints available for fault recovery or version tracing. Before deleting an unselected checkpoint, its invocation status must be confirmed. Deletion should only be performed on uninvoked checkpoints. If a checkpoint is being invoked by training tasks, backtracking analysis, or other operations, it must be waited for the invoking task to complete before deletion to avoid affecting ongoing business operations. By retaining only at least one checkpoint in each interval and deleting a large number of the rest, the total number of globally retained checkpoints no longer grows linearly with the total training time, but grows more and more slowly, i.e., it exhibits sublinear growth.
[0052] Understandably, the time span of the intervals expands rapidly from near to far (e.g., 1 day, 2 days, 3 days, 5 days, 8 days), and each interval, regardless of its span, retains only a fixed number of checkpoints. This optimizes checkpoint storage by ensuring that in the near term, the time span is short, the intervals are dense, and there are many retained checkpoints; while in the far term, the time span increases dramatically, the intervals become very sparse, and there are very few retained checkpoints. Retaining only a small number of critical checkpoints in each interval avoids redundant data occupying a large amount of storage resources, while ensuring the checkpoint backtracking capability at different time stages. Furthermore, the large time span of the far term intervals, while maintaining the same number of checkpoints as the near term intervals, causes the number of checkpoints retained per unit time to gradually decrease over time, directly driving a sublinear increase in overall storage volume.
[0053] In one feasible implementation, the steps of selecting at least one checkpoint from the checkpoint retention interval for retention processing and deleting unselected checkpoints from the checkpoint retention interval include: Extract the time attribute association information of checkpoints in the checkpoint retention interval, and create checkpoint filtering conditions based on the time attribute association information; Based on the checkpoint selection criteria, the checkpoints in the checkpoint retention interval are prioritized and sorted. Based on the sorting results, at least one checkpoint is selected to be retained, and the checkpoints in the checkpoint retention interval that are not selected are identified as checkpoints to be deleted. Determine the invocation status of the checkpoint to be deleted. If the invocation status of the checkpoint to be deleted is not invoked, delete the checkpoint directly.
[0054] It should be noted that the time-related information includes the checkpoint's generation timestamp, cumulative duration since the current time, training iterations, associated training task identifier, and model performance metrics, providing a quantitative basis for the selection criteria. Checkpoint selection criteria are based on temporal characteristics, such as retaining the most recently generated checkpoint within the interval or retaining the checkpoint closest to the current time. Priority sorting ranks checkpoints from highest to lowest according to their compliance with the selection criteria, and sorts them from newest to oldest by generation time; the sorting result is unique and traceable. Checkpoints to be deleted refer to those not included in the retention limit in the sorting results. The call status is determined by querying system processes and training task status identifiers, specifically including two states: not called and called. Deletion is only performed when the status is clearly not called, avoiding impact on ongoing business operations.
[0055] For example, the time dimension of the checkpoints ranges from 4 hours to 60 hours, and the adapted retention intervals are (4,7], (7,11], (11,18], (18,29], (29,47], and (47,76]. One checkpoint is retained for each interval. The filtering condition is to prioritize retaining the checkpoint with the most recent generation time within the interval; if the generation times are the same, the checkpoint with the highest validation set accuracy is retained. Taking the (11,18] interval as an example, this interval contains 7 checkpoints (12 hours, 13 hours, 14 hours, 15 hours, 16 hours, 17 hours, and 18 hours). First, the time attribute association information of each checkpoint is extracted: the generation time is 12-18 hours, and the validation set accuracy is 85%, 86%, 88%, 87%, 89%, 90%, and 91%, respectively. Based on the filtering conditions, the checkpoints are... Checkpoints were sorted in descending order of generation time, and then by accuracy in descending order within the same generation time. Priority was then applied, resulting in the following order: 18 hours (91%) > 17 hours (90%) > 16 hours (89%) > 15 hours (87%) > 14 hours (88%) > 13 hours (86%) > 12 hours (85%). Based on this sorting, checkpoints generated within the last 18 hours were selected as retained checkpoints, and the remaining six were designated for deletion. By querying system process and task status identifiers, it was determined that none of the six checkpoints to be deleted had been invoked by tasks such as fault recovery or performance backtracking; their invocation status was "not invoked." Therefore, these six checkpoints were directly deleted. This approach significantly optimized the sublinear growth of checkpoint storage, and the retained checkpoints for each interval were all key data within that timeframe, ensuring backtracking and recovery capabilities.
[0056] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 , Figure 4This is a simplified flowchart of the checkpoint management method provided in Embodiment 2 of this application. After determining the invocation status of the checkpoint to be deleted, and if the invocation status of the checkpoint to be deleted is not invocation, the method further includes steps S31 to S32: Step S31: When the call status of the checkpoint to be deleted is "called", record the calling task associated with the checkpoint to be deleted and monitor the execution status of the calling task in real time. Step S32: If the execution status of the calling task is "execution completed", delete the checkpoint to be deleted.
[0057] It should be noted that the record of the invoked task must include core task identification information, specifically the task ID, the training process number to which it belongs, the task start time, the task type, and the checkpoint invocation path, to ensure accurate tracking of the association between the task and the checkpoint to be deleted. The invoked task is monitored through a combination of polling and status signal listening. The polling period can be customized based on the execution duration of the training task, such as once every 5 minutes. Status signal listening is achieved by capturing the system task completion signal to ensure timely acquisition of the execution status. The execution status of the invoked task is determined to be complete when the invoked task terminates without abnormality, all processes associated with the invoked task have been released, and the checkpoint resources are no longer occupied.
[0058] The process involves acquiring checkpoints generated during model training, where checkpoints record training data. Based on the temporal attribute information of the checkpoints, they are assigned to corresponding checkpoint retention intervals. The time span of any subsequent checkpoint retention interval is greater than the time span of its immediately preceding checkpoint retention interval. After a checkpoint retention interval is filled, at least one checkpoint is selected from the remaining checkpoints for retention, and unselected checkpoints are deleted. Specifically, this embodiment uses a rapidly growing sequence to determine checkpoint retention intervals. Leveraging the inherent characteristic of rapidly growing sequences where the span between adjacent terms increases with increasing term order, the resulting checkpoint retention intervals exhibit a differentiated distribution. Standardization and verification of the temporal attribute information ensure the accuracy and comparability of the temporal information for each checkpoint. Precise allocation through interval matching mapping avoids the chaotic issues of checkpoints being repeatedly assigned across intervals or having no interval to assign to, while also allowing scattered checkpoints to form ordered management units based on their generation sequence. At least one checkpoint is selected from the checkpoints in the checkpoint retention interval for retention, and unselected checkpoints in the retention interval are deleted. Considering the differentiated interval distribution, only a small number of key checkpoints are retained in each interval, while redundant data is removed. Although the checkpoint density is high in recent small-span intervals, the number of retained checkpoints in these intervals is consistent with that in long-term large-span intervals, causing the number of retained checkpoints per unit time to gradually decrease over training time. This achieves a sublinear growth in checkpoint storage over time. Periodic manual intervention to adjust the management strategy is no longer required, significantly improving the stability and reliability of checkpoint management.
[0059] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the checkpoint management method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.
[0060] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the checkpoint management method in Embodiment 1 above.
[0061] The following is for reference. Figure 5The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0062] like Figure 5 As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. While electronic devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.
[0063] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0064] The electronic device provided in this application, employing the checkpoint management method in the above embodiments, can solve the technical problem in the prior art where periodic manual intervention is required due to the linear increase in checkpoint storage, leading to a significant reduction in the stability and reliability of checkpoint management. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the checkpoint management method provided in the above embodiments, and other technical features of this electronic device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.
[0065] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0066] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0067] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the checkpoint management method in the above embodiments.
[0068] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0069] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.
[0070] The aforementioned computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to: acquire checkpoints generated during model training, wherein the checkpoints are used to record training data during model training; Based on the time attribute information of the checkpoints, the checkpoints are assigned to the corresponding checkpoint retention intervals, wherein the time span of the checkpoint retention interval of any later stage is greater than the time span of the checkpoint retention interval of the immediately preceding stage. After the checkpoint retention interval is filled with checkpoints, at least one checkpoint is selected from the checkpoints in the checkpoint retention interval for retention processing, and the unselected checkpoints in the checkpoint retention interval are deleted.
[0071] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0072] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0073] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0074] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described checkpoint management method. This solves the technical problem in the prior art where the linear increase in checkpoint storage volume necessitates periodic manual intervention, leading to a significant reduction in the stability and reliability of checkpoint management. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the checkpoint management method provided in the above embodiments, and will not be elaborated upon here.
[0075] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the checkpoint management method described above.
[0076] The computer program product provided in this application can solve the technical problem that the linear increase in checkpoint storage requires periodic manual intervention, leading to a significant reduction in the stability and reliability of checkpoint management. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the checkpoint management method provided in the above embodiments, and will not be repeated here.
[0077] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.
Claims
1. A checkpoint management method, characterized in that, The checkpoint management method includes: Obtain checkpoints generated during model training, wherein the checkpoints are used to record training data during model training; Based on the time attribute information of the checkpoints, the checkpoints are assigned to the corresponding checkpoint retention intervals, wherein the time span of the checkpoint retention interval of any later stage is greater than the time span of the checkpoint retention interval of the immediately preceding stage. After the checkpoint retention interval is filled with checkpoints, at least one checkpoint is selected from the checkpoints in the checkpoint retention interval for retention processing, and the unselected checkpoints in the checkpoint retention interval are deleted.
2. The checkpoint management method as described in claim 1, characterized in that, The step of assigning the checkpoint to the corresponding checkpoint retention interval based on the checkpoint's time attribute information includes: Generate a rapidly growing sequence based on initial seed parameters and a preset sequence length; The checkpoint retention interval is obtained by dividing the time dimension range of the checkpoint using the rapidly growing sequence. Based on the time attribute information of the checkpoints, an interval matching mapping relationship is constructed, and the checkpoints are assigned to the corresponding checkpoint retention intervals through the interval matching mapping relationship.
3. The checkpoint management method as described in claim 2, characterized in that, The step of dividing the time dimension range of the checkpoints using the rapidly growing series to obtain the checkpoint retention interval includes: Verify the fit between the numerical range of the rapidly growing sequence and the time dimension range, and adjust the numerical range based on the fit between the numerical range and the time dimension range so that the rapidly growing sequence fits the time dimension range. The checkpoint retention interval is obtained by taking two consecutive adjacent terms in the adapted fast-growing sequence as the start and end time boundaries, respectively, so that the time span of the checkpoint retention interval matches the numerical span of two adjacent terms in the fast-growing sequence.
4. The checkpoint management method as described in claim 3, characterized in that, The step of verifying the fit between the numerical range of the rapidly growing sequence and the time dimension range, and adjusting the numerical range based on the fit result of the numerical range and the time dimension range, includes: Determine whether the numerical range can cover the time dimension range; If the numerical range cannot cover the time dimension range, a new term is generated according to the recursive mechanism of the rapidly growing sequence. If the numerical range covers the time dimension range, the corresponding items are selected from the rapidly growing sequence for partitioning based on the start and end times of the time dimension range.
5. The checkpoint management method as described in claim 2, characterized in that, The step of constructing an interval matching mapping relationship based on the time attribute information of the checkpoints, and assigning the checkpoints to corresponding checkpoint retention intervals through the interval matching mapping relationship includes: The time attribute information is standardized to convert it into initial time data; The initial time data is verified to obtain the target time data; Based on the target time data, an interval matching mapping relationship is constructed, and the checkpoint is matched to the corresponding checkpoint retention interval through the interval matching mapping relationship.
6. The checkpoint management method as described in claim 1, characterized in that, The step of selecting at least one checkpoint from the checkpoint retention interval for retention processing and deleting the unselected checkpoints in the checkpoint retention interval includes: Extract the time attribute association information of checkpoints in the checkpoint retention interval, and create checkpoint filtering conditions based on the time attribute association information; The checkpoints in the checkpoint retention interval are prioritized according to the checkpoint filtering criteria. At least one checkpoint is selected to be retained based on the sorting result, and the checkpoints that are not selected in the checkpoint retention interval are determined as checkpoints to be deleted. Determine the invocation status of the checkpoint to be deleted. If the invocation status of the checkpoint to be deleted is not invoked, delete the checkpoint directly.
7. The checkpoint management method as described in claim 6, characterized in that, After determining the invocation status of the checkpoint to be deleted, and deleting the checkpoint directly when the invocation status is not invocation, the method further includes: When the call status of the checkpoint to be deleted is "called", record the calling task associated with the checkpoint to be deleted and monitor the execution status of the calling task in real time. If the execution status of the invoked task is "execution completed", then the checkpoint to be deleted is deleted.
8. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the checkpoint management method as described in any one of claims 1 to 7.
9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the checkpoint management method as described in any one of claims 1 to 7.
10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the checkpoint management method as described in any one of claims 1 to 7.