A method, device and storage medium for dynamically setting checkpoints

Abstract

This specification discloses a method, device, and storage medium for dynamically setting checkpoints. In this method, during the training process, resource usage data of a large model training cluster in the previous historical period can be periodically collected, and based on the collected resource usage data, first dynamic relationship data used to characterize the dynamic relationship of the system resource occupancy state over time in a future target period can be predicted. Then, the first dynamic relationship data can be compared with the global dynamic relationship data expected for the entire process when no failure occurs during the training of the large model, so as to determine the prediction result of whether the large model training cluster is likely to fail in the target period, and configure checkpoints based on the prediction result, thereby improving the effectiveness of the checkpoints configured during the training process of the large-scale model, thereby reducing the fixed overhead of periodic checkpoints and the loss of fault recovery.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a method, device, and storage medium for dynamically setting checkpoints. Background Art

[0002] With the rapid development of artificial intelligence (AI), natural language processing (NLP) technology, represented by large language models (LLMs), has achieved significant breakthroughs. These models have enabled outstanding performance in tasks such as intelligent question answering, text summarization, and machine translation, greatly expanding the boundaries of AI applications. For example, OpenAI's GPT series of models, from GPT-1 to GPT-4, has seen a continuous increase in model parameter size, from hundreds of millions to trillions today, demonstrating the potential for performance improvements brought about by scaling models.

[0003] However, the ever-expanding scale of models also presents unprecedented challenges. To process massive amounts of data and perform complex computational tasks, large-scale distributed clusters composed of specialized accelerators such as GPUs or TPUs are typically required. These clusters are interconnected via high-speed networks to enable parallel computing and accelerate the training process. However, as cluster size increases, especially when training clusters reach tens of thousands of GPUs, a series of issues begin to emerge, including load balancing between computing nodes, data synchronization, and communication overhead. These issues severely impact training efficiency and stability. Consequently, software and hardware failures are often unavoidable in such large computing clusters, especially in ultra-large-scale training scenarios. For example, GPT-4 training relied on approximately 25,000 A100 GPUs and ran continuously for 90 to 100 days. However, the system utilization (MFU) during training was only 32% to 36%. This extremely low utilization rate is due to frequent software and hardware failures in the cluster and the resulting training interruptions.

[0004] To address training interruptions caused by hardware and software failures, existing training systems typically periodically save intermediate states of model training. However, this periodic checkpointing approach is overly simplistic, resulting in significant fixed checkpointing overhead and failure recovery losses.

[0005] Therefore, how to improve the effectiveness of checkpoints configured during the training process of large-scale models to reduce the fixed overhead of periodic checkpoints and fault recovery losses is an urgent problem to be solved. Summary of the Invention

[0006] This specification provides a method, device, and storage medium for dynamically setting a checkpoint to partially solve the above-mentioned problems existing in the prior art.

[0007] This manual adopts the following technical solutions:

[0008] This specification provides a method for dynamically setting checkpoints, which is applied to a management device of a large model training cluster. The method includes:

[0009] Collect resource usage data of the large model training cluster during the model training process, where the resource usage data is used to characterize the system resource occupancy status of the large model training cluster at each moment in a historical period;

[0010] Predicting first dynamic relationship data of changes in system resource occupancy status of the large model training cluster over time within a target period based on the resource usage data;

[0011] determining at least a portion of dynamic relationship data matching the target time period in the global dynamic relationship data as second dynamic relationship data;

[0012] Based on whether the difference between the first dynamic relationship data and the second dynamic relationship data meets the preset warning conditions, a prediction result of whether the large model training cluster will fail within the target period is determined, and a checkpoint is set based on the prediction result; the global dynamic relationship data is the expected dynamic relationship of the entire model training process when no failure occurs.

[0013] Optionally, predicting first dynamic relationship data of the large model training cluster within a target period based on the resource usage data specifically includes:

[0014] Extracting at least one periodic component data from the resource usage data, wherein the periodic component data refers to at least a portion of data used to characterize regular and repetitive changes in the resource usage data, and different periodic component data correspond to different regularities;

[0015] Based on the periodic component data, first dynamic relationship data of the large model training cluster within a target time period is predicted.

[0016] Optionally, predicting first dynamic relationship data of the large model training cluster within a target period based on the periodic component data specifically includes:

[0017] For each periodic component data, padding the length of the periodic component data to a specified length to obtain adjusted periodic component data;

[0018] The adjusted period component data are fused to predict the first dynamic relationship data of the large model training cluster within the target time period.

[0019] Optionally, fusing the adjusted period component data to predict the first dynamic relationship data of the large model training cluster within the target period specifically includes:

[0020] The adjusted period component data are fused with the preset residual data to predict the first dynamic relationship data of the large model training cluster within the target period; the residual data is obtained by modeling the residual data used to determine the historical first dynamic relationship data of different historical periods through an autoregressive model.

[0021] Optionally, the method further includes:

[0022] determining a difference between the historical first dynamic relationship data of the historical period and at least a portion of dynamic relationship data in the global dynamic relationship data that matches the historical period as a reference difference;

[0023] When it is determined that the difference is greater than the reference difference, the parameters of the autoregressive model are adjusted to obtain an adjusted autoregressive model.

[0024] Optionally, extracting at least one periodic component data from the resource usage data specifically includes:

[0025] Determining at least a portion of the resource usage data that has regular and repetitive changes as a periodic resource usage data segment;

[0026] According to the predicted key point extraction strategy, each periodic key point is extracted from the periodic resource usage data segment, and periodic component data is obtained based on each periodic key point. The key point extraction strategy is used to screen each data point contained in the periodic resource usage data segment.

[0027] Optionally, the method further includes:

[0028] determining a difference between the historical first dynamic relationship data of the historical period and at least a portion of dynamic relationship data in the global dynamic relationship data that matches the historical period as a reference difference;

[0029] When it is determined that the difference is greater than the reference difference, adding a restriction condition for screening the data points to the key point extraction strategy to obtain an adjusted key point extraction strategy;

[0030] According to the adjusted key point extraction strategy, the periodic component data is re-extracted.

[0031] This specification provides a dynamic checkpoint setting device, including:

[0032] A collection module is used to collect resource usage data of the large model training cluster during the model training process, wherein the resource usage data is used to characterize the system resource occupancy status of the large model training cluster at each moment in a historical period;

[0033] A prediction module, configured to predict, based on the resource usage data, first dynamic relationship data of changes in the system resource occupancy status of the large model training cluster over time within a target period;

[0034] a determining module, configured to determine at least a portion of the dynamic relationship data in the global dynamic relationship data that matches the target time period as second dynamic relationship data;

[0035] A setting module is used to determine a prediction result of whether the large model training cluster will fail within a target period based on whether the difference between the first dynamic relationship data and the second dynamic relationship data meets a preset warning condition, and to set a checkpoint based on the prediction result; the global dynamic relationship data is the expected dynamic relationship of the entire model training process when no failure occurs.

[0036] This specification provides a management device for dynamic setting of checkpoints, the management device comprising: a resource monitoring module, a pattern prediction module, and a checkpoint setting module;

[0037] The resource monitoring module is used to collect resource usage data of the large model training cluster during the model training process and transmit it to the pattern prediction module, wherein the resource usage data is used to characterize the system resource occupancy status of the large model training cluster at each moment in the historical period;

[0038] The pattern prediction module is used to predict first dynamic relationship data of the large model training cluster within a target period based on the resource usage data and transmit the data to the checkpoint setting module; the first dynamic relationship data is used to represent the dynamic relationship of the system resource occupancy status of the large model training cluster within the target period over time;

[0039] The checkpoint setting module is used to determine at least part of the dynamic relationship data in the global dynamic relationship data that matches the target time period as second dynamic relationship data, and determine a prediction result of whether the large model training cluster will fail within the target time period based on whether the difference between the first dynamic relationship data and the second dynamic relationship data meets a preset warning condition, and set a checkpoint based on the prediction result; the global dynamic relationship data is the expected dynamic relationship of the entire model training process in the absence of a failure.

[0040] This specification provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method for dynamically setting a checkpoint is implemented.

[0041] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0042] In the dynamic setting method of checkpoints provided in this specification, resource usage data of the large model training cluster is first collected during the model training process. The resource usage data is used to characterize the system resource occupancy status of the large model training cluster at each moment in the historical period. Based on the resource usage data, the first dynamic relationship data of the system resource occupancy status of the large model training cluster changing over time in the target period is predicted. At least part of the dynamic relationship data in the global dynamic relationship data that matches the target period is determined as the second dynamic relationship data. Based on whether the difference between the first dynamic relationship data and the second dynamic relationship data meets the preset early warning conditions, the prediction result of whether the large model training cluster will fail in the target period is determined, and based on the prediction result, a checkpoint is set. The global dynamic relationship data is the expected dynamic relationship of the entire model training process when no failure occurs.

[0043] It can be seen from the above method that before model training is performed through a large model training cluster, the dynamic relationship of the expected system resource occupancy status over time in the entire process can be determined based on the historical model training process when no abnormalities occur during the training of the large model, as global dynamic relationship data. During the training process, the resource usage data of the large model training cluster in the previous historical period is periodically collected to predict the first dynamic relationship data used to characterize the dynamic relationship of the system resource occupancy status over time in the future target period based on the collected resource usage data. The first dynamic relationship data can be compared with the global dynamic relationship data to determine the prediction result of whether the large model training cluster may fail in the target period, and checkpoints can be configured based on the prediction result. This can improve the effectiveness of the checkpoints configured during the training process of large-scale models and reduce the fixed overhead of periodic checkpoints and fault recovery losses. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0045] Figure 1 A flowchart of a dynamic checkpoint setting method provided in this specification;

[0046] Figure 2A schematic diagram of a process of a dynamic checkpoint setting method provided in this specification;

[0047] Figure 3 A schematic diagram of the adjustment process provided in this manual;

[0048] Figure 4 A schematic diagram of a dynamic setting device for a checkpoint provided in this specification;

[0049] Figure 5 This is a schematic diagram of a management device for dynamic setting of checkpoints provided in this specification. DETAILED DESCRIPTION

[0050] To make the objectives, technical solutions, and advantages of this specification more clear, the following will clearly and completely describe the technical solutions of this specification in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0051] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0052] Currently, to address training interruptions caused by software and hardware failures, existing training systems typically periodically save intermediate states of model training to roll back the training task to the most recent checkpoint in the event of a failure. However, due to the following issues with checkpoint setup: checkpoints need to save the complete state of the model, including model weights, optimizer state, random number seeds, etc., as the model scale increases, the storage requirements for this data will increase significantly (for example, a trillion-parameter model may require hundreds of terabytes of storage space to save checkpoints). In addition, the checkpoint saving process will block the training process, causing training to pause. This time overhead depends on the size of the model and the performance of the storage system (for example, saving a checkpoint of a large model may take hours). As a result, this periodic checkpointing method will result in large fixed checkpoint overhead and failure recovery losses.

[0053] Figure 1 The following is a flowchart of a method for dynamically setting a checkpoint provided in this specification, including the following steps:

[0054] S101: Collect resource usage data of the large model training cluster during the model training process, where the resource usage data is used to characterize the system resource occupancy status of the large model training cluster at each moment in a historical period.

[0055] In this specification, the above-mentioned dynamic setting method of checkpoints is applied to the management device of the large model training cluster, that is, the dynamic setting method of checkpoints can be executed by the management device to dynamically manage the checkpoints that need to be set during the model training process through the large model training cluster.

[0056] Among them, the above-mentioned management device can be a designated node device selected from the large model training cluster, or it can refer to a device deployed with a CPU, operating system, and task caller for executing a dynamic setting method for checkpoints.

[0057] Furthermore, the management device may collect resource usage data of the large model training cluster during the model training process at specified time intervals during the model training process using the large model training cluster.

[0058] Among them, the above-mentioned resource usage data is used to characterize the dynamic relationship between the system resource occupancy status of the large model training cluster in the historical period and time. The resource usage data here can be time series data composed of system resource occupancy data at different times included in the historical period.

[0059] The above-mentioned system resource occupancy data may be data used to reflect the occupancy of system resources such as CPU, memory, storage, etc., for example, CPU occupancy rate, memory usage rate, disk occupancy rate, etc.

[0060] The above historical period can be set based on actual needs. For example, a specified length of time before the current time can be used as the historical period. The specified length can be determined based on actual needs. For example, the length of time required for one round of iterative model training can be used as the specified length. Another example is to use every hour as the specified length.

[0061] S102: Predicting first dynamic relationship data of changes in the system resource occupancy status of the large model training cluster over time within a target time period based on the resource usage data.

[0062] In this specification, after collecting resource usage data, the management device can perform time series analysis based on the resource usage data to predict the first dynamic relationship data of the system resource occupancy status of the large model training cluster changing over time during the target period.

[0063] Specifically, the management device may input the resource usage data into a pre-trained prediction model to predict the first dynamic relationship data of the large model training cluster within the target period based on the resource usage data through the prediction mode.

[0064] It should be noted that during the process of model training using a large model training cluster, the resource usage data may contain different sub-data with regular and repetitive changes due to factors such as task scheduling strategies, resource allocation strategies, the system administrator's configuration of the large model training cluster, the system administrator's optimization habits for the large model training cluster, the training phase characteristics of the model training itself, and training data loading or processing conditions.

[0065] For example, large model training clusters often employ specific task scheduling strategies, such as round-robin and priority scheduling. These strategies regularly distribute model training tasks to different nodes based on their priority and resource requirements. For example, a batch of high-priority tasks may be scheduled in a certain time period each day, resulting in peak CPU, memory, and disk resource usage during that time period.

[0066] The different resource allocation strategies used by large model training clusters will cause them to reallocate resources at specific times each day to optimize resource utilization. This may cause short-term fluctuations in resource utilization after the reallocation.

[0067] If the system administrator configures the large model training cluster in a way that causes the nodes in the cluster to be unevenly configured, some nodes may take on more computing tasks in a specific time period, which will lead to periodic changes in resource utilization.

[0068] System administrators' habit of optimizing large model training clusters may lead them to regularly optimize the clusters, such as adjusting resource allocation parameters and upgrading software versions. These optimization operations may change resource utilization patterns in the short term, leading to periodic changes.

[0069] Large model training typically involves multiple phases, such as data loading, model training, and parameter updates. Different phases require different resources, which can lead to periodic fluctuations in resource utilization. For example, the data loading phase may require a large number of I / O operations, resulting in high disk resource utilization, while the model training phase may rely more heavily on CPU and memory resources.

[0070] The size and complexity of the training data also affect resource utilization. If a large amount of training data is loaded or processed during a certain period of time each day, it may cause periodic peaks in resource utilization.

[0071] Based on this, in order to improve the accuracy of the determined first dynamic relationship data, the server can also extract at least one periodic component data from the resource usage data, and predict the first dynamic relationship data of the large model training cluster within the target period based on the periodic component data.

[0072] In the above content, periodic component data refers to at least part of the data used to characterize regular and repetitive changes in resource usage data, and different periodic component data correspond to different regularities.

[0073] For example, if the length of the historical period is the past 72 hours, the resource usage data for the historical period is a time series consisting of the system resource usage data for each hour in the past 72 hours. In this case, the resource usage data within every 48 hours of the resource usage data for the historical period may show a periodic fluctuation due to the fact that large model training is usually divided into multiple stages and each stage has a different impact on the system resource usage data. In this case, a periodic component data extracted from the resource usage data for the historical period can be determined based on the resource usage data within every 48 hours of the resource usage data for the historical period.

[0074] For another example: if the length of the historical period is the past 72 hours, the resource usage data for the historical period is a time series consisting of the system resource usage data for each hour in the past 72 hours. At this time, due to the system administrator's habit of regularly optimizing the cluster, the resource usage data within every 24 hours of the resource usage data for the historical period may show a periodic fluctuation. At this time, a periodic component data extracted from the resource usage data for the historical period can be determined based on the resource usage data within every 24 hours of the resource usage data for the historical period.

[0075] Specifically, the management device can determine at least part of the data in the resource usage data that has regular and repetitive changes as periodic resource usage data segments, and then extract each periodic key point from the periodic resource usage data segments according to the predicted key point extraction strategy, and obtain periodic component data based on each periodic key point.

[0076] In the above content, the feature vector data used to characterize each periodic key point can be determined according to the value of each periodic key point as the periodic component data, wherein the value of each dimension in the feature vector data is the value of each periodic key point.

[0077] In the above content, the key point extraction strategy is used to filter the data points included in the periodic resource usage data segment. For example, the key point extraction strategy can be: starting from the first data point of each data point included in the periodic resource usage data segment, every specified number of data points (e.g., every 50 data points) are selected as key points.

[0078] For another example, the peak points, trough points, turning points (i.e., points where the data trend changes significantly, such as from an upward trend to a downward trend, or from a downward trend to an upward trend), mutation points (points where the data value suddenly changes significantly), and inflection points (points where the second-order derivative is zero, usually indicating a change in the curvature of the data) of each data point contained in the periodic resource usage data segment are selected as key points.

[0079] Furthermore, after extracting at least one periodic component data from the resource usage data, the management device can fill the length of each periodic component data to a specified length to obtain adjusted periodic component data, and fuse the adjusted periodic component data to calculate the first dynamic relationship data of the large model training cluster within the target time period.

[0080] For example, if the length of the periodic component data is 24 and the specified length is 48, the periodic component data can be copied and padded to feature vector data with a length of 48 to obtain the adjusted periodic component data.

[0081] To make the prediction results more accurate, the management device can also fuse the adjusted period component data with the preset residual data to calculate the first dynamic relationship data of the large model training cluster within the target period. For details, please refer to the following formula:

[0082]

[0083] In the above formula, That is, the value of the i-th dimension in the calculated first dynamic relationship data of the large model training cluster in the target period (that is, the feature vector data determined based on the adjusted period component data to characterize the time-varying system resource occupancy status of the large model training cluster in the target period), That is, the value of the i-th dimension contained in the j-th adjusted period component data, is the value of the i-th dimension contained in the residual data.

[0084] It can be seen from the above formula that the first dynamic relationship data of the large model training cluster in the target period can be modeled according to the system resource occupancy status of the large model training cluster at each moment in the historical period, so as to dynamically set the checkpoint according to the modeling results.

[0085] The above residual data is obtained by modeling the residual data used to determine the historical first dynamic relationship data of different historical periods through an autoregressive model. For details, please refer to the following formula:

[0086]

[0087] In the above formula, is the error term, and These are two parameters that can be adjusted according to actual needs. Of course, they can be prior data.

[0088] In addition, the management device can calculate the first dynamic relationship data of the large model training cluster within the target time period based on the periodic component data by inputting the above-mentioned periodic component data into a preset prediction model, so as to predict the dynamic relationship of the system resource occupancy status of the large model training cluster within the target time period over time based on the periodic component data through the preset prediction model.

[0089] S103: Determine at least a portion of the dynamic relationship data in the global dynamic relationship data that matches the target time period as second dynamic relationship data.

[0090] S104: Determine a prediction result of whether the large model training cluster will fail within a target period based on whether the difference between the first dynamic relationship data and the second dynamic relationship data meets a preset warning condition, and set a checkpoint based on the prediction result; the global dynamic relationship data is the expected dynamic relationship of the entire model training process when no failure occurs.

[0091] In this specification, the management device can determine at least part of the dynamic relationship data in the global dynamic relationship data that matches the target time period as the second dynamic relationship data, and then determine the prediction result of whether the large model training cluster will fail within the target time period based on whether the difference between the first dynamic relationship data and the above-mentioned second dynamic relationship data meets the preset warning condition, and then perform a checkpoint operation based on the prediction result to configure the checkpoint. Figure 2 shown.

[0092] Figure 2 This is a schematic diagram of the process of a dynamic checkpoint setting method provided in this specification.

[0093] Combine Figure 2 It can be seen that after each round of model training is completed, the management device can use the past round of model training as a historical period. Then, based on the resource usage data in the historical period, it can determine the prediction result of whether the large model training cluster will have a failure in the next round of model training, that is, within the target period. Checkpoints can be saved based on the prediction results. In this way, the fixed overhead and recovery loss of the checkpoint mechanism can be greatly reduced on the basis of realizing the intermediate training status recording of the model and avoiding the impact of system failures on training, thereby improving the efficiency of model training.

[0094] It should be noted that the global dynamic relationship data described above is the expected dynamic relationship for the entire model training process, assuming no failures, determined before model training based on the historical model training process. This can be understood as the initial integration of actual resource usage data collected at every moment of the historical model training task executed by the large model training cluster, excluding failures.

[0095] The above-mentioned warning conditions can be set according to actual needs. For example, if the difference between the above-mentioned first dynamic relationship data and at least part of the dynamic relationship data in the preset global dynamic relationship data that matches the target time period is greater than a preset difference threshold, it can be considered that the preset warning conditions are met.

[0096] In actual application scenarios, in order to improve the accuracy of the prediction results, the management device can also adjust the parameters and key point extraction strategies used in the process of calculating the first dynamic relationship data of the large model training cluster within the target period according to the change trend between the difference between the first dynamic relationship data and the second dynamic relationship data and the reference difference, as shown in the following example: Figure 3 shown.

[0097] Figure 3 This is a schematic diagram of the adjustment process provided in this manual.

[0098] Combine Figure 3 It can be seen that the management device can determine the difference between the historical first dynamic relationship data of the historical period and at least part of the dynamic relationship data in the global dynamic relationship data that matches the historical period as a reference difference. When it is determined that the difference between the first dynamic relationship data and the second dynamic relationship data is greater than the reference difference, the parameters of the autoregressive model (i.e., , and At least one parameter in is adjusted to obtain the adjusted autoregressive model.

[0099] In addition, when it is determined that the difference between the first dynamic relationship data and at least part of the dynamic relationship data in the preset global dynamic relationship data that matches the target time period is greater than the reference difference, the management device can also add restriction conditions for screening each data point in the key point extraction strategy to obtain an adjusted key point extraction strategy, and re-extract the periodic component data according to the adjusted key point extraction strategy.

[0100] The above constraints can be set based on actual needs. For example, if the above key point extraction strategy selects data points every 10 seconds in the periodic resource usage data segment as the periodic key points, the above new constraint can be set to select data points every 5 seconds in the periodic resource usage data segment as the periodic key points.

[0101] For another example, if the above key point extraction strategy selects data points corresponding to peaks or troughs in a periodic resource usage data segment as the selected periodic key points, then the above newly added restriction condition can be to select data points corresponding to turning points in the periodic resource usage data segment as the selected periodic key points.

[0102] It is worth noting that in actual application scenarios, if the difference between the above-mentioned first dynamic relationship data and at least part of the dynamic relationship data in the preset global dynamic relationship data that matches the target time period is greater than the reference difference, it indicates that the first dynamic relationship data of the large model training cluster calculated by the above method within the target time period may have errors.

[0103] Therefore, in order to avoid the adverse effects of errors, the management device can directly set a checkpoint without considering the prediction result when it determines that the difference between the above-mentioned first dynamic relationship data and at least part of the dynamic relationship data in the preset global dynamic relationship data that matches the target time period is greater than the reference difference, so as to prevent possible inaccurate prediction results from having an adverse impact on the model training process. At the same time, the parameters and key point extraction strategies used in the process of calculating the first dynamic relationship data of the large model training cluster within the target time period are adjusted through the above-mentioned adjustment method.

[0104] It can be seen from the above method that before model training is performed through a large model training cluster, the dynamic relationship of the expected system resource occupancy status over time in the entire process can be determined based on the historical model training process when no abnormalities occur during the training of the large model, as global dynamic relationship data. During the training process, the resource usage data of the large model training cluster in the previous historical period is periodically collected to predict the first dynamic relationship data used to characterize the dynamic relationship of the system resource occupancy status over time in the future target period based on the collected resource usage data. The first dynamic relationship data can be compared with the global dynamic relationship data to determine the prediction result of whether the large model training cluster may fail in the target period, and checkpoints can be configured based on the prediction result. This can improve the effectiveness of the checkpoints configured during the training process of large-scale models and reduce the fixed overhead of periodic checkpoints and fault recovery losses.

[0105] The above is a method for dynamically setting one or more implementation checkpoints of this specification. Based on the same idea, this specification also provides a corresponding dynamic setting device for checkpoints, such as Figure 4 shown.

[0106] Figure 4 A schematic diagram of a dynamic checkpoint setting device provided in this specification includes:

[0107] The collection module 401 is used to collect resource usage data of the large model training cluster during the model training process, wherein the resource usage data is used to represent the system resource occupancy status of the large model training cluster at each moment in a historical period;

[0108] Prediction module 402, configured to predict first dynamic relationship data of changes in system resource occupancy status of the large model training cluster over time within a target period based on the resource usage data;

[0109] A determination module 403 is configured to determine at least a portion of the dynamic relationship data in the global dynamic relationship data that matches the target time period as second dynamic relationship data;

[0110] The setting module 404 is used to determine the prediction result of whether the large model training cluster will fail within the target period based on whether the difference between the first dynamic relationship data and the second dynamic relationship data meets the preset warning condition, and set a checkpoint based on the prediction result; the global dynamic relationship data is the expected dynamic relationship of the entire model training process when no failure occurs.

[0111] Optionally, the prediction module 402 is specifically used to extract at least one periodic component data from the resource usage data, where the periodic component data refers to at least part of the data used to characterize regular and repetitive changes in the resource usage data, and different periodic component data correspond to different regularities; based on the periodic component data, predict the first dynamic relationship data of the large model training cluster within the target time period.

[0112] Optionally, the prediction module 402 is specifically used to fill the length of each periodic component data to a specified length to obtain adjusted periodic component data; and fuse the adjusted periodic component data to predict the first dynamic relationship data of the large model training cluster within the target time period.

[0113] Optionally, the prediction module 402 is specifically used to fuse each adjusted period component data with preset residual data to predict the first dynamic relationship data of the large model training cluster within the target period; the residual data is obtained by modeling the residual data used to determine the historical first dynamic relationship data of different historical periods through an autoregressive model.

[0114] Optionally, the prediction module 402 is also used to determine the difference between the historical first dynamic relationship data of the historical period and at least part of the dynamic relationship data in the global dynamic relationship data that matches the historical period as a reference difference; when it is determined that the difference is greater than the reference difference, the parameters of the autoregressive model are adjusted to obtain an adjusted autoregressive model.

[0115] Optionally, the prediction module 402 is specifically used to determine at least part of the data in the resource usage data that has regular and repetitive changes as periodic resource usage data segments; according to the predicted key point extraction strategy, extract each periodic key point from the periodic resource usage data segment, and obtain periodic component data based on each periodic key point, and the key point extraction strategy is used to screen each data point contained in the periodic resource usage data segment.

[0116] Optionally, the prediction module 402 is specifically used to determine the difference between the historical first dynamic relationship data of the historical period and at least part of the dynamic relationship data in the global dynamic relationship data that matches the historical period as a reference difference; when it is determined that the difference is greater than the reference difference, adding a restriction condition for screening the data points in the key point extraction strategy to obtain an adjusted key point extraction strategy; and re-extracting the periodic component data according to the adjusted key point extraction strategy.

[0117] For ease of understanding, this manual also provides a dynamic checkpoint setting system, specifically Figure 5 shown.

[0118] Figure 5 This is a schematic diagram of a management device for dynamic setting of checkpoints provided in this specification.

[0119] Combine Figure 5 It can be seen that the above-mentioned management device includes: a resource monitoring module, a pattern prediction module, and a checkpoint setting module.

[0120] The resource monitoring module is used to collect resource usage data of the large model training cluster during model training and transmit it to the pattern prediction module. The resource usage data here is used to represent the system resource occupancy status of the large model training cluster at each moment in the historical period.

[0121] The pattern prediction module is configured to calculate first dynamic relationship data for the large model training cluster during a target period based on the resource usage data and transmit the data to the checkpoint setting module. The first dynamic relationship data is used to represent the dynamic relationship between the system resource usage status of the large model training cluster during the target period and how it changes over time.

[0122] The checkpoint setting module is used to determine at least part of the dynamic relationship data in the global dynamic relationship data that matches the target time period as the second dynamic relationship data, and determine the prediction result of whether the large model training cluster will fail within the target time period based on whether the difference between the first dynamic relationship data and the second dynamic relationship data meets the preset warning condition, and set a checkpoint based on the prediction result.

[0123] The above-mentioned global dynamic relationship data is determined before model training based on the historical model training process.

[0124] It should be noted that the aforementioned resource monitoring module and pattern prediction module include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types for the dynamic checkpoint configuration method described above. In a large-model training environment, the dynamic checkpoint configuration method described above is implemented by the added resource monitoring module, pattern prediction module, and a checkpoint configuration module responsible for recording large-model training checkpoints. In a large-model training environment, the checkpoint configuration module is typically located in a node within the large-model training cluster.

[0125] It can be seen from the above content that before model training is performed through the large model training cluster, the dynamic relationship of the expected system resource occupancy status over time during the entire process can be determined based on the historical model training process when no abnormalities occur during the training of the large model, as global dynamic relationship data. During the training process, the resource monitoring module, pattern prediction module, and checkpoint setting module contained in the management device can be used to periodically collect resource usage data of the large model training cluster in the previous historical period, so as to predict the first dynamic relationship data used to characterize the dynamic relationship of the system resource occupancy status over time in the future target period based on the collected resource usage data. Then, the first dynamic relationship data can be compared with the global dynamic relationship data to determine the prediction result of whether the large model training cluster may fail in the target period, and checkpoints can be configured based on the prediction results, so as to improve the accuracy of the checkpoints configured during the training process of large-scale models, so as to improve the training efficiency of the model and the utilization of system resources.

[0126] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1Provides a dynamic checkpoint setting method.

[0127] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0129] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0130] The foregoing is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A method for dynamically setting a checkpoint, characterized in that: The method is applied to a management device of a large model training cluster, and the method includes: Collect resource usage data of the large model training cluster during the model training process, where the resource usage data is used to characterize the system resource occupancy status of the large model training cluster at each moment in a historical period; Extracting at least one periodic component data from the resource usage data, the periodic component data comprising: sub-data with regular and repetitive changes due to at least one of the following factors: training phase characteristics of the model training itself, task scheduling strategy, resource allocation strategy, system administrator's configuration of the large model training cluster, system administrator's optimization habits for the large model training cluster, and training data loading or processing conditions; For each periodic component data, padding the length of the periodic component data to a specified length to obtain adjusted periodic component data; The adjusted period component data are fused with preset residual data to predict the first dynamic relationship data of the large model training cluster in the target period; the residual data is obtained by modeling the residual data used in determining the historical first dynamic relationship data of different historical periods through an autoregressive model; determining at least a portion of dynamic relationship data matching the target time period in the global dynamic relationship data as second dynamic relationship data; Based on whether the difference between the first dynamic relationship data and the second dynamic relationship data meets the preset warning conditions, a prediction result of whether the large model training cluster will fail within the target period is determined, and a checkpoint is set based on the prediction result; the global dynamic relationship data is the expected dynamic relationship of the entire model training process when no failure occurs.

2. The method according to claim 1, wherein The method further comprises: determining a difference between the historical first dynamic relationship data of the historical period and at least a portion of dynamic relationship data in the global dynamic relationship data that matches the historical period as a reference difference; When it is determined that the difference is greater than the reference difference, the parameters of the autoregressive model are adjusted to obtain an adjusted autoregressive model.

3. The method according to claim 1, wherein Extracting at least one periodic component data from the resource usage data specifically includes: Determining at least a portion of the resource usage data that has regular and repetitive changes as a periodic resource usage data segment; According to the predicted key point extraction strategy, each periodic key point is extracted from the periodic resource usage data segment, and periodic component data is obtained based on each periodic key point. The key point extraction strategy is used to screen each data point contained in the periodic resource usage data segment.

4. The method according to claim 3, wherein The method further comprises: determining a difference between the historical first dynamic relationship data of the historical period and at least a portion of dynamic relationship data in the global dynamic relationship data that matches the historical period as a reference difference; When it is determined that the difference is greater than the reference difference, adding a restriction condition for screening the data points to the key point extraction strategy to obtain an adjusted key point extraction strategy; According to the adjusted key point extraction strategy, the periodic component data is re-extracted.

5. A dynamic setting device for checkpoints, characterized in that: include: A collection module is used to collect resource usage data of the large model training cluster during the model training process, wherein the resource usage data is used to characterize the system resource occupancy status of the large model training cluster at each moment in a historical period; A prediction module is configured to extract at least one periodic component data from the resource usage data, the periodic component data including sub-data with regular and repetitive changes due to at least one of the following factors: training phase characteristics of the model training itself, task scheduling strategy, resource allocation strategy, system administrator's configuration of the large model training cluster, system administrator's optimization habits for the large model training cluster, and training data loading or processing conditions; for each periodic component data, pad the length of the periodic component data to a specified length to obtain adjusted periodic component data; fuse each adjusted periodic component data with preset residual data to predict the first dynamic relationship data of the large model training cluster within a target time period; the residual data is obtained by modeling the residual data used in determining the historical first dynamic relationship data of different historical time periods using an autoregressive model; a determining module, configured to determine at least a portion of the dynamic relationship data in the global dynamic relationship data that matches the target time period as second dynamic relationship data; A setting module is used to determine a prediction result of whether the large model training cluster will fail within a target period based on whether the difference between the first dynamic relationship data and the second dynamic relationship data meets a preset warning condition, and to set a checkpoint based on the prediction result; the global dynamic relationship data is the expected dynamic relationship of the entire model training process when no failure occurs.

6. A management device for dynamic setting of checkpoints, characterized in that: The management device includes: a resource monitoring module, a pattern prediction module, and a checkpoint setting module; The resource monitoring module is used to collect resource usage data of the large model training cluster during the model training process and transmit it to the pattern prediction module, wherein the resource usage data is used to characterize the system resource occupancy status of the large model training cluster at each moment in the historical period; The pattern prediction module is used to extract at least one periodic component data from the resource usage data, and the periodic component data includes: sub-data with regular and repetitive changes due to at least one of the training phase characteristics of the model training itself, the task scheduling strategy, the resource allocation strategy, the system administrator's configuration of the large model training cluster, the system administrator's optimization habits for the large model training cluster, and the training data loading or processing conditions; for each periodic component data, the length of the periodic component data is padded to a specified length to obtain adjusted periodic component data; each adjusted periodic component data is fused with the preset residual data to predict the first dynamic relationship data of the large model training cluster within the target time period and transmit it to the checkpoint setting module; the residual data is obtained by modeling the residual data used when determining the historical first dynamic relationship data of different historical time periods through an autoregressive model; the first dynamic relationship data is used to characterize the dynamic relationship of the system resource occupancy status of the large model training cluster within the target time period over time; The checkpoint setting module is used to determine at least part of the dynamic relationship data in the global dynamic relationship data that matches the target time period as second dynamic relationship data, and determine a prediction result of whether the large model training cluster will fail within the target time period based on whether the difference between the first dynamic relationship data and the second dynamic relationship data meets a preset warning condition, and set a checkpoint based on the prediction result; the global dynamic relationship data is the expected dynamic relationship of the entire model training process in the absence of a failure.

7. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images