Model distributed training automatic fault tolerance method in large-scale cloud native scene

Abstract

The invention provides an automatic fault tolerance method for distributed training of a model in a large-scale cloud native scene, and relates to the technical field of data management, and the method comprises the steps: collecting and based on hardware monitoring data of each node in a cluster, scheduling a distributed training task to each node, and starting check point storage; monitoring the running state of each training task, collecting a training log and a chip acceleration platform log, and collecting CPU, memory and acceleration card resource index data of each node and each training task; building a model training fault detection classification model by combining a supervision and machine learning method on the basis of the collected logs and hardware index data; and based on the model training fault detection classification model, judging whether each running training task has a fault and the fault type, and if the node equipment has a fault, rescheduling and loading the latest check point data to finish training task breakpoint continuous training and automatic fault tolerance. According to the invention, model breakpoint continuous training is realized, and the stability and efficiency of model training are improved.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and in particular to an automatic fault-tolerant method for distributed model training in large-scale cloud-native scenarios. Background Technology

[0002] We are currently in the midst of an artificial intelligence boom, with large-scale models constantly emerging and rapidly iterating. Training large-scale models requires a significant number of accelerator cards, such as GPUs and NPUs. For example, OpenAI trained GPT-4 on approximately 25,000 A100 GPUs for 90 to 100 days. Baidu's Wenxin Yiyan 4.0 large model also took dozens of days to complete training on a cluster with tens of thousands of GPUs. The more training nodes there are, the longer the training time, and the greater the probability of node failure during training. Therefore, large-scale training jobs inevitably encounter machine failures. Frequent failures can lead to training interruptions, wasted computation, and cluster idleness, resulting in a significant waste of time and computing resources.

[0003] Currently, fault tolerance in distributed training scenarios ensures training stability to a certain extent. Distributed training uses data parallelism or model parallelism to distribute the workload, ensuring that if some nodes fail, others can continue working. Furthermore, based on Kubernetes' default container scheduling system within a cloud-native architecture, when a node fails, tasks can have resources reallocated and scheduled before training restarts. Training task exits or is interrupted for various reasons; the orchestration and scheduling system can only automatically schedule tasks based on their instance states and restart training.

[0004] In cloud-native Kubernetes scenarios, while it's possible to intervene by judging the status of training task instances to restart or schedule them—that is, to intervene and restart or schedule if a training task instance is not running—this method cannot determine whether the problem lies with the training program itself or the node hardware, and cannot automatically provide fault tolerance based on the cause. Furthermore, the fault tolerance process lacks mechanisms to safeguard the training process, such as whether the entire model training will be affected after rescheduling, and how to elastically scale task instance resources to cope with training failures due to insufficient instances or resources when no nodes are available for scheduling. Moreover, if training is interrupted, although some algorithm frameworks now have the ability to resume training from where it left off, this is stored in storage, and reloading checkpoints can take several minutes, increasing training time and risk. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides an automatic fault-tolerance method for distributed model training in large-scale cloud-native scenarios. Based on Kubernetes with custom extensions, it adds a diagnostic module for slow or suspended training of large models, in addition to monitoring the running status of task instances. The priority of task repair is determined by the task suspension duration. Simultaneously, training logs, GPU monitoring metrics, and CUDA running logs are analyzed to achieve fault detection and diagnosis. Based on the fault diagnosis results, fault handling is tiered, ranging from retrying to retraining. When a node fault is detected, an automatic node detection process is introduced to quickly resume training after a fault occurs. To enhance training continuity, the training state is saved to shared memory based on checkpoints and then written to the distributed storage system. When a fault occurs, the training task can load the most recent checkpoint from memory, allowing the training program to resume training from the state of the model and optimizer before the fault.

[0006] To achieve the above objectives, this invention provides an automatic fault-tolerance method for distributed model training in large-scale cloud-native scenarios, comprising: Collect hardware monitoring data from each node in the cluster, configure distributed training tasks, schedule the distributed training tasks to each node based on the hardware monitoring data of each node, and enable checkpoint saving on each node. Monitor the running status of each training task, collect training logs and chip acceleration platform logs, and collect CPU, memory and accelerator card resource metrics data for each node and training task. Based on training logs, chip acceleration platform logs, and CPU, memory, and accelerator card resource metrics data for each node and training task, a fault detection classification model based on multimodal data is constructed by combining supervised and machine learning methods. For each training task that is running, the system analyzes in real time whether there is a fault and the type of fault by combining the running status of each training task with the model training fault detection classification model. If the fault is a node device fault, the training task with the fault is rescheduled. After the training task is rescheduled due to a fault, the most recent checkpoint data is loaded upon startup to complete the training task breakpoint continuation and automatic fault tolerance.

[0007] As a further improvement of the present invention, hardware monitoring data of each node in the cluster is collected, distributed training tasks are configured, and the distributed training tasks are scheduled to each node based on the hardware monitoring data of each node, including: Collect hardware monitoring data from each node in the cluster, analyze the hardware monitoring data of each node in the cluster based on the node health detection model, determine whether each node has a fault, and set the faulty node to be uncallable. Configure the number of training instances, the resources required for each instance, and the log output directory, and submit them to Kubernetes; The distributed training scheduler in Kubernetes allocates training tasks to available nodes based on the submitted configuration information.

[0008] As a further improvement to the present invention, a node health detection model is constructed, including: Deploy a hardware detection model in the cluster to obtain hardware monitoring data of each node in real time, including the health status of the accelerator card and hardware monitoring index data. The hardware monitoring data is preprocessed, including deduplication, completion, and format standardization. Based on the preprocessed data, the correlation between the health status and various hardware monitoring indicators is analyzed. Based on the random forest regression algorithm and the correlation between the health status and various hardware monitoring indicators, a node health detection model based on hardware monitoring indicators is constructed.

[0009] As a further improvement of the present invention, the running status of each training task is monitored, training logs and chip acceleration platform logs are collected, and CPU, memory, and accelerator card resource index data of each node and training task are collected; including: Log collectors and monitoring metric collectors are deployed on each node device with an accelerator card, and data receivers and task instance status listeners are deployed on the management and control end of the node cluster. The log collector collects training logs and chip acceleration platform logs during the training task execution on the node and sends them to the data receiver. The monitoring index collector collects CPU, memory, and accelerator card resource index data of the node and the training task in the node. The management terminal periodically obtains the data collected by the monitoring index collector. The task instance status listener uses the list / watch pattern through the API provided by Kubernetes to listen to the running status and events of the training tasks, obtain the running status, nodes and instance events of each training task, and record when a status change occurs.

[0010] As a further improvement of the present invention, based on training logs, chip acceleration platform logs, and CPU, memory, and accelerator card resource index data of each node and training task, a fault detection classification model based on multimodal data is constructed by combining supervised and machine learning methods; including: The training logs, chip acceleration platform logs, and CPU, memory, and accelerator card resource metrics data of each node and training task are preprocessed and then aligned with time series to obtain log and monitoring synchronization data. An attention mechanism is introduced to assign different weights to each feature in the log and monitoring synchronization data to form a new vector. The vector is then enhanced and fused through an encoder layer with a cross-attention mechanism to obtain a fused vector. The features of the fused vector are learned based on the Transformer model to obtain the relationship between log data and monitoring data, and the model is trained based on a multi-classification algorithm to obtain a fault detection classification model.

[0011] As a further improvement of the present invention, the fault detection classification model is trained by calculating and optimizing the model based on the multivariate cross-entropy loss function.

[0012] As a further improvement of the present invention, for each training task currently in operation, the present invention combines the running status of each training task with the real-time analysis of the model training fault detection and classification model to determine whether a fault exists and its type, including: The model training fault detection classification model obtains training logs of each running training task, chip acceleration platform logs, and CPU, memory, and accelerator card resource index data of the node and training task, and calculates the fault type, including node device fault, task itself fault, and normal.

[0013] As a further improvement of the present invention, if a node device fails, the training task for the failed device is rescheduled; including: If a node device fails, then the failed node is set to unschedulable. The distributed training scheduler in Kubernetes reschedules training tasks based on the number of available nodes in the current cluster and the resource requirements of the failed training task.

[0014] As a further improvement of the present invention, after the training task for the fault is rescheduled, the most recent checkpoint data is loaded upon startup to complete the training task breakpoint continuation and automatic fault tolerance; including: Each training task uses a sidecar to run an independent process to obtain the running status of the training task. When any training task terminates, the sidecar of that training task will notify the distributed cache agent to quickly save the checkpoint file stored in memory to the storage system for data persistence. When the training task is rescheduled, the most recent checkpoint data is automatically loaded from memory when training begins. If the most recent checkpoint data is not found in memory, it is then read from the storage system.

[0015] As a further improvement of the present invention, the suspension duration of the training tasks is detected and counted in real time. When multiple training tasks fail, the priority of rescheduling the training tasks is determined based on the suspension duration of the training tasks.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention detects in real time whether there are any anomalies in training tasks and nodes during the training process, determines the cause of the fault, and automatically restarts or reschedules the task according to the cause of the fault, reducing human intervention and reducing the interruption time of training tasks. When restarting or rescheduling, it prioritizes loading the most recent checkpoint file from memory to realize model breakpoint resume training, thereby improving the stability and efficiency of large model training.

[0017] This invention presents a node health detection model based on hardware monitoring metrics, constructed using algorithms such as XGBoost and Random Forest Regression. The model parameters are determined through training, and the health of nodes in the cluster is monitored in real time. This invention utilizes structured and unstructured data from logs and monitoring, based on attention mechanisms and multi-classification algorithms, to construct a multimodal data-based model for fault detection and classification. This model is used to analyze in real time whether faults occur during model training and to identify the causes of these faults. The scheduler is extended using the Kubernetes Scheduling Framework to enable the rescheduling of training tasks.

[0018] This invention is based on Kubernetes sidecar technology. The sidecar container judges the status of the task training process in real time. If the task is interrupted, the sidecar container immediately notifies the distributed cache agent to persist the files in memory to the storage system, thereby speeding up the speed of resuming training after interruption and reducing the possibility of file loss. Attached Figure Description

[0019] Figure 1 This is a flowchart of an automatic fault-tolerant method for distributed model training in a large-scale cloud-native scenario, as disclosed in one embodiment of the present invention. Figure 2 This is a schematic diagram of the deployment of collectors and data collection on each node device in a node cluster disclosed in one embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the process of constructing a fault detection classification model based on multimodal data, as disclosed in one embodiment of the present invention. Figure 4 This is a flowchart illustrating the distributed training process, fault detection and classification, and breakpoint continuation training in one embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The present invention will now be described in further detail with reference to the accompanying drawings: like Figure 1 As shown, this invention provides an automatic fault-tolerance method for distributed model training in large-scale cloud-native scenarios. Based on Kubernetes with custom extensions, it adds a diagnostic module for slow or suspended training of large models, in addition to monitoring the running status of task instances. The priority of task repair is determined by the task suspension duration. Simultaneously, training logs, GPU monitoring metrics, and CUDA running logs are analyzed to achieve fault detection and diagnosis. Based on the fault diagnosis results, fault handling is tiered, from retrying to retraining. When a node fault is detected, an automatic node detection process is introduced to quickly resume training after a fault occurs. To enhance training continuity, the training state is saved to shared memory based on checkpoints and then written to the distributed storage system. When a fault occurs, the training task can load the most recent checkpoint from memory, allowing the training program to resume training from the state of the model and optimizer before the fault. Specifically, it includes: S1. Collect hardware monitoring data from each node in the cluster, configure distributed training tasks, schedule distributed training tasks to each node based on the hardware monitoring data of each node, and enable checkpoint saving on each node. in, Collect hardware monitoring data from each node in the cluster, analyze the hardware monitoring data of each node in the cluster based on the node health detection model, determine whether each node has a fault, and set the faulty node to be uncallable. Configure the number of training instances, the resources required for each instance, and the log output directory, and submit them to Kubernetes; The distributed training scheduler in Kubernetes allocates training tasks to available nodes based on the submitted configuration information.

[0022] Furthermore, a node health detection model is constructed, including: Deploy a hardware detection model in the cluster to obtain hardware monitoring data of each node in real time, including the health status of the accelerator card and hardware monitoring index data. Preprocess the hardware monitoring data, including deduplication, completion, and format standardization. Based on the preprocessed data, the correlation between health status and various hardware monitoring indicators is analyzed. Based on the random forest regression algorithm and the correlation between health status and various hardware monitoring indicators, a node health detection model based on hardware monitoring indicators is constructed.

[0023] Specifically, Node hardware monitoring status detection requires deploying a hardware detection model in the cluster and building a node health detection model based on hardware monitoring metrics. The process is as follows: (1) Data collection: Use accelerator card instructions to obtain real-time monitoring indicators of the accelerator card, including health status and monitoring data (temperature, power consumption, load, frequency, utilization rate, etc.). (2) Data Processing: Duplicate data is removed by checking timestamps. If data is missing, it is filled using the mode method. Simultaneously, the data is aligned to a time series format and converted to a uniform year-month-day-hour-minute-second format. The final data format is as follows: in, Indicates the timestamp of the data generation. Indicates the health status of the acceleration card. This indicates the monitoring metrics of the accelerator card.

[0024] (3) Since health_status can directly represent the health status of the hardware, correlation analysis is performed using the XGBoost algorithm to analyze the correlation between monitoring data and health status indicators. The model is trained using the feature importance assessment method provided by XGBoost. After training, the importance ranking of each data indicator is obtained by calling its importance method, and the top 10 indicators of the accelerator card health with strong correlation are obtained: (4) Construct a node monitoring and detection model based on hardware monitoring indicators using the random forest regression algorithm. Set up N trees in the random forest, and each tree has a number of... Calculate a predicted health value for 10 monitoring indicator samples (x). The final health score is It can be expressed by the following formula: Where N is the number of decision trees in the random forest. It is the predicted value of the i-th tree pair based on the 10 input features x. This represents a 10-dimensional vector.

[0025] The node health monitoring model analyzes the hardware metrics of each node and determines whether a node may have a problem based on the set node health level. To improve the accuracy of the prediction, if a node is determined to be faulty five times consecutively, the node is set to be unschedulable.

[0026] Users configure a distributed training job, including the number of training instances, required resources for each instance, and log output directory, and submit this configuration to Kubernetes. The distributed training scheduler allocates the training task to available nodes based on the submitted configuration. Before execution, the nodes are checked, and checkpointing is enabled. A distributed cache runs on each node to periodically save checkpoints. By default, the three most recent checkpoint files are saved, and hierarchical storage is used, with the most recent checkpoint file stored in memory for fast loading during retraining, and the remaining files stored in the storage system. The training job configuration includes totalWork (total number of tasks), perNode (number of training tasks running per node), groupNode (number of tasks per group), resources, and execution command parameters.

[0027] S2. Monitor the running status of each training task, collect training logs and chip acceleration platform logs, and collect CPU, memory and accelerator card resource index data of each node and training task. in, like Figure 2 As shown, a log collector and a metric exporter are deployed on each node device with an accelerator card, and a data receiver and a task instance status listener, statusEventMonitor, are deployed on the management end of the node cluster. The log collector collects training logs and chip acceleration platform logs during the training task execution on the node it is on, and sends them to the data receiver. The monitoring index collector collects CPU, memory and accelerator card resource index data of the node it is on and the training task in the node. The management terminal periodically obtains the data collected by the monitoring index collector. The task instance status listener uses the list / watch pattern through the API provided by Kubernetes to listen for the running status and events of training tasks, obtains the running status, nodes and instance events of each training task, and records when a status change occurs.

[0028] Specifically, The log collector, logCollector, is responsible for collecting runtime logs from training tasks and logs from various chip acceleration platforms such as CUDA. LogCollector runs on each node, with the corresponding log output directory based on the different configurations of the accelerator cards. The log directory for training instances is obtained through the distributed model training configuration. By collecting file logs in real time, the logs are sent to Elasticsearch on the management end for storage. In Elasticsearch, logs are stored according to node hardware device logs and training log types.

[0029] The metricExporter is responsible for collecting data on the CPU, memory, and accelerator card resources of nodes and training tasks. For each accelerator card type, a corresponding metricExporter runs on the node (e.g., GPU, NPU). Accelerator card metrics collected include card utilization, memory utilization, temperature, power consumption, health status, and errors / violations. The Prometheus management system periodically calls the metricExporter interface on each node to retrieve and save monitoring data. The retrieval period is configurable, with a default of 30 seconds.

[0030] The task instance status listener, `statusEventMonitor`, uses the list / watch pattern provided by Kubernetes API to monitor task status and events, acquiring the running status, nodes, and instance events of each task instance, and recording when status changes occur. Instance running states typically include Pending, Running, Succeeded, Failed, Unknown, Terminating, and ContainerCreating. Node states typically include Ready, NotReady, OutOfDisk, MemoryPressure, DiskPressure, PIDPressure, and NetworkUnavailable.

[0031] S3, such as Figure 3 As shown, based on training logs, chip acceleration platform logs, and CPU, memory, and accelerator card resource metrics data for each node and training task, a fault detection classification model based on multimodal data is constructed by combining supervised and machine learning methods. in, After preprocessing the training logs, chip acceleration platform logs, and CPU, memory, and accelerator card resource metrics data of each node and training task, time series alignment is performed to obtain log and monitoring synchronization data. An attention mechanism is introduced to assign different weights to each feature in the log and monitoring synchronization data to form a new vector. The vector is then enhanced and fused through an encoder layer with a cross-attention mechanism to obtain a fused vector. The features of the fused vector are learned based on the Transformer model to obtain the relationship between log data and monitoring data, and the model is trained based on a multi-classification algorithm to obtain a fault detection classification model.

[0032] Furthermore, The fault detection classification model is trained by calculating and optimizing the model based on the multivariate cross-entropy loss function.

[0033] Specifically, This invention processes two types of data separately, and comprehensively considers the characteristic relationship between logs and monitoring data, including the following steps: (1) Obtain logs and monitoring data from ElasticSearch and Prometheus respectively, and perform data processing.

[0034] The log data processing method involves: first, removing irrelevant characters, punctuation marks, stop words, etc.; then, segmenting the log text into words or sentences for keyword extraction; and finally, categorizing log entries according to tags such as warning, error, and normal. BERT is then used to convert each log entry into vector form. This represents log data (d is the vector dimension of the log data).

[0035] The processing method for monitoring data is as follows: Duplicate data is removed by checking timestamps; missing data is filled using the mode method. Next, the monitoring data is converted into time series features, and the mean, variance, maximum value, and other statistics for each monitoring indicator are calculated. Finally, a Fast Fourier Transform (FFT) is used to convert the monitoring data from the time domain to the frequency domain to capture abnormal patterns in the frequency spectrum. This represents the monitoring data (p is the vector dimension of the monitoring data).

[0036] The two data types are aligned for time series by segmenting the monitoring data into 30-second windows and merging it with log entries within the corresponding time period to achieve time synchronization between the monitoring data and log information.

[0037] (2) By introducing an attention mechanism, each feature of the logs and monitoring data will be assigned a different weight, forming a new vector: And using a cross-attention mechanism, the fusion is further enhanced through the encoder layer, resulting in the final output vector: ,in This represents the dimension of the fused vector.

[0038] (3) Based on the Transformer model, the fused features are learned to obtain the relationship between logs and monitoring data, and the model is trained based on a multi-classification algorithm to determine the fault type. The preliminary model is as follows: in, Represents the weight matrix, connecting the fusion vectors. And the output for each fault type, The bias term is represented, and the default value is 0; f represents the multi-class function, which transforms the output into the probability distribution of each class.

[0039] (4) Since fault detection requires determining the fault type, there are three types of faults (node ​​fault, task itself fault, normal). A smaller value indicates a more accurate result. The multivariate cross-entropy loss function is defined as follows: in, This represents the i-th element in the actual tag. This represents the i-th element of the predicted type vector.

[0040] (5) The final fault classification result output is a vector of length 3, representing the predicted probability of each fault type: (6) Save the feature weights of each vector, the three fault types, and the bias terms obtained after model training to the model file.

[0041] (7) Model training: The fault detection classification model periodically (every 30 seconds) obtains monitoring and log information and outputs them together into the model to obtain the probability of each fault type.

[0042] S4. For each training task that is running, analyze in real time whether there is a fault and the type of fault by combining the running status of each training task with the model training fault detection classification model. If the fault is a node device fault, the training task with the fault will be rescheduled. in, The model training fault detection and classification model obtains training logs of each running training task, chip acceleration platform logs, and CPU, memory, and accelerator card resource index data of the node and training task, and calculates the fault type, including node device fault, task itself fault, and normal.

[0043] If a node device fails, then the failed node is set to unschedulable. The system monitors and tracks the duration of training task suspension in real time. When multiple training tasks fail, the system determines the priority of rescheduling training tasks based on their suspension duration.

[0044] During rescheduling, the distributed training scheduler in Kubernetes reschedules the training tasks based on the number of available nodes in the current cluster and the resource requirements of the failed training tasks.

[0045] Specifically, like Figure 4 As shown, the task is automatically fault-tolerant and supports breakpoint retraining, including: The scheduler is extended using the Kubernetes-based Scheduling Framework. Based on the instance count requirement configured in TrainingJob, the scheduler schedules tasks according to the number of available nodes in the current cluster and the task resource requirements. With the support of the distributed training framework, the scheduler first selects the number of available nodes in the cluster. If the number of available nodes is less than the total number of task instances (totalWork), it needs to further determine the minimum number of task instances that can be started based on the number of instances in each task group. The process for determining the number of nodes is as follows: Based on the configuration requirements for task instance resources and quantity, the actual number of runnable task instances is calculated using the following formula: in, This indicates the number of target task instances for the current task. This indicates the number of training task objectives per group. This indicates the number of task instances that have not been started. Indicates the number of tasks that failed. This represents the actual number of allocatable task instances, rounded down.

[0046] Based on the calculated instance quantity requirement, the extended scheduler reschedules tasks according to the instance quantity requirement and resource requirements.

[0047] Once resources are available or a fault is recovered, the expansion scheduler will continue to dynamically adjust the number of training task instances to ensure that the number of nodes meets the requirements for elastic scaling. This can be represented by the following model, where... Indicates in Further dynamic adjustments are made based on this, such as adding other idle nodes, to ensure that the training task can continue to run without restarting all task instances. .

[0048] S5. After rescheduling the training task for the fault, load the most recent checkpoint data at startup to complete the training task breakpoint continuation and automatic fault tolerance.

[0049] in, Each training task uses a sidecar to run an independent process to obtain the running status of the training task. When any training task terminates, the sidecar of that training task will notify the distributed cache agent to quickly save the checkpoint file stored in memory to the storage system for data persistence. When the training task is rescheduled, the most recent checkpoint data is automatically loaded from memory when training starts. If the most recent checkpoint data is not found in memory, it is read from the storage system. This setting can save the IO overhead of reading from the storage system and improve the speed of retraining.

[0050] Advantages of this invention: This invention enables efficient training of large models. During the training process, it can detect whether there are any anomalies in training tasks and nodes in real time, determine the cause of the failure, and automatically restart or reschedule the task according to the cause of the failure, reducing human intervention and reducing the interruption time of training tasks. When restarting or rescheduling, it prioritizes loading the most recent checkpoint file from memory to realize model breakpoint resume training, thereby improving the stability and efficiency of large model training.

[0051] This invention presents a node health detection model based on hardware monitoring metrics, constructed using algorithms such as XGBoost and Random Forest Regression. The model parameters are determined through training, and the health of nodes in the cluster is monitored in real time. This invention utilizes structured and unstructured data from logs and monitoring, based on attention mechanisms and multi-classification algorithms, to construct a multimodal data-based model for fault detection and classification. This model is used to analyze in real time whether faults occur during model training and to identify the causes of these faults. The scheduler is extended using the Kubernetes Scheduling Framework to enable the rescheduling of training tasks.

[0052] This invention is based on Kubernetes sidecar technology. The sidecar container judges the status of the task training process in real time. If the task is interrupted, the sidecar container immediately notifies the distributed cache agent to persist the files in memory to the storage system, thereby speeding up the speed of resuming training after interruption and reducing the possibility of file loss.

[0053] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A model distributed training automatic fault tolerance method in a large-scale cloud native scenario, characterized in that, Comprise: Collecting hardware monitoring data of each node in the cluster, configuring a distributed training task, and scheduling the distributed training task to each node based on the hardware monitoring data of each node, each node starting checkpoint saving; Listening to the running state of each training task, collecting training logs and chip acceleration platform logs, and collecting CPU, memory, and accelerator card resource index data of each node and training task; Based on the training logs, chip acceleration platform logs, CPU, memory, and accelerator card resource index data of each node and training task, and combining supervised and machine learning methods, a model training fault detection classification model based on multi-modal data is constructed; For each running training task, the running state of the training task and the model training fault detection classification model are combined to analyze whether there is a fault and the fault type in real time, and if it is a node device fault, the training task with the fault is rescheduled; After rescheduling the training task with the fault, load the last checkpoint data when starting, complete the training task breakpoint resume training and automatic fault tolerance.

2. The model distributed training automatic fault-tolerant method in the large-scale cloud native scenario according to claim 1, characterized in that: Collecting hardware monitoring data of each node in the cluster, configuring a distributed training task, and scheduling the distributed training task to each node based on the hardware monitoring data of each node, comprising: Collecting hardware monitoring data of each node in the cluster, detecting and analyzing the hardware monitoring data of each node in the cluster based on a node health detection model, determining whether each node has a fault, and setting the node with a fault as not callable; Configure the number of training instances, instance required resources and log output directory, and submit to Kubernetes; The distributed training scheduler in Kubernetes allocates training tasks to available nodes according to the submitted configuration information.

3. The method of claim 2, wherein: Constructing a node health detection model, comprising: Deploying a hardware detection model in the cluster to obtain real-time hardware monitoring data of each node, including the health status of the accelerator card and hardware monitoring index data; Preprocessing the hardware monitoring data, including deduplication, padding, and format unification processing; Based on the preprocessed data, analyze the correlation between the health status and each hardware monitoring index; Based on the random forest regression algorithm and the correlation between the health status and each hardware monitoring index, a node health detection model based on hardware monitoring index is constructed.

4. The automatic fault-tolerant method for model distributed training in large-scale cloud-native scenarios according to claim 1, characterized in that: Listening to the running state of each training task, collecting training logs and chip acceleration platform logs, and collecting CPU, memory, and accelerator card resource index data of each node and training task; comprising: Deploying a log collector and a monitoring index collector on each node device with an accelerator card, and deploying a data receiver and a task instance state listener on the management end of the node cluster; The log collector collects training logs and chip acceleration platform logs when the training task runs on the node, and sends the data receiver, and the monitoring index collector collects CPU, memory, and accelerator card resource index data of the node and the training task in the node, and the management end periodically acquires data collected by the monitoring index collector; The task instance state listener adopts a list / watch mode through an API interface provided by kubernetes to listen to the running state and events of the training task, and obtains the running state, node and instance events of each training task, and records when the state changes.

5. The automatic fault-tolerant method for model distributed training in large-scale cloud-native scenarios according to claim 1, characterized in that: Based on the training log, chip acceleration platform log, CPU, memory and accelerator resource index data of each node and training task, and combined with supervised and machine learning methods, a model training fault detection classification model is constructed based on multi-modal data; including: The training log, chip acceleration platform log data and CPU, memory and accelerator resource index data of each node and training task are respectively preprocessed and time series aligned to obtain log and monitoring synchronization data; Introducing an attention mechanism, assigning different weights to each feature in the log and monitoring synchronization data to form a new vector, and enhancing and fusing the vector through the encoder layer of the cross-attention mechanism to obtain a fusion vector; Based on the Transformer model, the features of the fusion vector are learned to obtain the relationship between the log data and the monitoring data, and based on the multi-classification algorithm, the model training fault detection classification model is trained to obtain the model training fault detection classification model.

6. The model distributed training automatic fault-tolerant method in the large-scale cloud native scenario according to claim 5, characterized in that: Based on the multi-element cross-entropy loss function, the model training fault detection classification model is calculated and optimized.

7. The method of claim 1, wherein: For each running training task, the running state of each training task and the model training fault detection classification model are combined to analyze whether there is a fault and the fault type in real time, including: The model training fault detection classification model obtains the training log, chip acceleration platform log, CPU, memory and accelerator resource index data of the node and training task of each running training task, and calculates the fault type, including node device fault, task itself fault and normal.

8. The method of claim 1, wherein: If it is a node device fault, the training task with the fault is rescheduled; including: If it is a node device fault, the fault node is set as non-schedulable. Based on the Kubernetes distributed training scheduler, the available node number in the current cluster and the resource requirements of the fault training task are rescheduled.

9. The method of claim 1, wherein: After rescheduling the training task with the fault, the latest checkpoint data is loaded when starting, and the training task breakpoint continues and automatic fault tolerance are completed; including: Each training task uses a sidecar method to run an independent process to obtain the running state of the training task, and when any training task terminal, the sidecar of the training task will notify the distributed cache agent to quickly save the checkpoint file stored in the memory to the storage system for data persistence. When the training task is rescheduled, the latest checkpoint data is automatically loaded from the memory when starting training, and if the latest checkpoint data is not found in the memory, the latest checkpoint data is read from the storage system.

10. The method of claim 1, wherein: The running time of the training task is detected and counted in real time, and when multiple training tasks fail, the priority of training task rescheduling is determined according to the running time of the training task.

Images