Multi-cluster distributed training-oriented disaster recovery drill and performance evaluation method and system

Abstract

The invention relates to the technical field of cloud computing and artificial intelligence, in particular to a disaster recovery drill and performance evaluation system and method for multi-cluster distributed training. In order to solve the problem of disaster recovery and performance evaluation in multi-cluster distributed training, the method comprises the steps of environment analysis and modeling, fault injection and drill control, cross-cluster scheduling and resource monitoring, performance data acquisition and index aggregation, training result evaluation and tuning closed loop and the like. Multi-cluster environment elements are identified, a fault and index mapping relation is constructed, faults are injected by using a fault injection tool, and task scheduling, resource monitoring and performance data acquisition and analysis are realized by means of a multi-cluster management and monitoring tool, so that an adjusting and optimizing strategy is formed, and closed-loop management is realized. According to the method, faults can be dynamically simulated in a multi-cluster environment, the training performance is monitored, the disaster recovery and recovery efficiency is evaluated, the training efficiency and robustness of the system are improved, and the requirements for high availability and high performance in the fields of finance, automatic driving and the like are met.

Description

Technical Field

[0001] The present application relates to the field of cloud computing and artificial intelligence technology, and to a disaster recovery drill and performance evaluation method and system for multi-cluster distributed training. Background Art

[0002] At present, with the rapid development of cloud computing and artificial intelligence technology, distributed training and multi-cluster deployment have become important infrastructure for large-scale model development and application. Especially in the field of large-model training and fine-tuning, the demand for computing resources for training tasks continues to rise, and distributed training can make full use of the computing power advantages of multiple regions and multiple clusters. However, as the scale of models becomes larger and the application scenarios become more diversified, the traditional distributed training architecture is facing unprecedented challenges in dealing with multi-cluster disaster recovery and performance evaluation needs.

[0003] Multi-cluster disaster recovery is not only related to the high availability and data security of the system, but also directly affects the continuity and stability of model training. However, existing technologies usually only focus on load balancing and fault tolerance in a single cluster environment, and lack systematic testing and verification mechanisms for key issues such as cross-cluster failover, network jitter, and data consistency. Once a large-scale failure occurs in a cluster, how to migrate training tasks and quickly restore the model status in the shortest time becomes a difficult problem to ensure continuous business operations.

[0004] In terms of performance evaluation, traditional practices focus on basic indicators such as training duration and accuracy, while ignoring network bandwidth, cross-region latency, storage redundancy efficiency, and resource utilization changes during training operations during fault drills in a multi-cluster environment. For actual production scenarios (such as financial intelligent customer service, autonomous driving visual recognition, and medical image-assisted diagnosis), these dimensions are often equally important: whether the system can maintain training efficiency when large-scale failures or network partitions occur, how to schedule resources between multiple clusters, and how to ensure the continuity and performance stability of the model are all key factors that determine the quality of model development and deployment.

[0005] However, the traditional distributed training evaluation process is often a "one-time test in a fixed environment" and lacks dynamic monitoring and closed-loop feedback for fault injection and disaster drills. Engineers need to manually compare the changes in training performance before and after the failure, and speculate on the root cause of the bottleneck and potential optimization direction based on this, resulting in insufficient observability and optimizability of the multi-cluster training system, a long tuning cycle, and difficulty meeting the current business needs of high-speed iteration and high reliability.

[0006] As enterprises' reliance on multi-cluster intelligent applications continues to increase, the coverage of multi-cluster disaster recovery drills and performance evaluations has gradually expanded to multiple levels such as network, storage, scheduling, and model optimization. Traditional methods are difficult to balance resource efficiency and evaluation accuracy, and the evaluation cost rises exponentially with the scale of the model, hindering the rapid iteration and innovation of distributed training. On the other hand, the requirements for security compliance and service level agreements (SLAs) are becoming increasingly stringent, and a more transparent and comprehensive disaster recovery and performance evaluation system is also needed to meet audit and regulatory needs. In view of this, the present invention proposes a distributed training evaluation method that combines disaster recovery drills with multi-dimensional performance testing, integrating multi-cluster management, storage strategies, chaos injection, and automated orchestration, which becomes the key to ensuring the continuity of model training and accelerating performance optimization.

[0007] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0008] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0009] The disclosed embodiments provide a disaster recovery drill and performance evaluation method and system for multi-cluster distributed training. By automatically performing fault simulation, performance collection, and indicator analysis in a large-scale distributed training environment, real-time monitoring and closed-loop tuning of the model training process are achieved, providing efficient and reliable technical support for the development and deployment of multi-cluster distributed AI systems.

[0010] In some embodiments, the method comprises:

[0011] S1. Environmental analysis and modeling. Identify the network topology, storage layer redundancy strategy, and distributed mode of the multi-cluster environment, and build a mapping relationship between injectable fault types and training indicators;

[0012] S2. Fault simulation and drill control. Use the fault simulation module to inject different levels of fault scenarios during multi-cluster training, control the fault range and time window, and record the real-time status of training operations in each cluster;

[0013] S3, cross-cluster task scheduling and resource monitoring. When a cluster anomaly is detected, the multi-cluster management module automatically switches the training task to the backup node; the monitoring system is used to collect resource indicators of different clusters and track key training events;

[0014] S4. Performance data collection and index aggregation. Collect training performance indicators and model indicators before, during and after failure recovery according to unified standards, and aggregate them into central storage;

[0015] S5. Training result evaluation and closed-loop tuning. Use machine learning to analyze training performance, disaster recovery effects, and resource utilization based on aggregated indicators to form a reusable disaster recovery drill and performance optimization strategy.

[0016] Preferably, the identification of the network topology of the multi-cluster environment, the storage layer redundancy strategy, the distributed mode of the training framework, and the construction of a mapping relationship between the injectable fault type and the training indicator specifically includes:

[0017] Leverage cloud service provider network analysis tools to identify network paths, latency, and bandwidth between different clusters;

[0018] Evaluate and select distributed storage systems suitable for multi-cluster environments and design storage redundancy strategies;

[0019] For the selected distributed training framework, analyze its communication mode, determine the framework's sensitivity to network bandwidth, latency, and storage I / O, and focus on key performance indicators based on business needs;

[0020] Define the types of faults that can be injected, clarify the impact of these faults on the training process, determine the association between key training indicators and various types of faults, establish a mapping relationship model and verify its accuracy. The key training indicators are training throughput, training interruption rate, Checkpoint success rate, and resource utilization. The mapping relationship table is shown in the attached figure. Figure 4 shown.

[0021] Preferably, the fault simulation module injects different levels of fault types during multi-cluster training through a fault injection tool, controls the fault range and time window, and records the real-time status of the training operation in each cluster, specifically including:

[0022] Deploy the fault injection tool Chaos Mesh to each cluster, write fault injection configuration files, and integrate the fault injection process into the automated drill process;

[0023] Set the fault scope and intensity of each drill according to the environmental model, use the parameterization function of the fault injection tool to dynamically adjust the fault scale and duration, and monitor the fault impact scope in real time;

[0024] Design a multi-stage fault injection strategy, use an event-triggered mechanism to inject faults at critical stages of training, and verify the system's responsiveness and recovery effect through staged fault injection.

[0025] Preferably, the multi-cluster management module includes a cluster health monitoring unit and a fault switching and task migration unit. The cluster health monitoring unit collects the operating indicators of each cluster node and network in real time and stores them in a database; the fault switching and task migration unit uses the multi-cluster management tool Karmada to perform fault switching and cross-cluster load migration, specifically including:

[0026] When a cluster fails, the multi-cluster management tool Karmada automatically migrates training tasks to backup nodes based on health checks and priority policies to ensure that storage can be accessed across clusters. It also evaluates the system's disaster recovery capabilities by observing the training job migration delay and resource reallocation time indicators in conjunction with failure scenarios.

[0027] Preferably, the machine learning adopts a bidirectional LSTM model, and the training data includes training throughput, training interruption rate, GPU utilization, GPU utilization, Checkpoint saving success rate, and time; the tuning strategy categories include: no tuning required, expanding computing nodes, extending Checkpoint intervals, optimizing network parameters, rescheduling tasks, and adjusting storage configuration.

[0028] Reusable disaster recovery drills and performance optimization strategies, including adding cluster nodes, adjusting storage configuration, optimizing checkpoint strategies, and using custom scripts to automate tuning steps.

[0029] A disaster recovery drill and performance evaluation system for multi-cluster distributed training, including:

[0030] The environment analysis and modeling module is used to identify the network topology of the multi-cluster environment, the storage layer redundancy strategy, the distributed mode of the training framework, and build a mapping relationship between the injectable fault types and the training indicators;

[0031] The fault injection and drill control module is used to inject different levels of fault scenarios during multi-cluster training through the fault injection tool Chaos Mesh, control the fault scope and time window, and record the real-time status of training operations in each cluster;

[0032] Cross-cluster scheduling and resource monitoring module, which is used to perform failover and cross-cluster load migration with the help of the multi-cluster management tool Karmada, collect resource indicators of different clusters and track key training events using the monitoring system;

[0033] Performance data collection and index aggregation module, which is used to collect training performance indicators and model indicators before, during and after fault recovery according to unified standards, and summarize them in central storage;

[0034] The training result evaluation and tuning closed-loop module is used to analyze the training performance, disaster recovery effect, and resource utilization based on aggregated indicators. If problems are found, the tuning strategy is triggered to form a reusable disaster recovery drill and performance optimization strategy.

[0035] Preferably, the environmental analysis and modeling module includes:

[0036] A network topology analysis unit, which uses the network analysis tools of cloud service providers to identify network paths, delays, and bandwidths between different clusters, and to build a multi-cluster network topology map and display it in layers;

[0037] The storage strategy design unit is used to evaluate and select a distributed storage system suitable for a multi-cluster environment and design a storage redundancy strategy; (1) Replication: In a distributed system, data is completely replicated to multiple independent nodes or data centers; (2) Erasure coding: Data is divided into multiple data blocks and several check blocks are generated so that the original data can still be reconstructed when some blocks are lost; (3) Hybrid redundancy strategy: A combination of replication and erasure coding is used; (4) Geographic redundancy: Data is replicated across data centers in multiple geographical locations;

[0038] The training framework analysis unit is used to analyze the communication mode of the selected distributed training framework, determine the framework's sensitivity to network bandwidth, latency, and storage I / O, and focus on key performance indicators based on business needs;

[0039] The fault indicator mapping unit is used to define the types of faults that can be injected, clarify the impact of these faults on the training process, determine the association between key training indicators and various types of faults, establish a mapping relationship model and verify its accuracy.

[0040] Preferably, the fault injection and drill control module includes:

[0041] Injection configuration unit, used to deploy the fault injection tool Chaos Mesh to each cluster, write fault injection scripts, and use the tool's API or command line interface to integrate the fault injection process into the automated drill process;

[0042] The fault adjustment unit is used to set the fault scope and intensity of each drill according to the environmental model, dynamically adjust the fault scale and duration using the parameterization function of the fault injection tool, and monitor the fault impact scope in real time;

[0043] The trigger control unit is used to design a multi-stage fault injection strategy, use an event trigger mechanism to inject faults at critical stages of training, and verify the system's responsiveness and recovery effect through staged fault injection.

[0044] Preferably, the cross-cluster scheduling and resource monitoring module includes:

[0045] Failover unit: When a cluster fails, the multi-cluster management tool Karmada automatically migrates training tasks to standby nodes based on health checks and priority policies. It uses ResourceBinding and PropagationPolicy to keep training job resources consistent across multiple clusters and trigger failover.

[0046] The load migration evaluation unit is used to ensure that storage can be accessed across clusters. It also evaluates the system's disaster recovery capabilities by observing indicators such as the training job migration latency and resource reallocation time in the context of failure scenarios.

[0047] Resource monitoring unit, which is used to deploy Prometheus nodes in each cluster, use Grafana to uniformly visualize the key resource utilization of each cluster, aggregate multi-cluster monitoring data through Thanos components, and perform additional point monitoring and log collection for key training events;

[0048] Preferably, the training result evaluation and tuning closed-loop module includes:

[0049] The tuning execution unit is used to formulate tuning strategies based on the model prediction results, including adding cluster nodes, adjusting storage configuration, optimizing Checkpoint strategies, and automatically executing tuning steps by writing custom scripts;

[0050] The embodiment of the present disclosure provides a method and system for disaster recovery drill and performance evaluation for multi-cluster distributed training, which can achieve the following technical effects:

[0051] The core of the present invention is to deeply integrate the multi-cluster disaster recovery strategy with the training performance evaluation, while ensuring that the core business is not interrupted, comprehensively quantifying the changes in training indicators before and after the failure, and forming a closed-loop optimization mechanism based on monitoring data and fault recovery results, gradually improving the system's training efficiency and robustness in a multi-cluster environment. This method can dynamically simulate failures in a multi-cluster environment, monitor training performance in real time, accurately evaluate disaster recovery and recovery efficiency, and achieve closed-loop management of evaluation results and system tuning. In fields such as finance and autonomous driving that have extremely high requirements for high availability and high performance, the present invention can significantly improve the resilience and work efficiency of distributed training systems, ensuring that the training process can still maximize core indicators and recover quickly when a failure occurs.

[0052] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0054] Figure 1 It is a schematic flow chart of the method of the present invention;

[0055] Figure 2 It is a schematic diagram of a specific flow of the method provided by an embodiment of the present disclosure;

[0056] Figure 3 It is a schematic diagram of the device structure provided by an embodiment of the present disclosure.

[0057] Figure 4 It is a schematic diagram of the mapping relationship of the present invention. DETAILED DESCRIPTION

[0058] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0059] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so as to describe the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0060] Unless otherwise stated, the term "plurality" means two or more.

[0061] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0062] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0063] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0064] A disaster recovery drill and performance evaluation method for multi-cluster distributed training, including:

[0065] S1: Identify the network topology, storage layer redundancy strategy, and distributed mode of the multi-cluster environment, and build a mapping relationship between injectable fault types and training indicators.

[0066] S2: Use the fault injection tool to inject different levels of fault scenarios during multi-cluster training, control the fault scope and time window, and record the real-time status of training operations in each cluster.

[0067] S3: Use the multi-cluster management tool Karmada to perform failover and cross-cluster load migration, and use the monitoring system to collect resource indicators of different clusters and track key training events.

[0068] S4: Collect training performance indicators and model indicators before, during, and after fault recovery according to unified standards, and summarize them in central storage.

[0069] S5: Based on the aggregated indicators, the training performance, disaster recovery effect, and resource utilization are analyzed. If problems are found, the tuning strategy is triggered to form a reusable disaster recovery drill and performance optimization strategy.

[0070] As a refinement of the above embodiment, the identification of the network topology of the multi-cluster environment, the storage layer redundancy strategy, the distributed mode of the training framework, and the construction of a mapping relationship between the injectable fault type and the training indicator specifically include:

[0071] Use the cloud service provider's network analysis tools to identify network paths, latency, and bandwidth between different clusters, build a multi-cluster network topology diagram, and display it in layers;

[0072] Evaluate and select distributed storage systems suitable for multi-cluster environments and design storage redundancy strategies;

[0073] For the selected distributed training framework, analyze its communication mode, determine the framework's sensitivity to network bandwidth, latency, and storage I / O, and focus on key performance indicators based on business needs;

[0074] Define the types of faults that can be injected, clarify the impact of these faults on the training process, determine the association between key training indicators and various types of faults, establish a mapping relationship model and verify its accuracy.

[0075] As a refinement of the above embodiment, the fault injection tool is used to inject different levels of fault scenarios during multi-cluster training, control the fault range and time window, and record the real-time status of the training operation in each cluster, specifically including:

[0076] Deploy the fault injection tool Chaos Mesh to each cluster, write fault injection scripts, and use the tool's API or command line interface to integrate the fault injection process into the automated drill process;

[0077] Set the fault scope and intensity of each drill according to the environmental model, use the parameterization function of the fault injection tool to dynamically adjust the fault scale and duration, and monitor the fault impact scope in real time;

[0078] Design a multi-stage fault injection strategy, use an event-triggered mechanism to inject faults at critical stages of training, and verify the system's responsiveness and recovery effect through staged fault injection.

[0079] As a refinement of the above embodiment, the multi-cluster management tool Karmada is used to perform failover and cross-cluster load migration, and the monitoring system is used to collect resource indicators of different clusters and track key training events, which specifically includes:

[0080] When a cluster fails, the multi-cluster management tool Karmada automatically migrates training tasks to standby nodes based on health checks and priority policies. It uses ResourceBinding and PropagationPolicy to keep training job resources consistent across multiple clusters and trigger failover.

[0081] Ensure that storage is accessible across clusters, and use failure scenarios to observe training job migration latency and resource reallocation time indicators to evaluate system disaster recovery capabilities.

[0082] Deploy Prometheus nodes in each cluster, use Grafana to uniformly visualize the key resource utilization of each cluster, aggregate multi-cluster monitoring data through Thanos components, and perform additional point monitoring and log collection for key training events;

[0083] Set up Grafana alert rules to display the changing trends of various indicators during the drill on the dashboard.

[0084] As a refinement of the above embodiment, the use of aggregated indicators to analyze training performance, disaster recovery effect and resource utilization, triggering a tuning strategy if a problem is found, and forming a reusable disaster recovery drill and performance optimization strategy, specifically includes:

[0085] Based on the model prediction results, formulate a tuning strategy, including adding cluster nodes, adjusting storage configuration, optimizing Checkpoint strategy, and automatically executing the tuning steps by writing custom scripts;

[0086] Establish an archiving system for drill results, regularly review and analyze historical drill data, and build a knowledge base within the enterprise.

[0087] Specifically, the machine learning adopts a bidirectional LSTM model, which specifically includes: 1) data collection and preprocessing

[0088] The training data collection: training throughput, training interruption rate, GPU utilization, GPU utilization, Checkpoint saving success rate, time. The tuning strategy category is defined as follows:

[0089] 0: No tuning required (system status is normal)

[0090] 1: Expand computing nodes

[0091] 2: Extend the Checkpoint interval

[0092] 3: Optimize network parameters (such as reducing latency and adjusting bandwidth allocation)

[0093] 4: Reschedule tasks (load balancing adjustment)

[0094] 5: Adjust storage configuration (optimize I / O)

[0095] The above categories are defined as One-Hot encoding.

[0096] The preprocessing steps are: Feature engineering: extracting time series features, using data with 5 steps. Standardization: normalizing heterogeneous data.

[0097] 2) Timing model selection and training

[0098] Learning model: A bidirectional LSTM model based on self-attention mechanism is adopted to capture the temporal dependencies of performance indicators (e.g., the cumulative effect of delay after a failure occurs).

[0099] The training process: divide the data set into a training set (historical exercise data) and a test set (real-time monitoring data). Use a genetic algorithm that simulates the natural evolution process to optimize the model hyperparameters. This algorithm can search the hyperparameter space more comprehensively and converge quickly to ensure the generalization ability of the model. Finally, the loss function weighted mean square error is designed to improve the fitting efficiency, focusing on the model accuracy. At the same time, in order to improve the robustness of the model, a normal distribution function is used to generate simulated data of 1 / 3 the size of the original data to expand the amount of original data and assist the model in better learning and training.

[0100] A disaster recovery drill and performance evaluation system for multi-cluster distributed training, including:

[0101] The environment analysis and modeling module is used to identify the network topology of the multi-cluster environment, the storage layer redundancy strategy, the distributed mode of the training framework, and build a mapping relationship between the injectable fault types and the training indicators;

[0102] The fault injection and drill control module is used to inject different levels of fault scenarios during multi-cluster training through the fault injection tool Chaos Mesh, control the fault scope and time window, and record the real-time status of training operations in each cluster;

[0103] Cross-cluster scheduling and resource monitoring module, which is used to perform failover and cross-cluster load migration with the help of the multi-cluster management tool Karmada, collect resource indicators of different clusters and track key training events using the monitoring system;

[0104] Performance data collection and index aggregation module, which is used to collect training performance indicators and model indicators before, during and after fault recovery according to unified standards, and summarize them in central storage;

[0105] The training result evaluation and tuning closed-loop module is used to analyze the training performance, disaster recovery effect, and resource utilization based on aggregated indicators. If problems are found, the tuning strategy is triggered to form a reusable disaster recovery drill and performance optimization strategy.

[0106] As a refinement of the above embodiment, the environment analysis and modeling module includes:

[0107] A network topology analysis unit, which uses the network analysis tools of cloud service providers to identify network paths, delays, and bandwidths between different clusters, and to build a multi-cluster network topology map and display it in layers;

[0108] The storage strategy design unit is used to evaluate and select a distributed storage system suitable for a multi-cluster environment and design a storage redundancy strategy. The storage redundancy strategy in this method mainly includes: (1) Replication: In a distributed system, data is completely replicated to multiple independent nodes or data centers. For example, in a three-copy strategy, data is stored on three nodes to ensure that data can be recovered in the event of a single point of failure. Replication usually has a low computational overhead, but a high storage cost. (2) Erasure Coding: Data is divided into multiple data blocks and several check blocks are generated. When some blocks are lost, the original data can still be reconstructed. For example, using Reed-Solomon code, when k data blocks and m check blocks are saved, as long as any k blocks of data are available, the data can be recovered. The advantage is low storage overhead, but the disadvantage is high computational complexity and bandwidth consumption during data recovery. (3) Hybrid redundancy strategy: Replication and erasure coding are used in combination. Multiple copies are used for some data or scenarios, while erasure coding is used in other cases, so as to achieve a balance between performance, reliability and storage cost. For example, it is often used in distributed object storage systems, where different strategies can be selected based on data popularity. (4) Geo-replication: Data is replicated across data centers in multiple geographic locations, which not only improves fault tolerance but also reduces the risk of regional disasters. It is commonly used in cloud storage services to ensure high availability and low latency when global users access data;

[0109] The training framework analysis unit is used to analyze the communication mode of the selected distributed training framework, determine the framework's sensitivity to network bandwidth, latency, and storage I / O, and focus on key performance indicators based on business needs;

[0110] The fault indicator mapping unit is used to define the types of faults that can be injected, clarify the impact of these faults on the training process, determine the association between key training indicators and various types of faults, establish a mapping relationship model and verify its accuracy.

[0111] As a refinement of the above embodiment, the fault injection and drill control module includes:

[0112] Injection configuration unit, used to deploy the fault injection tool Chaos Mesh to each cluster, write fault injection scripts, and use the tool's API or command line interface to integrate the fault injection process into the automated drill process;

[0113] The fault adjustment unit is used to set the fault scope and intensity of each drill according to the environmental model, dynamically adjust the fault scale and duration using the parameterization function of the fault injection tool, and monitor the fault impact scope in real time;

[0114] The trigger control unit is used to design a multi-stage fault injection strategy, use an event trigger mechanism to inject faults at critical stages of training, and verify the system's responsiveness and recovery effect through staged fault injection.

[0115] As a refinement of the above embodiment, the cross-cluster scheduling and resource monitoring module includes:

[0116] Failover unit: When a cluster fails, the multi-cluster management tool Karmada automatically migrates training tasks to standby nodes based on health checks and priority policies. It uses ResourceBinding and PropagationPolicy to keep training job resources consistent across multiple clusters and trigger failover.

[0117] The load migration evaluation unit is used to ensure that storage can be accessed across clusters. It also evaluates the system's disaster recovery capabilities by observing indicators such as the training job migration latency and resource reallocation time in the context of failure scenarios.

[0118] Resource monitoring unit, which is used to deploy Prometheus nodes in each cluster, use Grafana to uniformly visualize the key resource utilization of each cluster, aggregate multi-cluster monitoring data through Thanos components, and perform additional point monitoring and log collection for key training events;

[0119] As a refinement of the above embodiment, the training result evaluation and tuning closed-loop module includes:

[0120] The tuning execution unit is used to formulate tuning strategies based on the model prediction results, including adding cluster nodes, adjusting storage configuration, optimizing Checkpoint strategies, and automatically executing tuning steps by writing custom scripts;

[0121] The archiving improvement unit is used to establish an archiving system for drill results, regularly review and analyze historical drill data, and establish a knowledge base within the enterprise.

[0122] Example 3

[0123] Financial intelligent customer service model training scenario

[0124] 1. Environmental analysis and modeling: The training of financial intelligent customer service models has extremely high requirements for data accuracy and service continuity. In the multi-cluster training environment of a financial institution, the network analysis tools of the cloud service provider were used to identify that the clusters were distributed in different regional data centers, with network delays ranging from 50-200ms and bandwidths ranging from 10-100Gbps. Ceph was selected as the distributed storage system, and a three-copy storage redundancy strategy (replication) was adopted to ensure data reliability. TensorFlow was selected as the training framework, and its communication mode is parameter server mode, which is sensitive to network bandwidth and latency. Network partition, node downtime, and Pod Kill are defined as injectable fault types, and a mapping relationship model with accuracy, throughput, latency, and resource utilization is constructed. For example, network partition may cause a significant increase in training latency and a decrease in throughput, which in turn affects accuracy.

[0125] 2. Fault injection and drill control: Deploy Chaos Mesh to each cluster. Write a fault injection script and set a drill to inject a network fault with a packet loss rate of 30% for 10 minutes into one of the clusters during the parameter synchronization phase of model training. Use Kubeflow Pipelines to integrate the fault injection process and achieve automatic triggering. During the drill, use the monitoring function of Chaos Mesh to view the scope of the fault in real time to ensure that it does not exceed expectations and avoid excessive interference with other normal businesses.

[0126] 3. Cross-cluster scheduling and resource monitoring: Karmada, a multi-cluster management tool, migrates training tasks to the backup cluster within 1 minute when a faulty cluster is detected based on the preset health check strategy. With the help of Prometheus and Grafana monitoring systems, it was observed that during the task migration process, the CPU utilization of the backup cluster rose to 80% in a short period of time, the GPU utilization reached 70%, and the network bandwidth utilization increased to 60%. At the same time, indicators such as the number of task restarts and the time taken to switch clusters were recorded for subsequent analysis.

[0127] 4. Performance data collection and index aggregation: According to the established standards, the training performance indicators before, during and after the failure are collected. It is found that when the failure occurs, the model accuracy drops by 5%, the throughput decreases by 30%, and the latency increases by 5 times. The data is aggregated into the data warehouse for comparative analysis.

[0128] 5. Training result evaluation and tuning closed loop: Grafana was used to visualize indicator changes, and log analysis was combined to find that some training tasks failed due to excessive resource utilization during the failure. Based on this, a tuning strategy was formulated to add 2 GPU nodes to the backup cluster, optimize the I / O configuration of the storage system, and shorten the Checkpoint interval from 30 minutes to 15 minutes. The Ansible automation tool was used to perform the tuning operation, and then the drill was conducted again. The comparison showed that the model training performance was significantly improved, the accuracy returned to normal, the throughput increased by 20%, and the latency decreased by 40%. The experience and optimization strategies of this drill were recorded in the enterprise knowledge base to provide reference for similar scenarios in the future.

[0129] Combination Figure 3 As shown, the embodiment of the present disclosure provides a disaster recovery drill and performance evaluation device 300 for multi-cluster distributed training, including a processor (processor) 304 and a memory (memory) 301. Optionally, the device may also include a communication interface (Communication Interface) 302 and a bus 303. Among them, the processor 304, the communication interface 302, and the memory 301 can communicate with each other through the bus 303. The communication interface 302 can be used for information transmission. The processor 304 can call the logical instructions in the memory 301 to execute the disaster recovery drill and performance evaluation method for multi-cluster distributed training of the above embodiment.

[0130] In addition, the logic instructions in the memory 301 described above can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0131] The memory 301 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 304 executes the functional application and data processing by running the program instructions / modules stored in the memory 301, that is, the disaster recovery drill and performance evaluation method for multi-cluster distributed training in the above embodiment is implemented.

[0132] The memory 301 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 301 may include a high-speed random access memory and may also include a non-volatile memory.

[0133] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned disaster recovery drill and performance evaluation method for multi-cluster distributed training.

[0134] The computer-readable storage medium mentioned above may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0135] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, or a transient storage medium.

[0136] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0137] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. The technicians may clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0138] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0139] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A disaster recovery drill and performance evaluation method for multi-cluster distributed training, characterized in that: include: Identify the network topology, storage layer redundancy strategy, and distributed mode of the training framework in a multi-cluster environment, and build a mapping relationship between injectable fault types and training metrics; Use the fault simulation module to inject different levels of fault scenarios during multi-cluster training, control the fault scope and time window, and record the real-time status of training operations in each cluster; When a cluster anomaly is detected, the multi-cluster management module automatically switches the training task to the backup node; Use the monitoring system to collect resource indicators of different clusters and track key training events; Collect training performance indicators and model indicators before, during, and after failure recovery according to unified standards, and summarize them in central storage; Machine learning is used to analyze training performance, disaster recovery effects, and resource utilization based on aggregated indicators to form reusable disaster recovery drills and performance optimization strategies.

2. The method for disaster recovery drill and performance evaluation for multi-cluster distributed training according to claim 1, characterized in that: The identification of the network topology of the multi-cluster environment, the storage layer redundancy strategy, and the distributed mode of the training framework, and the construction of a mapping relationship between the injectable fault type and the training indicator specifically include: Leverage cloud service provider network analysis tools to identify network paths, latency, and bandwidth between different clusters; Evaluate and select distributed storage systems suitable for multi-cluster environments and design storage redundancy strategies; For the selected distributed training framework, analyze its communication mode, determine the framework's sensitivity to network bandwidth, latency, and storage I / O, and focus on key performance indicators based on business needs; Define the types of faults that can be injected, clarify the impact of these faults on the training process, determine the association between key training indicators and various types of faults, establish a mapping relationship model and verify its accuracy. The key training indicators include training throughput, training interruption rate, CPU utilization, GPU utilization, Checkpoint save success rate, and time.

3. The method for disaster recovery drill and performance evaluation for multi-cluster distributed training according to claim 1, characterized in that: The fault injection tool injects different levels of fault scenarios during multi-cluster training, controls the fault scope and time window, and records the real-time status of training operations in each cluster, including: Deploy the fault injection tool Chaos Mesh to each cluster, write fault injection scripts, and integrate the fault injection process into the automated drill process; Set the fault scope and intensity of each drill according to the environmental model, use the parameterization function of the fault injection tool to dynamically adjust the fault scale and duration, and monitor the fault impact scope in real time; Design a multi-stage fault injection strategy, use an event-triggered mechanism to inject faults at critical stages of training, and verify the system's responsiveness and recovery effect through staged fault injection.

4. The method for disaster recovery drill and performance evaluation for multi-cluster distributed training according to claim 1, characterized in that: The multi-cluster management module includes a cluster health monitoring unit and a fault switching and task migration unit; The cluster health monitoring unit collects the operating indicators of each cluster node and network in real time and stores them in the database; The failover and task migration unit uses the multi-cluster management tool Karmada to perform failover and cross-cluster load migration, including: When a cluster fails, the multi-cluster management tool Karmada automatically migrates training tasks to backup nodes based on health checks and priority policies to ensure that storage can be accessed across clusters. It also evaluates the system's disaster recovery capabilities by observing the training job migration delay and resource reallocation time indicators in conjunction with failure scenarios.

5. The method for disaster recovery drill and performance evaluation for multi-cluster distributed training according to claim 1, characterized in that: The machine learning adopts a bidirectional LSTM model, and the training data includes training throughput, training interruption rate, CPU utilization, GPU utilization, Checkpoint saving success rate, and time; The tuning strategy categories include: no tuning required, expanding computing nodes, extending the checkpoint interval, optimizing network parameters, rescheduling tasks, and adjusting storage configuration; Reusable disaster recovery drills and performance optimization strategies, including adding cluster nodes, adjusting storage configuration, optimizing checkpoint strategies, and using custom scripts to automate tuning steps.

6. A disaster recovery drill and performance evaluation system for multi-cluster distributed training using the method described in any one of claims 1 to 5, characterized in that: include: The environment analysis and modeling module is used to identify the network topology of the multi-cluster environment, the storage layer redundancy strategy, the distributed mode of the training framework, and build a mapping relationship between the injectable fault types and the training indicators; The fault injection and drill control module is used to inject different levels of fault scenarios during multi-cluster training through the fault injection tool Chaos Mesh, control the fault scope and time window, and record the real-time status of training operations in each cluster; Cross-cluster scheduling and resource monitoring module, which is used to perform failover and cross-cluster load migration with the help of the multi-cluster management tool Karmada, collect resource indicators of different clusters and track key training events using the monitoring system; Performance data collection and index aggregation module, which is used to collect training performance indicators and model indicators before, during and after fault recovery according to unified standards, and summarize them in central storage; The training result evaluation and tuning closed-loop module is used to analyze the training performance, disaster recovery effect, and resource utilization based on aggregated indicators. If problems are found, the tuning strategy is triggered to form a reusable disaster recovery drill and performance optimization strategy.

7. The disaster recovery drill and performance evaluation system for multi-cluster distributed training according to claim 6, characterized in that: The environmental analysis and modeling module includes: A network topology analysis unit, which uses the network analysis tools of cloud service providers to identify network paths, delays, and bandwidths between different clusters, and to build a multi-cluster network topology map and display it in layers; The storage strategy design unit is used to evaluate and select a distributed storage system suitable for a multi-cluster environment and design a storage redundancy strategy, wherein the storage redundancy strategy includes: (1) replication: in a distributed system, the data is completely replicated to multiple independent nodes or data centers; (2) erasure coding: the data is divided into multiple data blocks and a number of check blocks are generated, so that the original data can still be reconstructed when some blocks are lost; (3) hybrid redundancy strategy: replication and erasure coding are used in combination; (4) geographic redundancy: data is replicated across data centers in multiple geographical locations; The training framework analysis unit is used to analyze the communication mode of the selected distributed training framework, determine the framework's sensitivity to network bandwidth, latency, and storage I / O, and focus on key performance indicators based on business needs; The fault indicator mapping unit is used to define the types of faults that can be injected, clarify the impact of these faults on the training process, determine the association between key training indicators and various types of faults, establish a mapping relationship model and verify its accuracy.

8. The disaster recovery drill and performance evaluation system for multi-cluster distributed training according to claim 6, characterized in that: The fault injection and drill control module includes: Injection configuration unit, used to deploy the fault injection tool Chaos Mesh to each cluster, write fault injection scripts, and use the tool's API or command line interface to integrate the fault injection process into the automated drill process; The fault adjustment unit is used to set the fault scope and intensity of each drill according to the environmental model, dynamically adjust the fault scale and duration using the parameterization function of the fault injection tool, and monitor the fault impact scope in real time; The trigger control unit is used to design a multi-stage fault injection strategy, use an event trigger mechanism to inject faults at key training stages, and verify the system's responsiveness and recovery effects through staged fault injection.

9. The disaster recovery drill and performance evaluation system for multi-cluster distributed training according to claim 6, characterized in that: The cross-cluster scheduling and resource monitoring module includes: Failover unit: When a cluster fails, the multi-cluster management tool Karmada automatically migrates training tasks to standby nodes based on health checks and priority policies. It uses ResourceBinding and PropagationPolicy to keep training job resources consistent across multiple clusters and trigger failover. The load migration evaluation unit is used to ensure that storage can be accessed across clusters. It also evaluates the system's disaster recovery capabilities by observing indicators such as the training job migration latency and resource reallocation time in the context of failure scenarios. Resource monitoring unit, which is used to deploy Prometheus nodes in each cluster, use Grafana to uniformly visualize the key resource utilization of each cluster, aggregate multi-cluster monitoring data through Thanos components, and perform additional point monitoring and log collection for key training events; The alarm display unit is used to set Grafana alarm rules and display the changing trends of various indicators during the drill in the dashboard.

10. The disaster recovery drill and performance evaluation system for multi-cluster distributed training according to claim 6, characterized in that: The training result evaluation and tuning closed-loop module includes: The tuning execution unit is used to formulate tuning strategies based on model prediction results, including adding cluster nodes, adjusting storage configuration, optimizing Checkpoint strategies, and automatically executing tuning steps by writing custom scripts.

Images