Disaster recovery method, device, equipment, storage medium and product

Abstract

The application discloses a disaster recovery method and device, equipment, a storage medium and a product, relates to the technical field of cloud computing big data edge computing, and the method comprises the following steps: comparing the allocated resource capacity of an abnormal running unit with the spare resource capacity of each normal running unit after detecting the abnormal running unit; when the spare resource capacity of a plurality of alternative running units in each normal running unit exceeds the allocated resource capacity, predicting the resource use trend of each alternative running unit by using a time sequence big model; determining a target running unit from each alternative running unit according to the resource use trend of each alternative running unit, and restoring the backup file and the business service of the abnormal running unit in the target running unit. The application can improve the efficiency and accuracy of Kubernetes cluster disaster recovery.

Description

Technical Field

[0001] This application relates to the fields of cloud computing, big data, and edge computing technologies, and in particular to disaster recovery methods, devices, equipment, storage media, and products. Background Technology

[0002] With the rapid development of information technology, cloud computing has become an important way for modern enterprises to store and process data. Kubernetes (a container orchestration system), as an open-source container orchestration system, is widely used in cloud services due to its efficient and flexible cluster management capabilities.

[0003] The complexity and dynamism of cloud environments make data protection and disaster recovery key challenges in cloud computing. However, current disaster recovery technologies often lack automation and intelligence, making it difficult to respond quickly to disaster events. Furthermore, they often employ static methods in resource allocation and recovery strategies, which cannot adapt to the dynamic changes in resources in cloud environments, resulting in low recovery efficiency.

[0004] In summary, improving the efficiency and accuracy of Kubernetes cluster disaster recovery has become a pressing technical problem that needs to be solved in this field. Summary of the Invention

[0005] The main purpose of this application is to provide a disaster recovery method, apparatus, device, storage medium, and product, which aims to improve the efficiency and accuracy of disaster recovery for Kubernetes clusters.

[0006] To achieve the above objectives, this application proposes a disaster recovery method, which includes:

[0007] After detecting an abnormal operating unit, the allocated resource capacity of the abnormal operating unit is compared with the available resource capacity of each normally operating unit.

[0008] When the available resource capacity of multiple candidate operating units in each of the normal operating units exceeds the allocated resource capacity, a time-series large model is used to predict the resource usage trend of each of the candidate operating units.

[0009] Based on the resource usage trends of each of the candidate operating units, a target operating unit is determined from the candidate operating units, and the backup files and business services of the abnormal operating unit are restored in the target operating unit.

[0010] In one embodiment, before the step of using a time-series large model to predict the resource usage trends of each of the candidate operating units, the method further includes:

[0011] The historical resource usage data of each running unit is used as the training dataset and input into a pre-trained temporal large model based on the Transformer (a type of decoder) decoder architecture;

[0012] The pre-trained temporal large model was fine-tuned using the LoRa (Low-Rank Adaptation of Large Language Models) low-rank fine-tuning technique.

[0013] The model performance is validated by calculating the mean absolute error of the fine-tuned pre-trained time series large model on the validation dataset, and the model parameters are optimized based on the validation results to obtain the time series large model.

[0014] In one embodiment, the step of using a time-series large model to predict the resource usage trends of each of the candidate operating units includes:

[0015] The resource usage data of each of the candidate running units is input into the time series large model, wherein the resource usage data includes CPU (Central Processing Unit) utilization, memory utilization, network congestion, IO (Input / Output) call saturation and operating system kernel system call saturation;

[0016] Based on the resource usage data, the resource usage trend of each of the candidate operating units is predicted at different future time steps, wherein the future time steps include short-term time steps and long-term time steps.

[0017] In one embodiment, the step of determining the target operating unit from the candidate operating units based on their resource usage trends includes:

[0018] For each of the candidate operating units, the resource usage trend of the candidate operating unit in different time steps is weighted and calculated. The weighting strategy includes assigning a first weight to the resource usage trend in the short time step and assigning a second weight to the resource usage trend in the long time step, wherein the first weight is greater than the second weight.

[0019] Based on the target resource usage trend obtained after weighted calculation, a target operating unit is selected from each of the candidate operating units.

[0020] In one embodiment, before the step of comparing the allocated resource capacity of the abnormal operating unit with the free resource capacity of each normally operating unit after detecting the abnormal operating unit, the method further includes:

[0021] Real-time monitoring of the running status of each running unit in Kubernetes;

[0022] When an abnormality occurs in the operating state, it is detected whether the cluster control agent of the cluster where the abnormality occurs has failed.

[0023] When the cluster control agent fails, the abnormal operating unit is identified as an abnormal cluster, and the normal operating units are each normal cluster in Kubernetes.

[0024] When the cluster control agent is not in failure, the abnormal operating unit is identified as an abnormal node, and the normal operating units are each normal node in the same cluster as the abnormal node.

[0025] In one embodiment, the step of restoring the backup files and business services of the abnormal operating unit in the target operating unit includes:

[0026] Retrieve the backup file of the abnormal operating unit from the preset backup storage area;

[0027] The backup file is restored in the target running unit to rebuild the application data of the abnormal running unit on the target running unit;

[0028] Based on the application data, the business services of the abnormal operating unit are restored in the target operating unit.

[0029] Furthermore, to achieve the above objectives, this application also proposes a disaster recovery device, which includes:

[0030] The capacity comparison module is used to compare the allocated resource capacity of the abnormal operating unit with the free resource capacity of each normally operating unit after detecting an abnormal operating unit.

[0031] The resource usage prediction module is used to predict the resource usage trend of each of the candidate operating units when the available resource capacity of multiple candidate operating units in each of the normal operating units exceeds the allocated resource capacity, using a time series large model.

[0032] The recovery module is used to determine the target running unit from the candidate running units based on the resource usage trends of each candidate running unit, and to restore the backup files and business services of the abnormal running unit in the target running unit.

[0033] In addition, to achieve the above objectives, this application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the disaster recovery method described above.

[0034] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the disaster recovery method described above.

[0035] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the disaster recovery method described above.

[0036] This application proposes a disaster recovery method. After detecting an abnormal operating unit, the allocated resource capacity of the abnormal operating unit is compared with the spare resource capacity of each normally operating unit. When there are multiple candidate operating units in each normally operating unit whose spare resource capacity exceeds the allocated resource capacity, a time-series large model is used to predict the resource usage trend of each candidate operating unit. Based on the resource usage trend of each candidate operating unit, a target operating unit is determined from each candidate operating unit, and the backup files and business services of the abnormal operating unit are restored in the target operating unit.

[0037] In summary, this application automatically compares the allocated resource capacity of the abnormal operating unit with the spare resource capacity of the normally operating unit when an abnormal operating unit is detected. When multiple alternative operating units meet the conditions, it uses a time-series large model to predict the resource usage trend of each alternative operating unit to accurately select the target operating unit. This allows for the rapid recovery of backup files and business services of the abnormal operating unit on the selected target operating unit, reducing human error and service interruption time in the face of cluster disasters and improving the efficiency and accuracy of cluster disaster recovery. Attached Figure Description

[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating an embodiment of the disaster recovery method of this application.

[0041] Figure 2 This is a schematic diagram of the model structure provided in Embodiment 1 of the disaster recovery method of this application;

[0042] Figure 3This is a schematic diagram of the cluster disaster recovery workflow provided in Embodiment 2 of the disaster recovery method of this application;

[0043] Figure 4 This is a schematic diagram of the node fault recovery workflow provided in Embodiment 2 of the disaster recovery method of this application;

[0044] Figure 5 This is a schematic diagram of the cluster disaster recovery implementation process provided in Embodiment 2 of the disaster recovery method of this application;

[0045] Figure 6 This is a schematic diagram of the cluster disaster recovery implementation process provided in Embodiment 2 of the disaster recovery method of this application;

[0046] Figure 7 This is a schematic diagram of the module structure of the disaster recovery device according to an embodiment of this application;

[0047] Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the disaster recovery method in the embodiments of this application.

[0048] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0050] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0051] With the rapid development of information technology, cloud computing has become an important way for modern enterprises to store and process data. Kubernetes, as an open-source container orchestration system, is widely used in cloud services due to its efficient and flexible cluster management capabilities.

[0052] The complexity and dynamism of cloud environments make data protection and disaster recovery key challenges in cloud computing. However, current disaster recovery technologies often lack automation and intelligence, making it difficult to respond quickly to disaster events. Furthermore, they often employ static methods in resource allocation and recovery strategies, which cannot adapt to the dynamic changes in resources in cloud environments, resulting in low recovery efficiency.

[0053] In summary, improving the efficiency and accuracy of Kubernetes cluster disaster recovery has become a pressing technical problem that needs to be solved in this field.

[0054] The main solution of this application embodiment is as follows: after detecting an abnormal operating unit, the allocated resource capacity of the abnormal operating unit is compared with the spare resource capacity of each normal operating unit; when there are multiple candidate operating units in each normal operating unit whose spare resource capacity exceeds the allocated resource capacity, a time series large model is used to predict the resource usage trend of each candidate operating unit; based on the resource usage trend of each candidate operating unit, a target operating unit is determined from each candidate operating unit, and the backup files and business services of the abnormal operating unit are restored in the target operating unit.

[0055] This application provides a solution that, when an abnormal operating unit is detected, automatically compares the allocated resource capacity of the abnormal operating unit with the free resource capacity of the normally operating unit. When multiple alternative operating units meet the conditions, a time-series large model is used to predict the resource usage trend of each alternative operating unit to accurately select the target operating unit. This allows for the rapid recovery of backup files and business services of the abnormal operating unit on the selected target operating unit, reducing human error and service interruption time in the face of cluster disasters and improving the efficiency and accuracy of cluster disaster recovery.

[0056] It should be noted that the execution entity in this embodiment can be a computing service device running Kubernetes, such as a physical server, virtual machine, or computing instance provided by a cloud service provider, or other devices capable of running Kubernetes. The following description uses a Kubernetes service system as an example to illustrate this embodiment and the subsequent embodiments.

[0057] Before describing specific embodiments of the disaster recovery method of this application, we will first introduce existing disaster recovery technologies for Kubernetes.

[0058] Specifically, current disaster recovery technologies primarily rely on regular data backups and manual recovery processes after a disaster. These processes typically include data backup, storage, and manual recovery in the event of data loss or service interruption. This process is not only time-consuming but also susceptible to human error, leading to inefficient recovery and frequent errors. In recent years, some research has begun to explore automated and intelligent disaster recovery technologies. For example, machine learning algorithms are used to predict system states to achieve more efficient resource management and faster recovery. However, these studies often focus on single service or application levels, lacking a comprehensive consideration of the entire Kubernetes cluster.

[0059] Data protection challenges in cloud environments include rapid data growth, diverse data types, and evolving business needs. Enterprises need to ensure data security and availability while meeting compliance requirements. Furthermore, data protection strategies must be able to withstand a variety of threats, including hardware failures, software errors, cyberattacks, and even natural disasters.

[0060] Managing a Kubernetes cluster involves multiple layers, including resource scheduling, service deployment, monitoring, and fault recovery. As the cluster size, the number of clusters, and the number of applications increase, manual cluster management becomes increasingly complex and inefficient, especially in the event of a cluster disaster, making it difficult to restore business operations quickly. While automation tools can provide some assistance, they lack the ability to quickly and intelligently recover from disasters.

[0061] Existing disaster recovery solutions typically lack automation and intelligence, hindering their ability to respond quickly to disaster events. Furthermore, these solutions often employ static methods for resource allocation and recovery strategies, failing to adapt to the dynamic nature of resources in cloud environments. Rapidly and accurately predicting resource demands and selecting the optimal target cluster for recovery during a disaster is a major challenge posed by current technologies.

[0062] Based on this, the embodiments of this application provide a disaster recovery method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the disaster recovery method of this application.

[0063] In this embodiment, the disaster recovery method includes steps S10 to S30:

[0064] Step S10: After detecting an abnormal operating unit, compare the allocated resource capacity of the abnormal operating unit with the free resource capacity of each normally operating unit.

[0065] It should be noted that the running unit in this embodiment refers to a cluster, node, or Pod (the smallest deployment unit in Kubernetes). Kubernetes is an open-source container orchestration system used to automate the deployment, scaling, and management of containerized applications. Kubernetes simplifies application management by organizing application containers into logical units.

[0066] After detecting an anomaly in a Kubernetes runtime unit, the allocated resource capacity of the abnormal runtime unit is compared with the available resource capacity of each normally operating runtime unit in Kubernetes. This ensures that the system can quickly identify alternative runtime units with sufficient resource capacity for subsequent resource allocation and recovery operations. The allocated resource capacity refers to the resource capacity allocated to the abnormal runtime unit during normal operation, including CPU utilization, memory utilization, network congestion, IO call saturation, and operating system kernel system call saturation. The available resource capacity refers to the unused resource capacity of the normally operating runtime unit, which can be used to receive and recover the business services of the abnormal runtime unit.

[0067] Step S20: When the available resource capacity of multiple candidate operating units exceeds the allocated resource capacity in each normal operating unit, a time series large model is used to predict the resource usage trend of each candidate operating unit.

[0068] If the available resource capacity of multiple alternative operating units in the normal operating unit exceeds the allocated resource capacity of the abnormal operating unit, then a time-series large model is used to predict the future resource usage trend of each alternative operating unit, that is, to predict the future resource usage of each alternative operating unit.

[0069] Step S30: Based on the resource usage trends of each candidate running unit, determine the target running unit from among the candidate running units, and restore the backup files and business services of the abnormal running unit in the target running unit.

[0070] Based on the resource usage trends of each candidate operating unit, the target operating unit is determined from the candidate operating units, and the backup files and business services of the abnormal operating unit are automatically restored in the target operating unit. The backup files of the abnormal operating unit contain the configuration information, application data and status information of the unit, which are used to restore the state of the abnormal operating unit in the event of a disaster. The business services refer to the applications and services running in the unit, which need to be restored in the event of a disaster to ensure business continuity.

[0071] In one feasible embodiment, steps A10 to A30 may be included before step S10:

[0072] Step A10: Use the historical resource usage data of each running unit as the training dataset and input it into the pre-trained temporal large model based on the Transformer decoder architecture.

[0073] It should be noted that the Transformer decoder architecture is a neural network architecture based on a self-attention mechanism, which is particularly suitable for processing sequential data. By processing all elements of the input sequence in parallel, the Transformer decoder can capture long-distance dependencies in the sequence, making it suitable for time series prediction tasks. Pre-trained models based on the Transformer decoder architecture can learn the dynamic characteristics of time series through a large amount of historical data and predict resource usage trends at future points in time.

[0074] Historical resource usage data for each running unit in Kubernetes is collected. This data includes key metrics such as CPU utilization, memory utilization, network congestion, I / O call saturation, and operating system kernel system call saturation. This historical resource usage data is organized into a time series format and used as a training dataset to be input into a pre-trained time series model based on the Transformer decoder architecture. The pre-trained time series model learns the dynamic characteristics of the time series through this historical resource usage data in order to accurately predict the future resource usage trends of each running unit.

[0075] Step A20: Use LoRa low-rank fine-tuning technology to fine-tune the pre-trained temporal large model.

[0076] LoRa low-rank fine-tuning is employed to fine-tune a pre-trained time-series model. LoRa low-rank fine-tuning is an efficient model tuning method suitable for optimizing models with limited data. In this case, since the amount of real-time monitoring data collected by Kubernetes is relatively small, LoRa low-rank fine-tuning, by adjusting the model's low-rank parameters, can significantly improve model performance without significantly increasing computational costs. During fine-tuning, a smaller hyperparameter, rank (e.g., rank=4), is chosen to accommodate the limited data volume while maintaining the model's flexibility and adaptability.

[0077] Step A30: Validate the model performance by calculating the mean absolute error of the fine-tuned pre-trained time series large model on the validation dataset, and optimize the model parameters based on the validation results to obtain the time series large model.

[0078] The model performance was validated by calculating the Mean Absolute Error (MAE) of the fine-tuned pre-trained time-series large model on a validation dataset. This validation dataset was collected from a running Kubernetes instance and contained up-to-date resource usage data not used in training. The MAE metric measures the average difference between the model's predictions and actual values; a lower MAE value indicates higher prediction accuracy. Based on the validation results, the model parameters were further optimized to ensure reliable resource usage trend predictions in real-world applications.

[0079] It is worth mentioning that this embodiment uses a pre-trained large-scale temporal model based on a pure Transformer decoder architecture. After fine-tuning using historical time-series datasets collected by a real-time monitoring component, it is used to automatically recover the target node / cluster resource metric prediction of the system. The large-scale temporal model based on the Transformer decoder architecture is particularly suitable for handling metric prediction tasks that require capturing long-term dependencies: using the Self-Attention mechanism, it can capture long-distance dependencies in the sequence and process all elements of the input sequence in parallel; compared with LSTM (Long Short-Term Memory), it can complete the training of long sequences faster and can handle even longer sequences; Transformer can automatically learn complex patterns in the input data without the feature engineering methods relied upon by traditional machine learning methods; compared with CNN (Convolutional Neural Network), Transformer has stronger performance in capturing global dependencies and is more flexible in handling dynamically changing time series.

[0080] Specifically, the large time series model used in this embodiment can adapt to different contexts (up to 512 time series lengths) and prediction lengths, and has the following characteristics:

[0081] 1. Data partitioning – decomposition:

[0082] The model processes time series data through a block-based process. Unlike processing the entire series at once, it divides the data into smaller, more manageable segments. This not only speeds up the model's processing but also allows it to focus on smaller, more detailed trends within the data.

[0083] 2. Decoder-only architecture:

[0084] The model is trained in decoder-only mode. The model optimizes the prediction of the next chunk as a function of all past chunks. Similar to LLMs (Large Language Models), this can be done in parallel across the entire context window, automatically enabling future predictions after observing varying numbers of input chunks.

[0085] 3. Generate longer prediction output blocks:

[0086] In large language models, outputs are typically generated sequentially using an autoregressive approach. However, for long-term time series prediction, large time series models can achieve better accuracy by predicting the entire forecast period at once compared to multi-step autoregressive decoding. This approach allows for the use of output blocks longer than the input blocks. For example, if a model receives a new time series of length 128 and is asked to predict the next 256 time steps, it will first predict time steps 129 to 256. Then, it will use the initial 128-length input and the generated output to predict time steps 257 to 384. A 256-step time series can be generated in just two steps. This proposal uses an input block length of 64 and an output block length of 128.

[0087] For example, the model structure of the large-scale time series model in this embodiment is as follows: Figure 2 As shown, the model structure of this large-scale time series model includes:

[0088] 1. Input residual:

[0089] The time series data is preprocessed and divided into continuous, non-overlapping patches. These patches are then processed into vectors of size equal to the model dimension (1280 dimensions). A binary mask is also provided to the Transformer layer along with the input. The binary mask indicates whether a given data point should be considered (0) or ignored (1).

[0090] 2. Token and Location Encoding:

[0091] A linear transformation is performed on each block to generate tokens. Since the Transformer's self-attention mechanism itself does not contain information about the sequence order, positional encoding is added to the input tokens to provide temporal order information.

[0092] 3. Transformer decoder:

[0093] The model employs a stacked Transformer approach, with each Transformer layer consisting of two main components: a multi-head self-attention mechanism and a feedforward neural network.

[0094] Multi-head self-attention mechanism: Each Transformer layer uses a multi-head self-attention mechanism (16 self-attention heads), allowing the model to focus on different parts of the input sequence simultaneously. For a given output label, the model considers multiple aspects of the previous label at the same time, enhancing its ability to capture complex patterns and dependencies in the data.

[0095] Feedforward network: Following the self-attention mechanism, each layer independently applies a feedforward network to each position in the sequence. This further processes attention information and enables the model to learn higher-level representations.

[0096] Causal self-attention: Ensures that each output label can only focus on the label preceding it. By doing so, the model follows the temporal order of the data, preventing information from future labels (which should not be available at the time of prediction) from influencing the current prediction.

[0097] 20-layer stacking: By stacking multiple Transformer layers, the model can progressively build a more abstract representation of the input data. Each layer optimizes the representations learned from the previous layers, enabling the model to capture complex patterns across different time spans.

[0098] 4. Output residual:

[0099] The task of the output residual is to map the output tokens to the predictions. A linear projection is applied to each output token, mapping them back to the original time series data space. An additional normalization step is performed to ensure the predicted values ​​are within a reasonable range. The output blocks can be longer than the input blocks, allowing the model to predict a larger portion of the time series based on information from the input blocks.

[0100] Monitoring data collected by a real-time monitoring component is used to train a large-scale time-series model based on a transformer, with the aim of optimizing cluster state prediction performance. The dataset includes detailed tracking information on job and task behavior, resource allocation, and machine activity within the cluster. The dataset is divided into multi-dimensional tables, including machine events, machine attributes, job events, task events, and task resources. Each table provides the following information:

[0101] Machine Event Table: This table contains one or more records for each machine in the cluster. Event types include add, remove, and update, and the CPU and memory capacity of each machine are standardized.

[0102] Machine Attributes Table: This table contains key-value pairs that represent machine characteristics, including kernel version, clock frequency, and IP address.

[0103] Job Event Table: This table includes the job's time, ID (Identity document), type, user, and scheduling information. It also records information on active (running) or pending (waiting) jobs, with each job containing scheduling constraints and scheduling delay information.

[0104] Task Event Table: This table contains information such as timestamp, job ID, task index, machine ID, event type, username, scheduling category, task priority, CPU cores, memory, network interface, and local disk space requests.

[0105] Task Resource Table: This table includes the start and end times of the measurement period, job ID, task index, machine ID, CPU utilization, memory utilization, disk I / O time, network congestion, kernel system call saturation, and cache usage. It contains important data needed to understand the actual resource usage in the cluster, such as average CPU utilization, normalized memory utilization, average disk I / O time, and average local disk space utilization.

[0106] Fine-tuning was performed on the pre-trained large-scale time series model to optimize its performance. Since the real-time monitoring data collected during cluster operation is relatively small compared to the pre-trained data (e.g., large-scale pre-trained time series models like TimesFM and TIME-LLM are typically pre-trained on massive real-world time series corpora containing hundreds of billions of data points), the transformer layer employed low-rank LoRa fine-tuning. A small value (rank=4) was chosen for the hyperparameter rank. Full parameter fine-tuning was used for the input / output residuals and positional encoding layers (excluding the transformer layer), and the model's predictive performance was improved through progressive optimization validated by MAE results.

[0107] This embodiment provides a disaster recovery method. When an abnormal operating unit is detected, the method automatically compares the allocated resource capacity of the abnormal operating unit with the spare resource capacity of the normally operating unit. When multiple alternative operating units meet the conditions, the method uses a time-series large model to predict the resource usage trend of each alternative operating unit to accurately select the target operating unit. This allows for the rapid recovery of the backup files and business services of the abnormal operating unit on the selected target operating unit, reducing human error and service interruption time in the face of cluster disasters and improving the efficiency and accuracy of cluster disaster recovery.

[0108] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, the step S20, "predicting the resource usage trend of each candidate operating unit using a time-series large model," may include steps S201 to S202:

[0109] Step S201: Input the resource usage data of each candidate running unit into the time series large model. The resource usage data includes CPU utilization, memory utilization, network congestion, IO call saturation and operating system kernel system call saturation.

[0110] Resource usage data for each candidate operating unit is input into the time-series model. This resource usage data includes CPU utilization, memory utilization, network congestion, I / O call saturation, and operating system kernel system call saturation. This resource usage data is organized in the form of a time series to ensure that the time-series model can capture the trend of resource usage changes over time.

[0111] It should also be noted that in this embodiment, CPU utilization refers to the proportion of CPU usage by the running unit per unit time, usually expressed as a percentage. High CPU utilization indicates a heavy load on the running unit. Memory utilization refers to the proportion of memory usage by the running unit per unit time, usually expressed as a percentage. High memory utilization indicates that the running unit's memory resources are strained. Network congestion refers to data transmission delays or packet loss that occur during network communication by the running unit, usually measured by indicators such as network bandwidth utilization, latency, and packet loss rate. I / O call saturation refers to the resource usage of the running unit when performing input / output operations, usually measured by indicators such as I / O operation wait time and queue length. High I / O call saturation indicates a heavy load on the storage system. Operating system kernel system call saturation refers to the resource usage of the running unit's operating system kernel when processing system calls, usually measured by indicators such as system call wait time and queue length.

[0112] Step S202: Based on resource usage data, predict the resource usage trend of each candidate operating unit in different future time steps, where the future time steps include short-term time steps and long-term time steps.

[0113] Based on the input resource usage data, the time series big data model predicts the resource usage trend of each candidate operating unit within different future time steps. The future time steps can be divided into short-term time steps (such as minute-level, hour-level) and long-term time steps (such as day-level, week-level). The time series big data model predicts the resource usage of candidate operating units in the short and long term by analyzing patterns and trends in historical data.

[0114] It should be noted that the forecasts within the short-term time step focus on immediate resource requirements to ensure that the target operating unit can provide sufficient resource support in the early stages of disaster recovery; while the forecasts within the long-term time step consider the continuous availability and stability of resources to ensure that business services can operate stably for a long period of time after recovery.

[0115] In one feasible embodiment, step S30, "determining the target operating unit from among the candidate operating units based on the resource usage trends of each candidate operating unit," may include steps S301 to S302:

[0116] Step S301: For each candidate operating unit, the resource usage trend of the candidate operating unit in different time steps is weighted and calculated. The weighting strategy includes assigning a first weight to the resource usage trend in the short time step and assigning a second weight to the resource usage trend in the long time step, wherein the first weight is greater than the second weight.

[0117] For each candidate operating unit, the system performs a weighted calculation of its resource usage trends at different time steps. The weighting strategy is designed to emphasize the importance of resource usage trends in the near future so that resource needs can be responded to quickly in the early stages of disaster recovery.

[0118] Specifically, a short-term time step refers to a shorter time frame, such as minutes or hours, primarily focusing on immediate resource needs after a disaster. The system assigns a higher weight (first weight) to resource usage trends within this short-term time step to ensure that future short-term resource usage is prioritized during recovery. A long-term time step refers to a longer time frame, such as days or weeks, primarily focusing on the continuous availability and stability of resources. The system assigns a lower weight (second weight) to resource usage trends within this long-term time step to ensure that future short-term resource usage trends dominate the overall assessment.

[0119] This weighted calculation allows for a more accurate assessment of the resource availability of each alternative operating unit in the early stages of disaster recovery, thereby enabling more rational recovery decisions.

[0120] Step S302: Select the target operating unit from each candidate operating unit based on the target resource usage trend obtained after weighted calculation.

[0121] After completing the weighted calculation, a target operating unit will be selected from the candidate operating units based on the obtained target resource usage trend. Specifically, the selection of the target operating unit will prioritize operating units with low resource utilization in the short term that can meet the recovery requirements, so as to ensure that sufficient resource support can be provided during the recovery process, thereby providing the best resource support for the recovery of the abnormal operating unit's business services and ensuring the efficiency and accuracy of the recovery process.

[0122] In one feasible embodiment, steps B10 to B40 may be included before step S10:

[0123] Step B10: Monitor the running status of each running unit in Kubernetes in real time.

[0124] Continuously monitor the running status of each running unit in Kubernetes. The monitoring metrics include key performance indicators such as CPU utilization, memory utilization, network congestion, IO call saturation, and operating system kernel system call saturation. These metrics can be collected in real time through Kubernetes' monitoring API (Application Programming Interface) and event system to ensure that the system can detect any potential anomalies in a timely manner.

[0125] Step B20: When an abnormality occurs in the running status, check whether the cluster control agent of the cluster where the abnormality occurs has failed.

[0126] When an abnormality is detected in the running status of a certain running unit, it is further checked whether the cluster control agent of the cluster where the abnormality is located has failed. The cluster control agent is a component in Kubernetes that is responsible for managing and coordinating the nodes and Pods in the cluster. Each cluster in Kubernetes has its corresponding cluster control agent. The cluster control agent can communicate with the Kubernetes API server to ensure that resource management and task scheduling in the cluster are carried out normally.

[0127] Step B30: When the cluster control agent fails, the abnormal operating unit is identified as the abnormal cluster, and the normal operating units are the normal clusters in Kubernetes.

[0128] If a cluster control agent failure is detected, the abnormal operating unit is determined to be an abnormal cluster. In this case, the entire cluster is considered an abnormal unit and disaster recovery operations are required. The normal operating unit is the other normal clusters in Kubernetes that have not experienced any abnormalities. In this case, the recovery operation will involve cross-cluster resource allocation and business service migration.

[0129] Step B40: If the cluster control agent is not ineffective, identify the abnormal operating unit as the abnormal node and the normal operating units as the normal nodes in the same cluster as the abnormal node.

[0130] If the cluster control agent is not faulty, the abnormal operating unit is identified as the abnormal node. In this case, the abnormality is limited to one or more nodes in the cluster, while other nodes in the same cluster are still operating normally. The normal operating unit is each normal node in the same cluster as the abnormal node. In this case, the recovery operation will focus on the resource reallocation and business service migration of the nodes within a single cluster.

[0131] For example, when the abnormal operating unit is an abnormal cluster, the cluster disaster recovery workflow is as follows: Figure 3As shown, when a Kubernetes cluster encounters a disaster, such as hardware failure, network attack, or other events that cause service interruption, the system executes the following cluster disaster recovery process:

[0132] Step 1: Continuously monitor the cluster status, utilizing Kubernetes' monitoring API and event system to capture any anomalies occurring within the cluster. Once an anomaly is detected, the event detection component will activate, identify the disaster type, and record the time of the disaster, its scope of impact, and possible causes.

[0133] Step 2: Determine the name of the affected cluster and compare its allocated CPU cores, memory, and storage capacity with other managed clusters to check if any cluster has more available resources than the affected cluster. If no other cluster meets this criterion, the recovery process will halt and an alert will be sent to the user because proper recovery is not possible. Conversely, if clusters are found to have more available resources than the affected cluster, their names are retrieved for the recovery process and sent to the recovery target cluster selection component. Since multiple clusters may have more CPU cores than the affected cluster, multiple cluster names can be transmitted to the recovery target cluster selection component.

[0134] Step 3: Using a time-series large-scale model, predict the CPU utilization, memory utilization, network congestion, I / O call saturation, operating system kernel system call saturation, and other key resource indicators of each cluster to assess which cluster is most suitable for recovery operations. Use historical resource usage data as a dataset and a time-series large-scale model to predict resource usage trends at different future time steps (minutes, hours, days). The cluster with the lowest resource utilization is selected as the recovery target based on the weighted average calculation.

[0135] Step 4: Once the target cluster has been identified, initiate the recovery process. This includes retrieving the latest backup files from backup storage (such as AWS S3). The recovery execution component uses Kubernetes backup and recovery tools (such as Velero) to restore services and data on the target cluster and leverages workflow automation tools (such as Argo Workflows) to execute complex business recovery processes.

[0136] Step 5: After recovery is complete, the system will collect feedback information on the recovery operation, evaluate the effectiveness of the recovery process, and identify any areas for improvement. Based on the feedback, the automated process will be improved to enhance the efficiency and effectiveness of future disaster recovery. Simultaneously, based on the feedback data and combined with real-time monitoring time-series datasets, the pre-trained time-series model will be fine-tuned to optimize its performance. Since the real-time monitoring data collected during cluster operation is relatively small compared to the pre-trained data (e.g., pre-trained time-series models such as TimesFM and TIME-LLM are typically pre-trained on large corpora of hundreds of billions of real-world time series data), the transformer layer uses LoRa low-rank fine-tuning. A small value (rank=4) is chosen for the hyperparameter rank, and the model's predictive performance is gradually improved by verifying the results of MAE. The input / output residuals and positional encoding layers, excluding the transformer layer, use full-parameter fine-tuning.

[0137] Step 6: Throughout the recovery process, meticulously record all operational steps, times, events, and logs to facilitate post-event auditing and troubleshooting. The logging component ensures that all critical information is properly preserved and available for analysis and review when needed.

[0138] For example, when the abnormal operating unit is an abnormal node, the node fault recovery workflow is as follows: Figure 4 As shown, when a Kubernetes compute node encounters a failure, such as a hardware failure, network attack, or other event that causes service interruption, the system enters the disaster recovery process as follows:

[0139] Step 1': Continuously monitor the cluster status, utilizing Kubernetes' monitoring API and event system to capture any anomalies occurring within the cluster. Once an anomaly is detected, the event detection component will activate and identify the disaster type, recording the time of the disaster, its scope of impact, and possible causes.

[0140] Step 2': Determine the name of the affected node, and compare the number of CPU cores, memory, and storage capacity allocated to this node with those of other healthy nodes in the cluster to check whether there are enough spare resources in the cluster for pod rescheduling.

[0141] Step 3': Utilize a time-series large-scale model to predict CPU utilization, memory utilization, network congestion, IO call saturation, operating system kernel system call saturation, and other key resource metrics for each cluster. This will help assess which nodes are suitable as target nodes for recovery operations to reschedule affected pods. Historical resource usage data will be used as the dataset, and a time-series large-scale model will be used to predict resource usage trends at different future time steps (minutes, hours, days). The target node with the lowest resource utilization will be selected as the recovery target based on the weighted average calculation.

[0142] Step 4: Within minutes of the failed node's kubelet stopping heartbeat reporting, the Kubernetes controller-manager will initiate the pod eviction and rescheduling process. After identifying the target node, the cluster scheduler assigns higher preference parameters to it, enabling it to achieve a higher scheduling score in the scheduling decision process minutes later, thus securing its scheduling.

[0143] Step 5': After recovery is complete, the system will collect feedback information on the recovery operation, evaluate the effectiveness of the recovery process, and identify any areas for improvement. Based on the feedback data, combined with the real-time monitored time-series dataset, the pre-trained time-series large model will be fine-tuned using LoRa low-rank tuning to optimize the model's performance.

[0144] Step 6': Throughout the recovery process, the system records all operation steps, times, events, and logs in detail to facilitate post-event auditing and troubleshooting. The logging component ensures that all critical information is properly stored and available for analysis and review when needed.

[0145] In one feasible embodiment, the step of "restoring the backup files and business services of the abnormal operating unit in the target operating unit" in step S30 may include steps S303 to S305:

[0146] Step S303: Retrieve backup files of abnormally operating units from the preset backup storage area.

[0147] Retrieve the backup file corresponding to the abnormal operating unit from the preset backup storage area. The backup storage area can be local storage, a distributed storage system, or a cloud storage service such as Amazon S3. The backup file contains the configuration information, application data, and system status information of the abnormal operating unit, which are necessary for restoring business services.

[0148] In practical applications, the system can accurately locate and retrieve the latest backup file of an abnormal operating unit by using the backup management system or API, based on the identifier and backup timestamp of the abnormal operating unit.

[0149] Step S304: Restore the backup file in the target running unit to rebuild the application data of the abnormal running unit on the target running unit.

[0150] The retrieved backup file is restored in the target running unit, and the data and configuration information in the backup file are applied to the target running unit to rebuild the application data of the abnormal running unit.

[0151] The specific operation may include the following steps:

[0152] The data (such as database files, log files, configuration files, etc.) in the backup files are restored to the corresponding storage location of the target running unit.

[0153] Apply the configuration information (such as network configuration, service configuration, environment variables, etc.) from the backup file to the target running unit to ensure that the configuration of the target running unit is consistent with that of the abnormal running unit.

[0154] Using the Kubernetes API, you can create or update necessary Kubernetes resource objects based on the resource definitions in the backup file, such as Pods, Services, Deployments (a controller that manages the deployment and updates of Pods), StatefulSets (a controller that manages stateful applications), and ConfigMaps (a key-value resource that stores configuration data).

[0155] Step S305: Based on the application data, restore the business services of the abnormal operating unit in the target operating unit.

[0156] Based on the recovered application data, the business services of the abnormally running unit are restored in the target running unit, and the restored business services are started to ensure that the services can run normally and respond to user requests.

[0157] The specific operation may include the following steps:

[0158] Start the restored business services in the target operating unit to ensure that the services can start normally and reach the expected operating state.

[0159] Kubernetes health check mechanisms are used to verify the health status of services and ensure that services can respond to requests normally.

[0160] Perform functional verification, including but not limited to service response time, data integrity, and the correctness of business logic, to ensure that the business services can operate normally after recovery.

[0161] Configure the network and storage connections of the target operating unit to ensure that the restored business services can access the required network resources and persistent storage normally.

[0162] In summary, this embodiment monitors the operational status of the running units in real time and uses a time-series large model based on the transformer decoder architecture to predict multiple comprehensive indicators, including CPU utilization, memory utilization, network congestion, IO call saturation, and host kernel system call saturation. It then intelligently selects the target running unit for recovery and automates the recovery process, thereby significantly improving the efficiency and accuracy of disaster recovery, reducing human error and system downtime, and providing enterprises with an innovative cloud environment business / data protection solution.

[0163] Furthermore, to aid in understanding the implementation scenario of the disaster recovery method obtained by combining this embodiment with the above embodiments, the following uses a disaster recovery system as an example. This disaster recovery system includes a real-time monitoring component, an event detection component, a resource comparison and alerting component, a recovery target cluster selection component based on a time-series large model, a recovery execution component, a customized intra-cluster scheduling component, a data backup and synchronization component, and log recording and auditing. The implementation process of the disaster recovery method will be explained in detail below:

[0164] Real-time monitoring component: Responsible for real-time monitoring of node performance metrics in the Kubernetes cluster, including CPU utilization, memory utilization, network congestion, IO call saturation, and operating system kernel system call saturation. It aggregates metric data at both the host and cluster levels to generate time-series metric data with different time precisions (minute, hour, and day).

[0165] The event detection component is responsible for real-time monitoring of the Kubernetes cluster's operational status. It collects health status, performance metrics, and network connectivity information for each node and Pod through hooks and probes integrated into the Kubernetes API. Leveraging Kubernetes' event listening mechanism, the component can subscribe to and respond to various events occurring within the cluster. When an abnormal event is detected, such as node failure, service unavailability, or network interruption, the component triggers an alert and initiates subsequent recovery procedures.

[0166] Resource Comparison and Alerts Component: Once the monitoring component detects a disaster, the resource comparison component immediately assesses the resources of the affected cluster. When a node fails, it is compared to other healthy nodes within the cluster; when a cluster experiences a disaster, it is compared to other healthy clusters to determine potential recovery targets. This component uses node and cluster resource usage reports and capacity data to algorithmically compare the availability and demand of computational resources (CPU utilization, memory utilization, network congestion, I / O call saturation, and operating system kernel system call saturation). If insufficient resources are detected or better recovery candidates exist, the component will issue alerts and recommend the best node or cluster for recovery.

[0167] The recovery target cluster selection component based on a time-series large-scale model is the core of the system. It is responsible for using the time-series large-scale model to predict the overall resource usage trends of each cluster and the resource usage trends of each cluster node, and then selecting the optimal recovery target. The time-series large-scale model learns the dynamic characteristics of time series data through historical data, using this as input to train the model to recognize patterns and trends. Based on this learning, the model can predict resource usage at future points in time and at multiple time steps, and assess which node and which cluster has the best resource status during recovery.

[0168] Recovery Execution Component: After a cluster disaster, once the recovery target is determined, the recovery execution component is responsible for the actual recovery operations, including data synchronization, service migration, and state restoration. The component restores services by executing a series of automated scripts and commands. This may include retrieving data from backup storage, recreating Kubernetes resource objects on the target cluster, and configuring necessary network and storage connections. The system requires a robust automation engine to orchestrate the various steps in the recovery process. Leveraging Kubernetes job and CronJob resources, the automation engine can schedule and execute complex sequences of tasks. Furthermore, more complex recovery processes can be created using workflow automation tools such as Argo Workflows.

[0169] Customized in-cluster scheduling component: After a node failure, resource usage metrics predicted by a time-series large model are used to identify the target node for recovery. A scheduling score is added to encourage pods to be rescheduled to the optimal target node.

[0170] Data backup and synchronization components: To ensure rapid recovery in the event of a disaster, the system needs to regularly back up the cluster state. Leveraging Kubernetes' snapshot and persistent volume features, these components can capture the current state of the cluster and store it in a remote or distributed storage system. Furthermore, real-time data synchronization technology ensures the consistency and up-to-dateness of the backup data.

[0171] Log recording and auditing: The system records detailed logs of all operations to facilitate post-event auditing and troubleshooting. The component uses a standard logging library to capture and store critical information such as operation time, operator, executed operation, and result. These logs can be used for security auditing and performance monitoring.

[0172] For example, to help understand the implementation process of the disaster recovery method obtained by combining the above embodiments, this example uses cluster disaster recovery scenario and node failure recovery scenario as examples to illustrate the implementation process of the disaster recovery method. Specifically:

[0173] like Figure 5 As shown, the cluster disaster recovery implementation process in this example includes:

[0174] 1. Real-time / Regular Backup:

[0175] Before a failure occurs, the system backs up the status and data of the business cluster to S3 object storage in real time or periodically, using a unified backup strategy. The backup files include all necessary configurations, application data, and system status information, ensuring that the cluster can be fully restored.

[0176] 2. Cluster failure detected:

[0177] The automatic recovery system communicates with Kubernetes cluster components through the apiserver to monitor the cluster status in real time. Once a fault is detected, the system immediately triggers an alert and logs fault details, including the time, scope of impact, and possible error messages.

[0178] 3. Initiate the disaster recovery control process:

[0179] Using a large time-series model, the system predicts CPU utilization, memory utilization, network congestion, IO call saturation, and operating system kernel system call saturation for each cluster at different time steps (minutes, hours, days) to assess which cluster is most suitable for recovery operations.

[0180] 4. Obtain the backup / restore file path:

[0181] Once a failure is confirmed, the automatic recovery system retrieves backup files from S3 object storage. The system will locate the latest backup file based on the backup timestamp and cluster identifier, and obtain the path to the backup file.

[0182] 5. The service cluster controller executes the recovery process:

[0183] The automatic recovery system coordinates multiple business cluster controllers, each managing different clusters. The system selects a healthy business cluster controller that meets key performance indicators as the recovery target. The selected controller will then prepare to execute the recovery task.

[0184] 6. Recovery process execution:

[0185] On the target business cluster controller, the automated recovery system will execute the recovery process. This includes pulling backup files from S3 object storage and then applying these backups to the target cluster to rebuild the cluster state and application data. The recovery process will ensure that all services and data are restored to their pre-failure state. Upon successful recovery, the system will verify the recovery results and terminate the recovery process after confirming that the cluster is back to normal operation.

[0186] like Figure 6 As shown, the cluster disaster recovery implementation process in this example includes:

[0187] 1. Compute node failure:

[0188] When a node experiences an incurable failure, including hardware failure, kernel failure, network connection interruption, or host crash, the node becomes unresponsive, and the kubelet (node ​​agent) stops reporting heartbeats.

[0189] 2. Report the metric status of the faulty node through the cluster control agent:

[0190] The business cluster marks a node as NotReady (unhealthy) after NodeMonitorGracePeriod (a time interval, usually 40 seconds) and notifies the business cluster control agent of this event. NodeMonitorGracePeriod is a configuration parameter in Kubernetes used to define the grace period for waiting for a node's response before marking it as unhealthy (NotReady), avoiding prematurely marking a node as unhealthy, which could lead to unnecessary Pod evictions and rescheduling. After receiving a node failure event, the control agent component waits for half of PodEvictionTimeout (a time interval, typically 300s) before reporting the historical operating metrics of the failed node (including CPU utilization, memory utilization, network congestion, IO call saturation, and operating system kernel system call saturation) and the status information of node API resources to the business cluster controller. PodEvictionTimeout is a configuration parameter in Kubernetes used to define the timeout before evicting Pods on a node. It is used to handle node failures or unreachable situations, ensuring that Pods are not immediately evicted after a node is marked as unhealthy, thereby avoiding premature rescheduling and reducing unnecessary resource waste and business interruptions.

[0191] 3. Obtain monitoring metrics for healthy nodes and push them to the management cluster:

[0192] After the business cluster controller learns about the failure event of a node in the business cluster, the status information of the node API (application programming interface related to the node), and the historical time-series indicators, it calls the real-time indicator monitoring component through the business cluster control agent to obtain the historical operating indicators of other healthy nodes in the cluster.

[0193] 4. The time-series large-scale model predicts indicators and selects the target node for recovery:

[0194] Using a large time-series model, the system predicts CPU utilization, memory utilization, network congestion, I / O call saturation, and operating system kernel system call saturation for each cluster at different time steps (minutes, hours, days) to assess which cluster is most suitable for recovery operations. The specific steps are as follows:

[0195] Using a large time series model, the system predicts CPU utilization, memory utilization, network congestion, IO call saturation, and operating system kernel system call saturation for each cluster at different time steps (minutes, hours, days).

[0196] The expected remaining capacity is calculated by subtracting the predicted metrics of each healthy node from the node's capacity after smoothing the metrics and giving greater influence to the most recent data points. The expected values ​​are then sorted in descending order and summed to calculate the index (pointer) at which the sum exceeds the expected future resource usage of the failed node. The list of target nodes is then defined from 0 to the index.

[0197] 5. Adjust the optimal parameters of the target node:

[0198] The business cluster control agent normalizes the list of expected remaining resources for candidate healthy nodes and adds the values ​​to the annotations of the node API of the target candidate node via the API server.

[0199] 6. Pod rescheduling:

[0200] After a pod is evicted from a failed node, the scheduler receives a new scheduling request and makes a scheduling decision. In the optimization phase, after filtering out unsuitable nodes, normalized optimization parameters are obtained through annotations in the node's API. These parameters are multiplied by a preset score cap constant to obtain an optimization score, which is then added to the total score of the optimization phase. This results in the target candidate node receiving a higher scheduling score.

[0201] Throughout the automated recovery process, the system is designed to minimize human intervention, maximizing recovery speed and accuracy. Here are some key considerations:

[0202] Real-time monitoring: The system needs to have real-time monitoring capabilities to detect and respond immediately when a fault occurs.

[0203] Backup strategy: A regular backup strategy is crucial to ensuring that data and state can be fully recovered.

[0204] Fault tolerance: The system should be designed to tolerate faults, for example, by storing backups in multiple regions or on multiple devices to avoid single points of failure.

[0205] Automation: Automated processes reduce human error and improve the consistency and reliability of recovery.

[0206] Recovery Verification: After recovery, the system should automatically verify the recovery results to ensure that all services and data have been correctly restored.

[0207] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the disaster recovery method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0208] This application also provides a disaster recovery device, please refer to... Figure 7 The disaster recovery device includes:

[0209] The capacity comparison module 10 is used to compare the allocated resource capacity of the abnormal operating unit with the free resource capacity of each normally operating unit after detecting the abnormal operating unit.

[0210] The resource usage prediction module 20 is used to predict the resource usage trend of each alternative operating unit when the available resource capacity of multiple alternative operating units in each normal operating unit exceeds the allocated resource capacity, using a time series large model.

[0211] The recovery module 30 is used to determine the target running unit from among the candidate running units based on the resource usage trends of each candidate running unit, and to restore the backup files and business services of the abnormal running unit in the target running unit.

[0212] Optionally, the disaster recovery device also includes a model training module (not shown), which is used for:

[0213] Before the step of using a time-series large model to predict the resource usage trends of each candidate operating unit, the following steps are also included:

[0214] The historical resource usage data of each running unit is used as the training dataset and input into the pre-trained temporal large model based on the Transformer decoder architecture;

[0215] LoRa low-rank fine-tuning technique is used to fine-tune the pre-trained temporal large model;

[0216] The model performance is validated by calculating the mean absolute error of the fine-tuned pre-trained time series large model on the validation dataset, and the model parameters are optimized based on the validation results to obtain the time series large model.

[0217] Optionally, the resource usage prediction module 20 is also used for:

[0218] The resource usage data of each candidate running unit is input into the time series model. The resource usage data includes CPU utilization, memory utilization, network congestion, I / O call saturation and operating system kernel system call saturation.

[0219] Based on resource usage data, predict the resource usage trends of each candidate operating unit at different future time steps, where the future time steps include short-term time steps and long-term time steps.

[0220] Optionally, the resource usage prediction module 20 is also used for:

[0221] For each candidate operating unit, the resource usage trend of the candidate operating unit in different time steps is weighted and calculated. The weighting strategy includes assigning a first weight to the resource usage trend in the short time step and assigning a second weight to the resource usage trend in the long time step, wherein the first weight is greater than the second weight.

[0222] Based on the target resource usage trend obtained after weighted calculation, the target operating unit is selected from each candidate operating unit.

[0223] Optionally, the disaster recovery device also includes a status monitoring module (not shown), which is used for:

[0224] Real-time monitoring of the running status of each running unit in Kubernetes;

[0225] When an abnormality occurs during operation, check whether the cluster control agent of the cluster where the abnormality occurs has failed;

[0226] When the cluster control agent fails, the abnormal operating unit is identified as the abnormal cluster, and the normal operating units are the normal clusters in Kubernetes.

[0227] When the cluster control agent is not ineffective, the abnormal operating unit is identified as the abnormal node, and the normal operating units are each normal node in the same cluster as the abnormal node.

[0228] Optionally, the recovery module 30 is also used for:

[0229] Retrieve backup files of abnormally operating units from the preset backup storage area;

[0230] Restore the backup files in the target run unit to rebuild the application data of the abnormal run unit on the target run unit;

[0231] Based on application data, restore the business services of the abnormal operating unit in the target operating unit.

[0232] The disaster recovery device provided in this application, employing the disaster recovery method described in the above embodiments, can improve the efficiency and accuracy of Kubernetes cluster disaster recovery. Compared with the prior art, the beneficial effects of the disaster recovery device provided in this application are the same as those of the disaster recovery method provided in the above embodiments, and other technical features in the disaster recovery device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0233] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the disaster recovery method in the first embodiment described above.

[0234] The following is for reference. Figure 8 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The disaster recovery device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0235] like Figure 8As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following devices may be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although electronic devices with various devices are shown in the figures, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0236] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0237] The electronic device provided in this application, employing the disaster recovery method described in the above embodiments, can improve the efficiency and accuracy of Kubernetes cluster disaster recovery. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the disaster recovery method provided in the above embodiments, and other technical features of the electronic device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0238] It should be understood that the various parts disclosed in the embodiments of this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0239] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0240] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the disaster recovery method in the above embodiments.

[0241] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0242] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0243] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by an electronic device, the electronic device: upon detecting an abnormal operating unit, compares the allocated resource capacity of the abnormal operating unit with the spare resource capacity of each normally operating unit; when there are multiple candidate operating units in each normally operating unit whose spare resource capacity exceeds the allocated resource capacity, uses a time-series large model to predict the resource usage trend of each candidate operating unit; and based on the resource usage trend of each candidate operating unit, determines a target operating unit from among the candidate operating units, and restores the backup files and business services of the abnormal operating unit in the target operating unit.

[0244] Computer program code for performing the operations of the embodiments of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0245] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using dedicated hardware-based apparatus to perform the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0246] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0247] The readable storage medium provided in this application embodiment is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described disaster recovery method, thereby improving the efficiency and accuracy of Kubernetes cluster disaster recovery. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application embodiment are the same as the beneficial effects of the disaster recovery method provided in the above embodiments, and will not be repeated here.

[0248] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the disaster recovery method described above.

[0249] The computer program product provided in this application can improve the efficiency and accuracy of disaster recovery for Kubernetes clusters. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the disaster recovery methods provided in the above embodiments, and will not be repeated here.

[0250] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A disaster recovery method, characterized in that, The disaster recovery method includes: After detecting an abnormal operating unit, the allocated resource capacity of the abnormal operating unit is compared with the available resource capacity of each normally operating unit. When the available resource capacity of multiple candidate operating units in each of the normal operating units exceeds the allocated resource capacity, a time-series large model is used to predict the resource usage trend of each of the candidate operating units. Based on the resource usage trends of each of the candidate operating units, a target operating unit is determined from the candidate operating units, and the backup files and business services of the abnormal operating unit are restored in the target operating unit.

2. The disaster recovery method as described in claim 1, characterized in that, Before the step of using a time-series large model to predict the resource usage trends of each of the candidate operating units, the method further includes: The historical resource usage data of each running unit is used as the training dataset and input into the pre-trained temporal large model based on the Transformer decoder architecture; The pre-trained temporal large model was fine-tuned using LoRa low-rank fine-tuning technology; The model performance is validated by calculating the mean absolute error of the fine-tuned pre-trained time series large model on the validation dataset, and the model parameters are optimized based on the validation results to obtain the time series large model.

3. The disaster recovery method as described in claim 1, characterized in that, The step of using a time-series large model to predict the resource usage trends of each of the candidate operating units includes: The resource usage data of each of the candidate running units is input into the time series model, wherein the resource usage data includes CPU utilization, memory utilization, network congestion, IO call saturation and operating system kernel system call saturation. Based on the resource usage data, the resource usage trend of each of the candidate operating units is predicted at different future time steps, wherein the future time steps include short-term time steps and long-term time steps.

4. The disaster recovery method as described in claim 3, characterized in that, The step of determining the target operating unit from the candidate operating units based on their resource usage trends includes: For each of the candidate operating units, the resource usage trend of the candidate operating unit in different time steps is weighted and calculated. The weighting strategy includes assigning a first weight to the resource usage trend in the short time step and assigning a second weight to the resource usage trend in the long time step, wherein the first weight is greater than the second weight. Based on the target resource usage trend obtained after weighted calculation, a target operating unit is selected from each of the candidate operating units.

5. The disaster recovery method as described in claim 1, characterized in that, Before the step of comparing the allocated resource capacity of the abnormal operating unit with the free resource capacity of each normally operating unit after detecting the abnormal operating unit, the method further includes: Real-time monitoring of the running status of each running unit in Kubernetes; When an abnormality occurs in the operating state, it is detected whether the cluster control agent of the cluster where the abnormality occurs has failed. When the cluster control agent fails, the abnormal operating unit is identified as an abnormal cluster, and the normal operating units are each normal cluster in Kubernetes. When the cluster control agent is not in failure, the abnormal operating unit is identified as an abnormal node, and the normal operating units are each normal node in the same cluster as the abnormal node.

6. The disaster recovery method according to any one of claims 1 to 5, characterized in that, The step of restoring the backup files and business services of the abnormal operating unit in the target operating unit includes: Retrieve the backup file of the abnormal operating unit from the preset backup storage area; The backup file is restored in the target running unit to rebuild the application data of the abnormal running unit on the target running unit; Based on the application data, the business services of the abnormal operating unit are restored in the target operating unit.

7. A disaster recovery device, characterized in that, The disaster recovery device includes: The capacity comparison module is used to compare the allocated resource capacity of the abnormal operating unit with the free resource capacity of each normally operating unit after detecting an abnormal operating unit. The resource usage prediction module is used to predict the resource usage trend of each of the candidate operating units when the available resource capacity of multiple candidate operating units in each of the normal operating units exceeds the allocated resource capacity, using a time series large model. The recovery module is used to determine the target running unit from the candidate running units based on the resource usage trends of each candidate running unit, and to restore the backup files and business services of the abnormal running unit in the target running unit.

8. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the disaster recovery method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the disaster recovery method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the disaster recovery method as described in any one of claims 1 to 6.

Images