Intelligent cluster fault tolerance method based on distributed component dynamic migration

Abstract

The invention discloses a distributed component dynamic migration-based cluster intelligent fault-tolerant method, which comprises the following steps of: firstly, collecting historical load state data of each node, container and component service of a cluster through a distributed monitoring system, preprocessing the historical load state data as input of a deep learning model, training the model by using a deep learning algorithm, and performing fault tolerance on the model; based on a prediction result of the model, after the system generates an intelligent scheduling decision and executes scheduling, the system continuously monitors cluster resources and service states, discovers resource bottlenecks or faults in time and carries out adjustment, and finally, a scheduling strategy is adjusted through real-time monitoring data feedback and incremental training is carried out on the deep learning model. According to the method, deep learning modeling and prediction are carried out on historical load state data of the cluster, the load change trend is recognized in advance, scheduling decision making is carried out, and through the cluster self-healing mechanism and prediction scheduling, the system fault time is effectively shortened, the service continuity is guaranteed, and the system reliability is improved. The method can be applied to a cloud platform, a micro-service architecture and a large-scale distributed system.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a cluster intelligent fault-tolerant method based on dynamic migration of distributed components. Background Art

[0002] With the development of cloud computing and large-scale distributed systems, enterprises and organizations are increasingly relying on efficient and reliable computing platforms to process massive data and complex tasks. Modern distributed systems are usually composed of multiple nodes, each of which may host multiple applications or services. Resource utilization efficiency, load balancing, and fault tolerance are crucial to the overall performance and reliability of the system. However, as the scale of the system expands and the load changes continue to increase, how to effectively schedule resources, avoid overload, ensure high availability, and ensure rapid recovery has become a huge challenge.

[0003] In a distributed computing environment, a cluster is usually composed of multiple computing nodes that work together to complete tasks and provide services. The resources on each node (such as CPU, memory, storage, bandwidth, etc.) are limited, while the task or service requirements in the system may fluctuate greatly. Therefore, the management and scheduling of cluster resources becomes particularly important. Traditional cluster management methods usually rely on static resource allocation and scheduling strategies, which often have difficulty in coping with complex load changes and high concurrency requirements.

[0004] Although the existing cluster management, container scheduling, load balancing and fault recovery technologies have made great progress, they still face many challenges. First, the existing scheduling algorithms are usually based on fixed rules and strategies and lack the ability to predict dynamic load changes. Secondly, during the container migration and service migration process, they may be affected by network latency, data consistency and other issues, resulting in performance loss during the scheduling process. In addition, the existing fault tolerance mechanisms usually rely on manual intervention or static backup strategies and lack intelligent self-healing capabilities.

[0005] Therefore, the combination of load prediction, dynamic migration and intelligent scheduling methods can better cope with the resource management and fault tolerance challenges in large-scale distributed clusters and improve the flexibility, efficiency and reliability of the system.

[0006] In the future, with the continuous development of artificial intelligence, machine learning and big data analysis technology, cluster load prediction and intelligent scheduling technologies based on historical data and real-time monitoring information will be more widely used. These intelligent algorithms can not only predict system load in advance, but also dynamically adjust scheduling strategies based on prediction results, realize dynamic migration of containers and services, efficient allocation of resources and fault recovery, thereby greatly improving the stability, scalability and fault tolerance of distributed clusters. Summary of the invention

[0007] To solve the above technical problems, the present invention provides a cluster intelligent fault-tolerant method based on dynamic migration of distributed components. By performing deep learning modeling and prediction on the historical load data of the cluster, the load change trend can be identified in advance, thereby making accurate scheduling decisions and ensuring high availability and load balancing of the system.

[0008] The technical solution adopted by the present invention is: a cluster intelligent fault-tolerant method based on dynamic migration of distributed components, and the specific steps are as follows:

[0009] S1. Establish a cluster monitoring and scheduling system, including: a distributed monitoring system and a distributed scheduling system. The distributed monitoring system is used to monitor each node, container, and service component in the distributed component cluster, and collect historical load status data in the cluster at a set frequency, i.e., historical monitoring data;

[0010] The distributed monitoring system includes: Prometheus, Zabbix, Nagios, Icinga, Datadog; the historical monitoring data includes: CPU usage, memory usage, hard disk usage, disk I / O, and network bandwidth.

[0011] S2, preprocessing the historical load status data collected in step S1, and constructing a data set for the preprocessed data as input of the deep learning model;

[0012] The preprocessing includes: denoising, cropping, standardization and normalization;

[0013] First, the historical load status data collected in step S1 is cleaned, denoised, and zero-mean normalized, and then standardized and cropped. The cropping length is set to k, where the first m data of each sample data segment are used as the input of the deep learning model, and the last km data are used as the output prediction of the model; the constructed data set contains n samples, the dimension of the data set is n*5*m, and the training set and the validation set are divided according to the set ratio.

[0014] Among them, n represents the number of samples, and the normalized expression is as follows:

[0015]

[0016] Among them, X represents the original data, X * represents the normalized data, μ represents the mean of the original data, and δ represents the standard deviation of the original data. The specific expressions of μ and δ are as follows:

[0017]

[0018] Among them, x i Represents the i-th sample in the original data.

[0019] S3, based on the data set generated in step S2, input the constructed deep learning model for training to obtain a trained deep learning model;

[0020] The deep learning model includes: CNN, RNN, LSTM, GRU, Transformer. The constructed deep learning model is used to input the data set generated in step S2 for model training, the input data is predicted by the model, the error estimation is constructed between the predicted data and the real data, the model is back-propagated by the optimizer to update the model parameters, and the training is iterated until the loss function converges and the model tends to be stable, and the model training stops.

[0021] S4, intelligent monitoring and scheduling of system status based on the distributed scheduling system, that is, regular collection of real-time monitoring data, pre-processing and input into the model trained in step S3 for inference and prediction, and inferring the status of nodes, containers and components;

[0022] The real-time monitoring data includes: CPU usage, memory usage, hard disk usage, disk I / O, and network bandwidth. The real-time monitoring data is predicted to obtain the future status of the real-time monitoring data. According to the future status of each indicator, the resource consumption trend of each node, the resource demand change of the container, and the load demand of the service are obtained to identify possible overloads, resource bottlenecks, or potential failure points in the future.

[0023] S5. Based on the load prediction result obtained in step S4, a fault judgment is performed on the load prediction result. If no fault is found, the process returns to step S4 to continue intelligent monitoring and scheduling of the system status. If a fault is found to be about to occur, the system generates an intelligent scheduling decision and enters step S6.

[0024] The scheduling decision includes: component service scheduling and container scheduling.

[0025] Among them, component service scheduling decides whether to migrate services to other nodes based on the load demand and priority of the services to avoid resource bottlenecks of high-load nodes. Container scheduling decides whether containers need to be migrated to other nodes based on the cluster resource status and the load prediction results of the containers.

[0026] S6. Based on step S5, the scheduling is performed, the container or component service is migrated, and the monitoring and scheduling system goes to step S4 to continue to monitor the resource usage of the cluster, the health status of the service, and the operation status of the container in real time, and at the same time triggers training;

[0027] S7. Based on step S6, after the training is triggered, the monitoring and scheduling system organizes the historical monitoring data of the recent period, performs data preprocessing and updates the data set, and then repeats steps S3-S6 to continuously iterate and optimize the model to ensure the longer-term stability of the system and realize intelligent fault tolerance of the cluster.

[0028] Among them, the monitoring and scheduling system continuously updates the model based on feedback, improves load prediction accuracy and scheduling decisions, and enhances the distributed monitoring and scheduling system's ability to adapt to dynamic loads.

[0029] Beneficial effects of the present invention:

[0030] The method of the present invention first collects the historical load status data of each node, container and component service of the cluster through a distributed monitoring system and pre-processes it as the input of the deep learning model, and then uses the deep learning algorithm to train the model. Based on the prediction results of the model, the system generates intelligent scheduling decisions. After the scheduling is executed, the system continuously monitors the cluster resources and service status, promptly discovers resource bottlenecks or failures, and makes adjustments. Finally, the scheduling strategy is adjusted through real-time monitoring data feedback and the deep learning model is incrementally trained. The method of the present invention performs deep learning modeling and prediction on the historical load status data of the cluster, identifies the load change trend in advance, and thus makes accurate scheduling decisions, ensures the high availability and load balancing of the system, avoids the system from being overloaded or service interrupted when the load is high, and realizes more efficient resource allocation and scheduling. Through this cluster self-healing mechanism and predictive scheduling, the system failure time can be effectively reduced, and service continuity can be guaranteed. It can be applied to cloud platforms, microservice architectures and large-scale distributed systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The present invention is a flow chart of a cluster intelligent fault-tolerant method based on dynamic migration of distributed components.

[0032] Figure 2 Schematic diagram of the deep learning model structure in an embodiment of the present invention.

[0033] Figure 3 Schematic diagram of the distributed component cluster structure in an embodiment of the present invention.

[0034] Figure 4 Schematic diagram of service component migration in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The method of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0036] like Figure 1 As shown, a flow chart of a cluster intelligent fault-tolerant method based on dynamic migration of distributed components of the present invention, the specific steps are as follows:

[0037] S1. Establish a cluster monitoring and scheduling system, including: a distributed monitoring system and a distributed scheduling system. The distributed monitoring system is used to monitor each node, container, and service component in the distributed component cluster, and collect historical load status data in the cluster at a set frequency (the collection frequency in this embodiment is once per second), i.e., historical monitoring data;

[0038] The distributed monitoring system includes: Prometheus, Zabbix, Nagios, Icinga, Datadog; the historical monitoring data includes: CPU usage, memory usage, hard disk usage, disk I / O, and network bandwidth.

[0039] S2, preprocessing the historical load status data collected in step S1, and constructing a data set for the preprocessed data as input of the deep learning model;

[0040] The preprocessing includes: denoising, cropping, standardization and normalization;

[0041] First, the historical load status data collected in step S1 is cleaned, denoised, and normalized to zero mean, and then standardized and cropped. In this embodiment, the cropping length is set to k = 18000, wherein the first m = 10800 data of each segment of sample data are used as the input of the deep learning model, and the last km of data are used as the output prediction of the model; the constructed data set contains a sample size of n = 7200, the dimension of the data set is n*5*m, and the training set and the validation set are divided according to the ratio set in the actual situation.

[0042] Among them, n represents the number of samples, and the normalized expression is as follows:

[0043]

[0044] Among them, X represents the original data, X * represents the normalized data, μ represents the mean of the original data, and δ represents the standard deviation of the original data. The specific expressions of μ and δ are as follows:

[0045]

[0046] Among them, x i Represents the i-th sample in the original data.

[0047] S3, based on the data set generated in step S2, input the constructed deep learning model for training to obtain a trained deep learning model;

[0048] The deep learning models include: CNN, RNN, LSTM, GRU, Transformer; Figure 2As shown, the model of this embodiment uses a customized multi-input network based on a deep learning neural network, that is, a deep learning network based on CNN+LSTM. The goal of the model is to learn the laws and trends of system load changes, so as to predict the demand for cluster resources in the future. In this embodiment, the data of the previous three hours can be used to predict the status of each node in the next two hours.

[0049] Use the constructed deep learning model to input the data set generated in step S2 for model training, use the model to predict the input data, build an error estimate for the predicted data and the real data, backpropagate the model through the optimizer to update the model parameters, iterate the training until the loss function converges, the model tends to be stable, the error between the predicted data and the real data becomes very small, the verification set has a high accuracy, and the model training stops.

[0050] Among them, the input of the deep learning network based on CNN+LSTM is five types of data, namely CPU usage, memory usage, disk usage, disk I / O, and network bandwidth. The size of each type of data is 1*10800. The network uses 64, 128, and 256 convolutions and normalizations for feature processing on the five types of input data respectively, and the network extraction part is parameter sharing; then, the features are predicted using LSTM with an output of 7200 twice to obtain the feature vectors predicted for the five types of data.

[0051] S4, based on the distributed scheduling system, intelligent monitoring and scheduling of the system status is performed, that is, the real-time monitoring data is collected regularly, pre-processed and input into the model trained in step S3 for inference and prediction, and the node, container and component status are inferred (the prediction frequency in this embodiment is 1 minute);

[0052] The real-time monitoring data includes: CPU usage, memory usage, hard disk usage, disk I / O, and network bandwidth. The real-time monitoring data is predicted to obtain the future status of the real-time monitoring data. According to the future status of each indicator, the resource consumption trend of each node, the resource demand change of the container, and the load demand of the service are obtained to identify possible overloads, resource bottlenecks, or potential failure points in the future.

[0053] S5. Based on the load prediction result obtained in step S4, a fault judgment is performed on the load prediction result. If no fault is found, the process returns to step S4 to continue intelligent monitoring and scheduling of the system status. If a fault is found to be about to occur, the system generates an intelligent scheduling decision and enters step S6.

[0054] The scheduling decision includes: component service scheduling and container scheduling.

[0055] Among them, component service scheduling decides whether to migrate services to other nodes based on the load demand and priority of the services to avoid resource bottlenecks of high-load nodes. Container scheduling decides whether containers need to be migrated to other nodes based on the cluster resource status and the load prediction results of the containers to achieve reasonable resource allocation and load balancing.

[0056] The method of the present invention greatly improves the cluster resource utilization rate. The present invention combines container scheduling and component service scheduling, accurately predicts the cluster load through a deep learning model, and can identify resource bottlenecks and potential load problems in advance, so as to perform intelligent scheduling. The core of load prediction is to use historical data to train the model and understand the dynamic change law of the system load, which enables the cluster to predict and take corresponding scheduling measures before the load surges. The component service scheduling part intelligently adjusts its running nodes according to the load requirements of different services to avoid the resource bottleneck of overloaded nodes; while container scheduling optimizes resource allocation at the container level to ensure that the container always runs on the appropriate node. In this way, the system can avoid overload and resource waste while ensuring efficient resource allocation, and significantly improve the resource utilization rate of the cluster. Compared with the existing static scheduling and manual intervention methods, the intelligent scheduling mechanism of the method of the present invention can dynamically adjust resources according to real-time load, ensure efficient and flexible cluster resource management, and further improve the performance and scalability of the overall system.

[0057] S6. Based on step S5, the scheduling is performed, the container or component service is migrated, and the monitoring and scheduling system goes to step S4 to continue to monitor the resource usage of the cluster, the health status of the service, and the operation status of the container in real time, and at the same time triggers training;

[0058] Through real-time monitoring of the scheduling system, any possible resource bottlenecks, failures or performance degradation can be identified, and scheduling strategies and resource allocation can be adjusted in a timely manner to ensure the stable operation of the cluster.

[0059] S7. Based on step S6, after the training is triggered, the monitoring and scheduling system will organize the historical monitoring data of the recent period, perform data preprocessing to update the data set (incremental learning), and then repeat steps S3-S6 to continuously iterate and optimize the model, thereby ensuring the longer-term stability of the system and realizing intelligent fault tolerance of the cluster.

[0060] Among them, the monitoring and scheduling system continuously updates the model based on feedback, improves load prediction accuracy and scheduling decisions, and enhances the distributed monitoring and scheduling system's ability to adapt to dynamic loads.

[0061] In this embodiment, the cluster distributed component cluster structure is as follows: Figure 3As shown in the figure, its structure is a one-to-many form. The server is deployed on a high-performance server. The intelligent monitoring and scheduling system not only requires GPU for reasoning but also requires GPU resources for incremental learning. Each computing node and the server are connected through a core switch. Multiple containers can be deployed in the computing node, and component services can be deployed in the container.

[0062] In this embodiment, the service component migration is as follows Figure 4 As shown in the figure, the computing nodes in the computing cluster are not necessarily exactly the same, and the resources required by each component and each algorithm are also different. Therefore, during the operation of the computing cluster, the system predicts resource usage through a deep learning model and performs dynamic component migration. It supports changing components on computing nodes during operation and migrating components with high resource usage to computing nodes with relatively low resource usage.

[0063] The intelligent fault-tolerant mechanism of the method of the present invention enhances the high availability and stability of the cluster. By combining container scheduling with component service scheduling and designing a multi-input network based on deep learning for multi-dimensional data fusion, the intelligent fault-tolerant mechanism of the cluster is realized. This enables the system to respond quickly and automatically self-heal when facing node failures, overloads, network delays or other sudden problems. Through the load prediction of the deep learning model, the system can identify load anomalies, potential failures or resource bottlenecks in advance, so as to migrate containers or reschedule services before the problem occurs, avoiding cluster resource overload or service interruption. In the event of a failure, the system will automatically transfer the load to other healthy nodes, maintain the availability of the service, and ensure the continuity of the cluster. Compared with existing fault-tolerant methods, which usually rely on manual intervention or static backup, the intelligent fault-tolerant mechanism can improve the response speed and accuracy, reduce manual intervention and errors, and significantly improve the stability and fault-tolerant capability of the cluster. This self-healing capability not only reduces the time and cost of fault recovery, but also ensures that the system can continue to operate stably in various complex and dynamic environments.

[0064] The automation and intelligence of cluster management of the method of the present invention improves operation and maintenance efficiency. By using deep learning and artificial intelligence methods to dynamically schedule and optimize cluster resources, the automation and intelligence level of cluster management is greatly improved. Existing cluster management systems usually rely on manual configuration and static scheduling, and cannot cope with dynamically changing loads and resource requirements. The method of the present invention automatically generates scheduling decisions through real-time monitoring and load prediction without manual intervention. Based on the intelligent management of container and component service scheduling, the system can dynamically adjust resource allocation, optimize the scheduling position of containers, ensure that each container runs on the most suitable node, and avoid resource waste or overload. This increase in automation allows operation and maintenance personnel to focus more on optimization and innovation at the business level, and no longer be troubled by cumbersome cluster management. In addition, the feedback mechanism of the system enables the prediction model to be continuously optimized, and the scheduling strategy gradually becomes more accurate. Over time, the management efficiency and stability of the cluster will continue to improve. This intelligent and automated cluster resource management method greatly reduces the incidence of human errors, reduces manual intervention and operation and maintenance costs, and improves overall operation and maintenance efficiency.

[0065] In summary, the method of the present invention performs deep learning modeling and prediction on the historical load status data of the cluster, identifies the load change trend in advance, and thus makes accurate scheduling decisions, ensuring the high availability and load balancing of the system, avoiding system overload or service interruption under high load, and achieving more efficient resource allocation and scheduling. Through this cluster self-healing mechanism and predictive scheduling, it can effectively reduce system failure time and ensure service continuity, and can be applied to cloud platforms, microservice architectures, and large-scale distributed systems.

[0066] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A cluster intelligent fault tolerance method based on dynamic migration of distributed components, the specific steps are as follows: S1. Establish a cluster monitoring and scheduling system, including: Distributed monitoring system and distributed scheduling system: monitor each node, container and service component in the distributed component cluster through the distributed monitoring system, and collect historical load status data in the cluster at a set frequency, i.e. historical monitoring data; The distributed monitoring system includes: Prometheus, Zabbix, Nagios, Icinga, Datadog; the historical monitoring data includes: CPU usage, memory usage, hard disk usage, disk I / O, network bandwidth; S2, preprocessing the historical load status data collected in step S1, and constructing a data set for the preprocessed data as input of the deep learning model; The preprocessing includes: denoising, cropping, standardization and normalization; First, the historical load status data collected in step S1 is cleaned, denoised, and normalized to zero mean, and then standardized and cropped. The cropping length is set to k, where the first m data of each sample data segment are used as the input of the deep learning model, and the last km data are used as the output prediction of the model; the constructed data set contains n samples, the dimension of the data set is n*5*m, and the training set and the validation set are divided according to the set ratio; Among them, n represents the number of samples, and the normalized expression is as follows: Among them, X represents the original data, X * represents the normalized data, μ represents the mean of the original data, and δ represents the standard deviation of the original data. The specific expressions of μ and δ are as follows: Among them, x i Represents the i-th sample in the original data; S3, based on the data set generated in step S2, input the constructed deep learning model for training to obtain a trained deep learning model; The deep learning model includes: CNN, RNN, LSTM, GRU, Transformer; the constructed deep learning model is used to input the data set generated in step S2 for model training, the input data is predicted by the model, an error estimate is constructed between the predicted data and the real data, the model is back-propagated by the optimizer to update the model parameters, and the training is iterated until the loss function converges and the model tends to be stable, and the model training stops; S4, intelligent monitoring and scheduling of system status based on the distributed scheduling system, that is, regular collection of real-time monitoring data, pre-processing and input into the model trained in step S3 for inference and prediction, and inferring the status of nodes, containers and components; The real-time monitoring data includes: CPU usage, memory usage, hard disk usage, disk I / O, and network bandwidth; predict the real-time monitoring data to obtain the future status of the real-time monitoring data, and obtain the resource consumption trend of each node, the resource demand change of the container, and the load demand of the service according to the future status of each indicator, so as to identify possible overloads, resource bottlenecks, or potential failure points in the future; S5. Based on the load prediction result obtained in step S4, a fault judgment is performed on the load prediction result. If no fault is found, the process returns to step S4 to continue intelligent monitoring and scheduling of the system status. If a fault is found to be about to occur, the system generates an intelligent scheduling decision and enters step S6. The scheduling decision includes: component service scheduling and container scheduling; Component service scheduling determines whether to migrate services to other nodes based on service load requirements and priorities to avoid resource bottlenecks on high-load nodes. Container scheduling determines whether containers need to be migrated to other nodes based on cluster resource conditions and container load prediction results. S6. Based on step S5, the scheduling is performed, the container or component service is migrated, and the monitoring and scheduling system goes to step S4 to continue to monitor the resource usage of the cluster, the health status of the service, and the operation status of the container in real time, and at the same time triggers training; S7. Based on step S6, after the training is triggered, the monitoring and scheduling system organizes the historical monitoring data of the recent period, performs data preprocessing and updates the data set, and then repeats steps S3-S6 to continuously iterate and optimize the model to ensure the longer-term stability of the system and realize intelligent fault tolerance of the cluster; Among them, the monitoring and scheduling system continuously updates the model based on feedback, improves load prediction accuracy and scheduling decisions, and enhances the distributed monitoring and scheduling system's ability to adapt to dynamic loads.

Images