Intelligent container cluster scheduling method and device

Abstract

The invention relates to the technical field of computers, and particularly provides an intelligent container cluster scheduling method and device, and the method comprises the following steps: S1, a data collection module obtains real-time operation state information from each node in a container cluster; s2, the feature extraction layer converts the original data into a form suitable for machine learning model processing; s3, generating an optimal scheduling strategy based on the large model and the deep learning model; s4, the execution mechanism converts the scheduling instruction generated by the decision engine into an actual operation command, and issues the actual operation command to the target container; s5, continuously improving the performance of the large model by a feedback mechanism; and S6, setting an anomaly detection and recovery mechanism. Compared with the prior art, the problems of unbalanced load, low resource utilization rate, slow response to burst traffic and the like in traditional container cluster scheduling can be solved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and specifically provides a smart container cluster scheduling method and device. Background Art

[0002] As microservice architecture gradually becomes one of the mainstream software development models, container technology has shown great advantages in deployment and operation with its lightweight virtualization characteristics. However, in actual application, how to reasonably manage and schedule containers running on different physical nodes to ensure the optimal performance of the entire cluster and quickly adapt to changes in business needs has become a technical problem that needs to be solved urgently.

[0003] Existing solutions mostly rely on preset rules or static configuration files to make decisions, which lack sufficient flexibility. At the same time, in the face of increasingly complex network environments and diverse workload types, these methods often find it difficult to provide satisfactory performance.

[0004] Therefore, it is necessary to introduce more advanced algorithms and technical means to improve the current situation. Summary of the invention

[0005] The present invention aims to overcome the above-mentioned deficiencies in the prior art and provides a highly practical intelligent container cluster scheduling method.

[0006] A further technical task of the present invention is to provide a smart container cluster scheduling device that is reasonably designed, safe and applicable.

[0007] The technical solution adopted by the present invention to solve its technical problem is:

[0008] The intelligent container cluster scheduling method has the following steps:

[0009] S1, the data collection module obtains real-time operation status information from each node in the container cluster;

[0010] S2, the feature extraction layer converts the raw data into a form suitable for processing by the machine learning model;

[0011] S3, Generate the optimal scheduling strategy based on large models and deep learning models;

[0012] S4. The execution mechanism converts the scheduling instructions generated by the decision engine into actual operation commands and sends them to the target container;

[0013] S5. Feedback mechanism continuously improves the performance of large models;

[0014] S6. Set up anomaly detection and recovery mechanism.

[0015] Furthermore, in step S1, a large amount of indicator data is captured and stored using the Prometheus database. cAdvisor provides detailed container-level monitoring data, and for specific application scenarios or services, customized logging is performed to capture more detailed information.

[0016] Furthermore, in step S2, data cleaning is first performed to remove noise and outliers, statistical methods are used for feature selection, new features are created to enhance the expressiveness of the model, the average load and peak load over a period of time are collected and calculated, and then, data of different scales are converted to the same range, using minimum-maximum scaling and z-score normalization methods;

[0017] Finally, for time series data, it is converted into a sliding window form, and each window is input into the model as a sample.

[0018] Further, in step S3, a reinforcement learning framework is used in combination with a time series prediction model, wherein the reinforcement learning framework includes DQN and PPO;

[0019] The DQN uses the Q-learning algorithm combined with a deep neural network to learn the optimal action strategy by interacting with the environment;

[0020] The PPO is a policy gradient-based method that improves learning efficiency while ensuring exploration;

[0021] The time series prediction model includes LSTM, Transformer and multi-task learning model, and the LSTM is used to capture long-term dependencies in the time series and predict future resource requirements;

[0022] The Transformer processes long sequence data through the self-attention mechanism and is suitable for complex multi-dimensional time series prediction;

[0023] The multi-task learning model includes a shared representation layer and hard parameter sharing.

[0024] Furthermore, the shared representation layer allows multiple tasks to share the same basic network, and then different output layers are added on top for different decision-making tasks;

[0025] The hard parameter sharing means that all tasks share the same parameters but have different loss functions.

[0026] Further, in step S4, the feedback mechanism includes real-time performance monitoring, historical data analysis and user feedback collection;

[0027] The real-time performance monitoring collects real-time performance data by continuously monitoring the operating status of the system;

[0028] The historical data analysis is to regularly analyze historical data, evaluate the effectiveness of scheduling strategies, and identify potential problems;

[0029] The user feedback collection establishes a user feedback channel to collect user experiences and suggestions as part of model training.

[0030] Further, in step S5, the anomaly detection and recovery mechanism is health check, automatic failover, self-healing capability and safety protection;

[0031] The health check is to regularly check the health of the nodes and services in the cluster, and take immediate measures when abnormalities are found;

[0032] The automatic failover means that when a node fails, it can switch to a standby node to ensure uninterrupted service.

[0033] Furthermore, the self-healing capability is to automatically repair some common misconfigurations or software defects, and the security protection is to protect the cluster from attacks through a multi-level security mechanism.

[0034] An intelligent container cluster scheduling device includes: at least one memory and at least one processor;

[0035] The at least one memory is used to store a machine-readable program;

[0036] The at least one processor is used to call the machine-readable program to execute the smart container cluster scheduling method claimed in the claim.

[0037] Compared with the prior art, the intelligent container cluster scheduling method and device of the present invention have the following outstanding beneficial effects:

[0038] (1) Improve resource utilization: By real-time monitoring and analyzing the resource usage in the cluster, the system can automatically adjust resource configuration to ensure that resources are fully utilized. For example, during low-load periods, resources can be concentrated on key tasks; during high-load periods, resources can be dynamically expanded to cope with sudden traffic.

[0039] By using the time series forecasting model, the system can predict future resource requirements in advance and make pre-allocations accordingly, thus avoiding resource waste or shortage.

[0040] (2) Enhanced system response speed: The deep learning-based decision engine can generate the optimal scheduling solution within milliseconds, greatly shortening the time required for traditional static rules or manual intervention;

[0041] Once a scheduling plan is generated, the execution agency will immediately call the Kubernetes API or other container orchestration tools to quickly implement the changes and ensure that the system can quickly adapt to new load conditions.

[0042] (3) Improve system stability and reliability. The system's built-in anomaly detection mechanism can promptly detect and handle various faults, such as node downtime and network interruption, and ensure service continuity through automatic failover and self-healing capabilities.

[0043] The key components adopt redundant design, which can ensure the normal operation of the whole system even if a part fails, and improve the fault tolerance and availability of the system.

[0044] (4) Optimize user experience. Through efficient resource management and scheduling, users' applications can achieve more stable performance and reduce delays and freezes caused by resource bottlenecks.

[0045] The system supports seamless expansion and can easily add new nodes or services according to business needs, meet growing user needs, and provide a better user experience.

[0046] (5) Reduce operation and maintenance costs: Reduce the need for manual intervention, reduce the workload of operation and maintenance personnel, and improve operation and maintenance efficiency.

[0047] Through intelligent resource scheduling, unnecessary resource waste is reduced and the overall operating cost of the cloud computing platform is lowered.

[0048] (6) Enhanced security: The system integrates multiple security mechanisms, including firewalls, intrusion detection systems (IDS), access control lists (ACLs), etc., to protect the cluster from external attacks. Encryption technology is used during data transmission and storage to ensure the security of sensitive information.

[0049] (7) Cross-platform compatibility: The system design supports multiple operating systems and hardware architectures. Whether it is x86, ARM or other architectures, it can be seamlessly integrated, enhancing the applicability and flexibility of the system.

[0050] (8) Continuous optimization: By continuously collecting performance data and user feedback, the system can continuously learn and improve, optimize scheduling strategies, and further improve overall performance. The machine learning model in the decision engine can be regularly updated through online learning to adapt to the ever-changing environment and needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Attached Figure 1 The invention is a flowchart of a method for real-time dynamic analysis of streaming serial port data. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] A best embodiment is given below:

[0055] like Figure 1 As shown, a method for real-time dynamic analysis of streaming serial port data in this embodiment has the following steps:

[0056] S1, the data collection module obtains real-time operation status information from each node in the container cluster;

[0057] This information includes but is not limited to key indicators such as CPU usage, memory usage, disk I / O, network bandwidth, etc. In order to ensure the accuracy and timeliness of data, the following technical means can be used:

[0058] Prometheus: An open source monitoring system and time series database that can efficiently capture and store large amounts of metric data.

[0059] cAdvisor: An open source container resource usage and performance analysis tool that provides detailed container-level monitoring data.

[0060] Customized logging: For specific application scenarios or services, you can capture more detailed information through customized logging.

[0061] The data collection process needs to take into account the consistency and integrity of the data, and reduce the risk of data loss by setting a reasonable collection frequency and buffer mechanism based on the information entropy weight.

[0062] S2, the feature extraction layer converts the raw data into a form suitable for processing by the machine learning model;

[0063] This process usually includes the following steps:

[0064] (1) Data cleaning: remove noise and outliers. Use statistical methods (such as Z-score) to detect and remove data points that deviate from the normal range;

[0065] (2) Feature selection: Determine which indicators are most important for scheduling decisions. Use statistical methods such as correlation analysis and principal component analysis (PCA) to perform feature selection.

[0066] (3) Feature Engineering: Create new features to enhance the expressiveness of the model. Collect and calculate the average load and peak load over a period of time.

[0067] (4) Data standardization: convert data of different scales to the same range, using minimum-maximum scaling and z-score standardization;

[0068] (5) Time series data processing: Time series data can be converted into a sliding window format, with each window being input into the model as a sample.

[0069] S3, Generate the optimal scheduling strategy based on large models and deep learning models;

[0070] A reinforcement learning framework combined with a time series forecasting model was used to achieve this goal.

[0071] Among them, the reinforcement learning framework includes DQN (Deep Q-Network) and PPO (Proximal Policy Optimization);

[0072] DQN (Deep Q-Network) combines the Q-learning algorithm with a deep neural network to learn the optimal action strategy by interacting with the environment;

[0073] PPO (Proximal Policy Optimization) is a policy gradient-based method that can improve learning efficiency while ensuring exploration.

[0074] Time series prediction models include LSTM (Long Short-Term Memory), Transformer, and multi-task learning models. LSTM (Long Short-Term Memory) is used to capture long-term dependencies in time series and predict future resource requirements.

[0075] Transformer processes long sequence data through the self-attention mechanism, which is particularly suitable for complex multi-dimensional time series prediction;

[0076] The multi-task learning model includes a shared representation layer and hard parameter sharing. The shared representation layer shares the same basic network for multiple tasks, and then adds different output layers on top for different decision-making tasks.

[0077] Hard parameter sharing shares the same parameters for all tasks but has different loss functions.

[0078] S4. The execution mechanism converts the scheduling instructions generated by the decision engine into actual operation commands and sends them to the target container, including:

[0079] Kubernetes API call: Dynamically adjust the deployment location, quantity, and resource configuration of Pods by calling the Kubernetes API.

[0080] Automation scripts: Write shell scripts or other types of scripts to perform specific tasks, such as restarting containers, migrating data, etc.

[0081] External system integration: Interact with external services such as load balancers and storage systems to achieve global optimization.

[0082] S5. Feedback mechanism continuously improves the performance of large models;

[0083] Includes the following:

[0084] Real-time performance monitoring: Collect real-time performance data by continuously monitoring the system's operating status.

[0085] Historical data analysis: Regularly analyze historical data to evaluate the effectiveness of scheduling strategies and identify potential problems.

[0086] User feedback collection: Establish user feedback channels to collect user experiences and suggestions as part of model training.

[0087] S6. Set up anomaly detection and recovery mechanisms;

[0088] In order to ensure high availability and stability of the system, the present invention also includes a complete set of anomaly detection and recovery mechanisms:

[0089] Health check: Regularly perform health checks on nodes and services in the cluster, and take immediate action if an abnormality is found.

[0090] Automatic failover: When a node fails, the system can quickly switch to the backup node to ensure uninterrupted service.

[0091] Self-healing capability: The system has a certain degree of self-healing capability and can automatically repair some common misconfigurations or software defects.

[0092] Security protection: The cluster is protected from attacks through multi-level security mechanisms, including firewalls, intrusion detection systems (IDS), access control lists (ACLs), etc.

[0093] Based on the above method, the intelligent container cluster scheduling device in this embodiment includes: at least one memory and at least one processor;

[0094] The at least one memory is used to store a machine-readable program;

[0095] The at least one processor is used to call the machine-readable program to execute the smart container cluster scheduling method claimed in the claim.

[0096] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0097] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent container cluster scheduling method, characterized in that: The steps are as follows: S1, the data collection module obtains real-time operation status information from each node in the container cluster; S2, the feature extraction layer converts the raw data into a form suitable for processing by the machine learning model; S3, Generate the optimal scheduling strategy based on large models and deep learning models; S4. The execution mechanism converts the scheduling instructions generated by the decision engine into actual operation commands and sends them to the target container; S5. Feedback mechanism continuously improves the performance of large models; S6. Set up anomaly detection and recovery mechanism.

2. The intelligent container cluster scheduling method and device according to claim 1, characterized in that: In step S1, the Prometheus database is used to capture and store a large amount of indicator data. cAdvisor provides detailed container-level monitoring data. For specific application scenarios or services, customized logging is performed to capture more detailed information.

3. The intelligent container cluster scheduling method according to claim 1, characterized in that: In step S2, data cleaning is first performed to remove noise and outliers, and statistical methods are used for feature selection to create new features to enhance the expressiveness of the model. The average load and peak load over a period of time are collected and calculated. Then, data of different scales are converted to the same range, and minimum-maximum scaling and z-score normalization are used; Finally, for time series data, it is converted into a sliding window form, and each window is input into the model as a sample.

4. The intelligent container cluster scheduling method according to claim 3, characterized in that: In step S3, a reinforcement learning framework is used in combination with a time series prediction model, wherein the reinforcement learning framework includes DQN and PPO; The DQN uses the Q-learning algorithm combined with a deep neural network to learn the optimal action strategy by interacting with the environment; The PPO is a policy gradient-based method that improves learning efficiency while ensuring exploration; The time series prediction model includes LSTM, Transformer and multi-task learning model, and the LSTM is used to capture long-term dependencies in the time series and predict future resource requirements; The Transformer processes long sequence data through the self-attention mechanism and is suitable for complex multi-dimensional time series prediction; The multi-task learning model includes a shared representation layer and hard parameter sharing.

5. The intelligent container cluster scheduling method according to claim 4, characterized in that: The shared representation layer allows multiple tasks to share the same basic network, and then different output layers are added on top for different decision-making tasks; The hard parameter sharing means that all tasks share the same parameters but have different loss functions.

6. The intelligent container cluster scheduling method according to claim 5, characterized in that: In step S4, the feedback mechanism includes real-time performance monitoring, historical data analysis and user feedback collection; The real-time performance monitoring collects real-time performance data by continuously monitoring the operating status of the system; The historical data analysis is to regularly analyze historical data, evaluate the effectiveness of scheduling strategies, and identify potential problems; The user feedback collection establishes a user feedback channel to collect user experiences and suggestions as part of model training.

7. The intelligent container cluster scheduling method according to claim 5, characterized in that: In step S5, the anomaly detection and recovery mechanism is health check, automatic failover, self-healing capability and safety protection; The health check is to regularly check the health of the nodes and services in the cluster, and take immediate measures when abnormalities are found; The automatic failover means that when a node fails, it can switch to a standby node to ensure uninterrupted service.

8. The intelligent container cluster scheduling method according to claim 7, characterized in that: The self-healing capability is to automatically repair some common misconfigurations or software defects, and the security protection is to protect the cluster from attacks through a multi-level security mechanism.

9. Intelligent container cluster scheduling device, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 8.

Images