A configuration method, device and storage medium for improving k8s cluster stability
Through fault simulation playback and reinforcement learning algorithms, a stability guarantee recommendation strategy is generated, which solves the problem of inaccurate configuration after failure drills in Kubernetes cluster management, and realizes automatic adjustment of cluster resources and configuration and improves stability.
Patent Information
- Application Number
- CN202211718354.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-30
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-12-30
AI Technical Summary
The existing Kubernetes cluster management technology lacks efficient flexibility and accuracy of resource allocation after the failure drill. It relies on human configuration and makes it difficult to achieve automation and accuracy of cluster adjustments.
By collecting fault event information, filtering and delivering it to the fault simulation cluster for fault simulation playback, generating fault distribution models, formulating stability guarantee recommendation strategies, and optimizing configuration through reinforcement learning algorithms to achieve automated adjustment of resources and configuration.
It improves the flexibility and accuracy of the stability configuration of the Kubernetes cluster, reduces the dependence on human configuration, realizes rapid automatic adjustment of resources and configuration, and improves the failure coverage rate and stability acceptance capabilities.
Smart Images

Figure CN115883319B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of cloud platform optimization, and in particular to a configuration method, a device and a storage medium thereof for improving the stability of a k8s cluster. Background Art
[0002] Fault injection is a reliability verification technology that deliberately introduces faults into the system through controlled experiments and observes the behavior of the system when faults exist. Among them, Kubernetes (abbreviated as k8s, with 8 replacing the 8 characters "ubernete" in the middle of the name) cluster fault injection technology is implemented through a hybrid fault injection technology based on hardware and software, which is reflected by injecting faults into hardware resources and networks such as the host machine where the cluster is located, and injecting faults into software such as container services in the cluster. However, many of the existing Kubernetes cluster management technologies only use fault injection capabilities to perform fault drills, but the data after the drill is more used as a human reference to adjust the configuration of the Kubernetes cluster, lacking the high efficiency and flexibility of cluster adjustment and the accuracy of resource configuration. Summary of the invention
[0003] In order to improve the flexibility and accuracy of Kubernetes cluster configuration and reduce the dependence on manual configuration, the present invention proposes a configuration method for improving the stability of k8s cluster, comprising the steps of:
[0004] S1: Collect and report the failure event information that occurs during the stability verification of the k8s cluster injection system, and filter and record it according to the failure type;
[0005] S2: Place the target fault into the fault simulation cluster of the corresponding classification according to the fault type, and run each fault simulation cluster to perform fault simulation playback;
[0006] S3: Generate a cluster fault distribution model based on the cluster fault stability report accumulated during the simulation playback process;
[0007] S4: Generate a stability assurance recommendation strategy based on the cluster failure distribution model;
[0008] S5: Send the recommended stability assurance strategy to the corresponding fault simulation cluster to adjust resources and configuration.
[0009] Furthermore, the k8s cluster contains several fault simulation clusters classified according to fault types.
[0010] Furthermore, in the step S1, the fault event information includes the time of occurrence of the fault, cluster node configuration, CPU, memory, disk, number of containers, network bandwidth and fault type.
[0011] Furthermore, in the step S3, the cluster fault stability report includes two parts: cluster configuration information and stability information.
[0012] Furthermore, the cluster configuration information includes: the number of cluster control nodes, the number of cluster computing nodes, the node operating system, CPU size, memory size, cluster network bandwidth configuration, cluster disk capacity and quantity, and cluster real-time resource occupancy.
[0013] Furthermore, the stability information includes: fault occurrence time, fault type, fault occurrence frequency, and fault occurrence range.
[0014] Furthermore, in the step S3, the cluster fault distribution model is obtained by constructing a mapping between cluster configuration information, stability information and cluster stability configuration strategy through a generative adversarial network learning method.
[0015] Furthermore, in the step S4, the stability assurance recommendation strategy is obtained by optimizing the configuration recommendation indicators through a reinforcement learning algorithm, and continuously verifying the configuration recommendation strategy through fault simulation exercises.
[0016] Furthermore, in the step S3, if a new fault event is generated during the fault simulation playback, the step S3 further includes:
[0017] S31: Report the new fault time to the management end and add it to the simulated fault candidate list.
[0018] Furthermore, in the step S3, before the fault simulation is played back, a custom simulated fault of the same category may be added to the fault simulation cluster by manual addition.
[0019] The present invention also proposes a configuration device for improving the stability of a k8s cluster, comprising:
[0020] The fault screening module is used to collect and report the fault event information that occurs during the stability verification process of the k8s cluster injection system, and filter and record it according to the fault type;
[0021] The simulation playback module is used to place the target fault into the fault simulation cluster of the corresponding classification according to the fault type, and run each fault simulation cluster to perform fault simulation playback;
[0022] A model training module is used to generate a cluster fault distribution model based on the cluster fault stability reports accumulated during the simulation playback process;
[0023] The strategy update module is used to generate stability assurance recommendation strategies based on the cluster failure distribution model;
[0024] The policy delivery module is used to deliver stability assurance recommended policies to the corresponding fault simulation cluster to adjust resources and configurations.
[0025] Furthermore, in the model training module, the cluster fault stability report includes two parts: cluster configuration information and stability information.
[0026] Furthermore, the cluster configuration information includes: the number of cluster control nodes, the number of cluster computing nodes, the node operating system, CPU size, memory size, cluster network bandwidth configuration, cluster disk capacity and quantity, and cluster real-time resource occupancy.
[0027] Furthermore, the stability information includes: fault occurrence time, fault type, fault occurrence frequency, and fault occurrence range.
[0028] Furthermore, in the model training module, the cluster fault distribution model is obtained by constructing a mapping between cluster configuration information, stability information and cluster stability configuration strategy through a generative adversarial network learning method.
[0029] Furthermore, in the strategy update module, the stability assurance recommendation strategy is obtained by optimizing the configuration recommendation indicators through a reinforcement learning algorithm, and continuously verifying the configuration recommendation strategy through fault simulation exercises.
[0030] Furthermore, in the model training module, if a new fault event is generated during the fault simulation playback, a fault candidate module is also included, which is used to report the new fault time to the management end and add it to the simulated fault candidate list.
[0031] Furthermore, in the simulation playback module, before the fault simulation playback, a custom simulated fault of the same category can be added to the fault simulation cluster by manual addition.
[0032] The present invention also includes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the configuration method described above are implemented.
[0033] The present invention also includes a device for processing data, comprising:
[0034] a storage device having a computer program stored thereon;
[0035] A processor is used to execute the computer program in the memory to implement the steps of the configuration method described above.
[0036] Compared with the prior art, the present invention has at least the following beneficial effects:
[0037] (1) The configuration method, device and storage medium for improving the stability of a k8s cluster described in the present invention, when a basic Kubernetes cluster is injected, collects and reports fault events, and updates the stability assurance recommendation strategy by analyzing the cluster fault step-by-step model, thereby quickly and automatically adjusting the resources and configuration of the cluster;
[0038] (2) Improve the stability acceptance coverage of Kubernetes clusters, automatically record Kubernetes cluster failures that have occurred, and generate failure simulation scenarios for regression;
[0039] (3) Build a Kubernetes application cluster stability configuration strategy recommendation model to provide an important reference data foundation for certain types of clusters that need to provide more precise stability configurations for resources such as bandwidth, nodes or storage space, as well as configurations such as container orchestration and network orchestration. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 This is a schematic diagram of the steps of a configuration method to improve the stability of the k8s cluster;
[0041] Figure 2 A module diagram of a configuration device for improving the stability of a k8s cluster. DETAILED DESCRIPTION
[0042] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to these embodiments.
[0043] Embodiment 1
[0044] Considering that conventional Kubernetes cluster fault drills usually involve interactions between multiple applications and complex drill steps, it is difficult and inefficient for test operation and maintenance personnel to design and accurately simulate drill scenarios, and high-frequency scenarios cannot be guaranteed to be covered. The present invention proposes a configuration method for improving the stability of the k8s cluster, which is applied to the edge container cluster ECK. The generated recommended configuration strategy is sent to the corresponding Kubernetes cluster for adjustment to meet the precise adjustment of resource allocation and container and network orchestration methods, so as to maximize the rationalization of resource utilization, and enable ECK users to enjoy the most stable cluster operation guarantee service at the lowest cost, such as Figure 1 As shown, it specifically includes the following steps:
[0045] S1: Collect and report the failure event information that occurs during the stability verification of the k8s cluster injection system, and filter and record it according to the failure type;
[0046] S2: Place the target fault into the fault simulation cluster of the corresponding classification according to the fault type, and run each fault simulation cluster to perform fault simulation playback;
[0047] S3: Generate a cluster fault distribution model based on the cluster fault stability report accumulated during the simulation playback process;
[0048] S4: Generate a stability assurance recommendation strategy based on the cluster failure distribution model;
[0049] S5: Send the recommended stability assurance strategy to the corresponding fault simulation cluster to adjust resources and configuration.
[0050] If a new fault event occurs during the fault simulation playback, the following steps are included after step S3:
[0051] S31: Report the new fault time to the management end and add it to the simulated fault candidate list.
[0052] In the present invention, when the system performs stability verification on the injected basic Kubernetes cluster (different fault simulation clusters under different categories), and a fault event occurs during the verification process, the monitoring agent is injected into the cluster to realize the fault data monitoring and reporting system capabilities. Then the relevant information of the fault event (fault event information includes fault occurrence time, cluster node configuration, CPU, memory, disk, number of containers, network bandwidth and fault type, etc.) will be automatically reported to the background, and the records will be filtered according to the corresponding fault type. Basic fault classification includes process failure, network attack, container failure, file system injection, kernel failure, clock offset, stress test, etc.
[0053] Then, according to the type of fault event, it is placed in the fault simulation cluster of the corresponding classification, and the simulated fault cluster that needs to be replayed is selected for fault simulation playback. According to the specified fault simulation cluster, a fault resource list will be created in the form of custom resources and sent to the cluster. After the resource controller on the cluster monitors the resource change, the corresponding resource will be created to perform a fault drill. The simulation playback will first build a basic fault simulation scenario, including process failure, network attack, container failure, file system injection, kernel failure, clock offset, stress test and other scenarios. For each type of scenario, corresponding fault parameters can be set. For example, the process failure is simulated by the kill parameter to terminate or pause the process; the network route and flow control are modified by using tools such as iptables, ipsets, tc to simulate network failure. From the perspective of fault drive, the present invention mines the hidden faults in the cluster architecture, link and system through data mining, and derives the fault events that fit the business scenario based on the online historical fault data analysis, so as to realize the automatic arrangement of the fault simulation scenario workflow for the basic fault.
[0054] It should be noted that in order to further improve the stability testing capability of the Kubernetes cluster for target faults, you can manually add custom simulated faults of the same category to the fault simulation cluster before replaying the fault simulation.
[0055] The cluster fault distribution model is generated based on the cluster fault stability report accumulated during the playback process, where the cluster fault stability report includes two parts: cluster configuration information and stability information. The cluster configuration information includes: the number of cluster control nodes, the number of cluster computing nodes, the node operating system, the CPU size, the memory size, the cluster network bandwidth configuration, the cluster disk capacity and quantity, the cluster real-time resource occupancy and other data. The stability information includes: the time of failure, the type of failure, the frequency of failure, the scope of failure and other contents. Specifically, the cluster fault distribution model is obtained by constructing a mapping between cluster configuration information, stability information and cluster stability configuration strategy through the generative adversarial network learning method.
[0056] Based on the cluster fault distribution model, cluster fault data is input into the recommendation model to obtain the recommended configuration strategy, which can be sent to the specified cluster. After receiving the configuration change, the configuration monitoring component on the cluster actively applies for resource changes to the cluster base. In this way, the configuration recommendation indicators are optimized through the reinforcement learning algorithm, and the configuration recommendation strategy is continuously verified through fault simulation exercises, thereby obtaining the stability guarantee recommendation strategy.
[0057] It is not difficult to see that the present invention is based on the principle of fault injection. It realizes rapid automatic reporting of Kubernetes cluster faults and rule screening and generation, generates fault simulation scenarios and customized simulated fault designs, improves the types of simulated faults for edge computing ECK services, and focuses on recording faults of Kubernetes clusters and generating simulation scenarios and customized fault scenarios. Finally, through cluster fault analysis, it outputs the recommended strategies for resources such as network bandwidth, storage space, number of nodes, etc. required to meet the stability requirements of the Kubernetes cluster, as well as optimizing configurations such as container orchestration and network orchestration.
[0058] Embodiment 2
[0059] In order to better understand the technical content of the present invention, this embodiment describes the present invention in the form of a system and mechanism, such as Figure 2 As shown, a configuration device for improving the stability of a k8s cluster includes:
[0060] The fault screening module is used to collect and report the fault event information that occurs during the stability verification process of the k8s cluster injection system, and filter and record it according to the fault type;
[0061] The simulation playback module is used to place the target fault into the fault simulation cluster of the corresponding classification according to the fault type, and run each fault simulation cluster to perform fault simulation playback;
[0062] A model training module is used to generate a cluster fault distribution model based on the cluster fault stability reports accumulated during the simulation playback process;
[0063] The strategy update module is used to generate stability assurance recommendation strategies based on the cluster failure distribution model;
[0064] The policy delivery module is used to deliver stability assurance recommended policies to the corresponding fault simulation cluster to adjust resources and configurations.
[0065] Furthermore, in the model training module, the cluster failure stability report includes two parts: cluster configuration information and stability information.
[0066] Furthermore, in the model training module, the cluster fault distribution model is obtained by constructing a mapping between cluster configuration information, stability information and cluster stability configuration strategy through a generative adversarial network learning method.
[0067] Furthermore, in the strategy update module, the stability assurance recommendation strategy is obtained by optimizing the configuration recommendation indicators through a reinforcement learning algorithm, and continuously verifying the configuration recommendation strategy through fault simulation drills.
[0068] Furthermore, in the model training module, if a new fault event is generated during the fault simulation playback, a fault candidate module is also included to report the new fault time to the management end and add it to the simulated fault candidate list.
[0069] Furthermore, in the simulation playback module, before the fault simulation playback, a custom simulated fault of the same category can be added to the fault simulation cluster by manual addition.
[0070] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the configuration method described in the optional embodiment when the program is executed by a processor.
[0071] The present disclosure also provides a device for processing data, comprising:
[0072] a storage device having a computer program stored thereon;
[0073] A processor is used to execute the computer program in the memory to implement the steps of the configuration method of the optional embodiment.
[0074] In summary, the configuration method, device and storage medium for improving the stability of the k8s cluster described in the present invention, when the basic Kubernetes cluster is injected, collects and reports fault events, and updates the stability assurance recommended strategy by analyzing the cluster fault step-by-step model, thereby quickly and automatically adjusting the resources and configuration of the cluster.
[0075] Improve the stability acceptance coverage of Kubernetes clusters, automatically record the Kubernetes cluster failures that have occurred, and generate failure simulation scenarios for regression. Build a Kubernetes application cluster stability configuration strategy recommendation model to provide important reference data foundation for clusters of certain types that need to provide more accurate stability configurations for resources such as bandwidth, nodes or storage space, as well as configurations such as container orchestration and network orchestration.
[0076] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.
[0077] In addition, in the present invention, descriptions such as "first", "second", "one", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0078] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0079] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.
Claims
1. A configuration method for improving the stability of a k8s cluster, wherein k8s is the abbreviation of Kubernetes, and is characterized in that: Includes steps: S1: Collect and report the failure event information that occurs during the stability verification of the k8s cluster injection system, and filter and record it according to the failure type; S2: Place the target fault into the fault simulation cluster of the corresponding classification according to the fault type, and run each fault simulation cluster to perform fault simulation playback; S3: Generate a cluster fault distribution model based on the cluster fault stability reports accumulated during the simulation playback process; in the step S3, the cluster fault distribution model is obtained by constructing a mapping between cluster configuration information, stability information and cluster stability configuration strategy by using a generative adversarial network learning method; S4: Generate a stability assurance recommendation strategy based on the cluster failure distribution model; Based on the cluster fault distribution model, cluster fault data is input into the recommendation model to obtain the recommended configuration strategy, which can be configured to the specified cluster. After receiving the configuration change, the configuration monitoring component on the cluster actively applies for resource changes to the cluster base. In the step S4, the stability guarantee recommendation strategy is obtained by optimizing the configuration recommendation index through the reinforcement learning algorithm and continuously verifying the configuration recommendation strategy through fault simulation exercises. S5: Send the recommended stability assurance strategy to the corresponding fault simulation cluster to adjust resources and configuration.
2. A configuration method for improving the stability of a k8s cluster as claimed in claim 1, characterized in that: The k8s cluster contains several fault simulation clusters classified according to fault types.
3. A configuration method for improving the stability of a k8s cluster as claimed in claim 1, characterized in that: In the S1 step, the fault event information includes the time of fault occurrence, cluster node configuration, CPU, memory, disk, number of containers, network bandwidth and fault type.
4. A configuration method for improving the stability of a k8s cluster as claimed in claim 1, characterized in that: In the step S3, the cluster failure stability report includes two parts: cluster configuration information and stability information.
5. A configuration method for improving the stability of a k8s cluster as described in claim 4, characterized in that: The cluster configuration information includes: the number of cluster control nodes, the number of cluster computing nodes, the node operating system, the CPU size, the memory size, the cluster network bandwidth configuration, the cluster disk capacity and quantity, and the cluster real-time resource occupancy.
6. A configuration method for improving the stability of a k8s cluster as claimed in claim 4, characterized in that: The stability information includes: fault occurrence time, fault type, fault occurrence frequency, and fault occurrence range.
7. A configuration method for improving the stability of a k8s cluster as claimed in claim 1, characterized in that: In the step S3, if a new fault event is generated during the fault simulation playback, the following steps are also included: S31: Report the new fault time to the management end and add it to the simulated fault candidate list.
8. A configuration method for improving the stability of a k8s cluster as claimed in claim 1, characterized in that: In the step S3, before the fault simulation is played back, a custom simulated fault of the same category may be added to the fault simulation cluster by manual addition.
9. A configuration device for improving the stability of a k8s cluster, wherein k8s is the abbreviation of Kubernetes, characterized in that: include: The fault screening module is used to collect and report the fault event information that occurs during the stability verification process of the k8s cluster injection system, and filter and record it according to the fault type; The simulation playback module is used to place the target fault into the fault simulation cluster of the corresponding classification according to the fault type, and run each fault simulation cluster to perform fault simulation playback; A model training module is used to generate a cluster fault distribution model based on the cluster fault stability reports accumulated during the simulation playback process; in the model training module, the cluster fault distribution model is obtained by constructing a mapping between cluster configuration information, stability information and cluster stability configuration strategy by using a generative adversarial network learning method; The strategy update module is used to generate stability assurance recommendation strategies based on the cluster failure distribution model; Based on the cluster fault distribution model, cluster fault data is input into the recommendation model to obtain the recommended configuration strategy, which can be configured to the specified cluster. After receiving the configuration change, the configuration monitoring component on the cluster actively applies for resource changes to the cluster base. The stability guarantee recommendation strategy is obtained by optimizing the configuration recommendation indicators through the reinforcement learning algorithm and continuously verifying the configuration recommendation strategy through fault simulation exercises. The policy delivery module is used to deliver stability assurance recommended policies to the corresponding fault simulation cluster to adjust resources and configurations.
10. A configuration device for improving the stability of a k8s cluster as claimed in claim 9, characterized in that: In the model training module, the cluster failure stability report includes two parts: cluster configuration information and stability information.
11. A configuration device for improving the stability of a k8s cluster as claimed in claim 10, characterized in that: The cluster configuration information includes: the number of cluster control nodes, the number of cluster computing nodes, the node operating system, the CPU size, the memory size, the cluster network bandwidth configuration, the cluster disk capacity and quantity, and the cluster real-time resource occupancy.
12. A configuration device for improving the stability of a k8s cluster as claimed in claim 10, characterized in that: The stability information includes: fault occurrence time, fault type, fault occurrence frequency, and fault occurrence range.
13. A configuration device for improving the stability of a k8s cluster as claimed in claim 9, characterized in that: In the model training module, if a new fault event is generated during the fault simulation playback, a fault candidate module is also included to report the new fault time to the management end and add it to the simulated fault candidate list.
14. A configuration device for improving the stability of a k8s cluster as claimed in claim 9, characterized in that: In the simulation playback module, before the fault simulation playback, a custom simulated fault of the same category can be added to the fault simulation cluster by manual addition.
15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the configuration method described in any one of claims 1 to 8 are implemented.
16. A device for processing data, characterized in that: include: a storage device having a computer program stored thereon; A processor, configured to execute the computer program in the storage to implement the steps of the configuration method according to any one of claims 1 to 8.
