Cluster project automatic operation and maintenance method, device, storage medium, and computer equipment

Abstract

The cluster project automatic operation and maintenance method, device, storage medium, and computer equipment provided in the present application can first obtain the aggregated analysis data of each workload in each cluster project before automatically operating and maintaining the cluster project in the current operation and maintenance cycle, so as to determine the operation and maintenance data of each workload, and then analyze the operation and maintenance data to determine the target workloads that need to be adjusted in each workload, as well as the intermediate cluster projects where each target workload is located. Then, based on the operation and maintenance project group obtained by the user adjusting the scope of the cluster project for automatic operation and maintenance in the previous operation and maintenance cycle, the target cluster projects that need to be automatically operated and maintained can be screened out from all intermediate cluster projects to form a set of projects to be operated and maintained, and the workload corresponding to each target cluster project in the set of projects to be operated and maintained can be automatically adjusted in turn to improve the stability of the overall cluster service and realize automatic operation and maintenance of each of the cluster projects.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, storage medium, and computer equipment for automatic operation and maintenance of a cluster project. Background Art

[0002] With the widespread adoption of internet technology, container technology has become a mainstream microservices implementation solution for Paas (Platform as a Service), and Kubernetes clusters (K8S clusters) are a leading container orchestration and management technology. Each node in a cluster has limited CPU and memory resources. As more and more microservices run in K8S clusters, it is crucial to properly apply for resources and quotas, and to regularly optimize resources for each service in the K8S cluster.

[0003] Currently, in the cloud-native field, a K8S cluster may have thousands of microservices. When services are deployed to the corresponding K8S cluster, the usage of resources such as CPU and memory will be estimated for different services, and the initial requests and limits of the CPU and memory resources of the services will be set. Excessively large or too small requests and limits in the cluster will affect the stability of the overall cluster service. However, as factors such as business, demand, or business traffic change or adjust, the actual resource usage of each microservice will change, resulting in an unreasonable state between the requests and limits values ​​and the current service resource usage, which in turn affects the stability of the overall cluster service. Summary of the Invention

[0004] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the technical defect in the prior art that the resource initialization requests and limits values ​​of the CPU and memory of each microservice in the cluster form an unreasonable state with the current service resource occupancy, thereby affecting the stability of the overall cluster service.

[0005] This application provides a cluster project automatic operation and maintenance method, the method comprising:

[0006] During the current maintenance cycle, the aggregated analysis data for each workload in each cluster project is obtained, as well as the maintenance project group obtained by adjusting the automatic maintenance cluster project scope based on the aggregated analysis data of the previous maintenance cycle;

[0007] Determine the operation and maintenance data for each workload based on the aggregated analytical data for each workload;

[0008] Analyze the operation and maintenance data of each workload to determine multiple target workloads that need to be adjusted within each workload, as well as the intermediate cluster projects where each target workload is located;

[0009] According to the operation and maintenance project group, target cluster projects that need automatic operation and maintenance are screened out from all intermediate cluster projects to form a set of projects to be operated and maintained;

[0010] According to the operation and maintenance data of each workload, the target workload corresponding to each target cluster project in the set of projects to be operated and maintained is automatically adjusted in turn to achieve automatic operation and maintenance of each cluster project.

[0011] Optionally, obtaining aggregated analysis data of each workload in each cluster project during the current operation and maintenance cycle includes:

[0012] During the current operation and maintenance cycle, obtain the average CPU resource, maximum CPU resource, average memory resource, and maximum memory resource of each workload in each cluster project;

[0013] According to the preset adjustment coefficient, the CPU resource average value, CPU resource maximum value, memory resource average value and memory resource maximum value of each workload are adjusted to obtain the CPU resource request value corresponding to the CPU resource average value, the CPU resource limit value corresponding to the CPU resource maximum value, the memory resource request value corresponding to the memory resource average value and the memory resource limit value corresponding to the memory resource maximum value, and form the aggregated analysis data of each workload.

[0014] Optionally, obtaining the average CPU resource value, the maximum CPU resource value, the average memory resource value, and the maximum memory resource value of each workload in each cluster project during the current operation and maintenance cycle includes:

[0015] During the current operation and maintenance cycle, the performance indicators of each workload in each cluster project are obtained, including the CPU resource indicators and memory resource indicators corresponding to each workload;

[0016] Convert the data format of the CPU resource indicators of each workload to obtain the average CPU resource and maximum CPU resource of each workload;

[0017] The data format of the memory resource indicator of each workload is converted to obtain the average memory resource and the maximum memory resource of each workload.

[0018] Optionally, determining the operation and maintenance data of each workload based on the aggregated analysis data includes:

[0019] Obtaining persistent data for each workload in the previous operation and maintenance cycle, wherein the persistent data is data obtained by secondary aggregation of aggregated analysis data for each workload in the previous operation and maintenance cycle;

[0020] Integrate the persistent data of each workload with the aggregated analysis data of each workload to obtain the operation and maintenance data of each workload.

[0021] Optionally, the operation and maintenance project group is used to select target cluster projects that require automatic operation and maintenance from all intermediate cluster projects to form a set of projects to be operated and maintained, including:

[0022] After determining the intermediate cluster projects that are identical to the cluster projects in the operation and maintenance project group among all the intermediate cluster projects, and taking the determined intermediate cluster projects as target cluster projects that require automatic operation and maintenance, a set of projects to be operated and maintained is formed.

[0023] Optionally, the automatically adjusting the target workload corresponding to each target cluster project in the set of projects to be operated and maintained according to the operation and maintenance data of each workload in sequence includes:

[0024] For the target workload corresponding to each target cluster project in the set of projects to be maintained, the operation and maintenance data of the target workload is obtained, and the operation and maintenance data is analyzed to determine the performance index to be adjusted for the target workload, and the performance index is automatically adjusted.

[0025] Optionally, the method further includes:

[0026] After the automatic operation and maintenance of each cluster project in the current operation and maintenance cycle is completed, the performance indicator of the target workload corresponding to each target cluster project in the set of projects to be operated and maintained is updated.

[0027] This application also provides a cluster project automatic operation and maintenance device, including:

[0028] The aggregated analysis data acquisition module is used to obtain the aggregated analysis data of each workload in each cluster project during the current operation and maintenance cycle, as well as the operation and maintenance project group obtained by adjusting the scope of the automatic operation and maintenance cluster project based on the aggregated analysis data of the previous operation and maintenance cycle;

[0029] an operation and maintenance data acquisition module, configured to determine operation and maintenance data for each workload based on the aggregated analysis data;

[0030] The data analysis module is used to analyze the operation and maintenance data of each workload, determine the multiple target workloads that need to be adjusted in each workload, and the intermediate cluster project where each target workload is located;

[0031] A data screening module is used to screen target cluster projects that need automatic operation and maintenance from all intermediate cluster projects according to the operation and maintenance project group to form a set of projects to be operated and maintained;

[0032] The automatic operation and maintenance module is used to automatically adjust the target workload corresponding to each target cluster project in the set of projects to be operated and maintained according to the operation and maintenance data of each workload, so as to realize automatic operation and maintenance of each cluster project.

[0033] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the cluster project automatic operation and maintenance method as described in any of the above embodiments.

[0034] The present application also provides a computer device, comprising: one or more processors, and a memory;

[0035] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the cluster project automatic operation and maintenance method as described in any one of the above embodiments are performed.

[0036] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0037] The automatic operation and maintenance method, device, storage medium, and computer equipment of cluster projects provided by the present application can first obtain the aggregated analysis data of each workload in each cluster project before the automatic operation and maintenance of the cluster project in the current operation and maintenance cycle, so as to determine the operation and maintenance data of each workload, and then analyze the operation and maintenance data to determine the actual resource usage in the cluster project, and obtain whether the performance indicators of each workload and the current service resource occupancy are reasonable. If not, adjustments need to be made. After determining the target workloads that need to be adjusted in each workload, and the intermediate cluster projects where each target workload is located, the present application can obtain the operation and maintenance project group obtained by the user adjusting the range of the automatic operation and maintenance cluster projects in the previous operation and maintenance cycle, and screen out the target cluster projects that need to be automatically operated and maintained from all intermediate cluster projects to form a set of projects to be operated and maintained, and automatically adjust the target workload corresponding to each target cluster project in the set of projects to be operated and maintained in turn according to the operation and maintenance data of each workload, so as to improve the stability of the overall cluster service and realize the automatic operation and maintenance of each cluster project. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 A flowchart of a cluster project automatic operation and maintenance method provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of the structure of a cluster project automatic operation and maintenance method provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0043] With the widespread adoption of internet technology, container technology has become a mainstream microservices implementation solution for PaaS (PaaS), and Kubernetes clusters are a leading provider of container orchestration and management technologies. Each node in a cluster has limited CPU and memory resources. As more microservices run in Kubernetes clusters, it's crucial to rationally apply for resources and quotas, and to regularly optimize resources for each service within the Kubernetes cluster.

[0044] Currently, in the cloud-native field, a K8S cluster may have thousands of microservices. When services are deployed to the corresponding K8S cluster, the usage of resources such as CPU and memory will be estimated for different services, and the initial requests and limits of the CPU and memory resources of the services will be set. Excessively large or too small requests and limits in the cluster will affect the stability of the overall cluster service. However, as factors such as business, demand, or business traffic change or adjust, the actual resource usage of each microservice will change, resulting in an unreasonable state between the requests and limits values ​​and the current service resource usage, which in turn affects the stability of the overall cluster service.

[0045] Therefore, the purpose of this application is to solve the technical problem in the prior art that the resource initialization requests and limits of the CPU and memory of each microservice in the cluster form an unreasonable state with the current service resource usage, thereby affecting the stability of the overall cluster service, and propose the following technical solutions:

[0046] In one embodiment, Figure 1 As shown, Figure 1This is a flow chart of a cluster project automatic operation and maintenance method provided in an embodiment of the present application. This application provides a cluster project automatic operation and maintenance method, which specifically includes the following:

[0047] S110: In the current operation and maintenance cycle, aggregated analysis data of each workload in each cluster project is obtained, as well as an operation and maintenance project group obtained by adjusting the automatic operation and maintenance cluster project scope by the user according to the aggregated analysis data of the previous operation and maintenance cycle.

[0048] In this step, before the cluster project performs automatic operation and maintenance in the current operation and maintenance cycle, the performance indicator data of each workload in each cluster project in the current operation and maintenance cycle can be obtained first, and the aggregated analysis data of the current operation and maintenance cycle can be obtained by analysis, so as to obtain the operation and maintenance data of each workload.

[0049] It's understandable that the cluster projects here refer to individual projects within each cluster. A cluster is a group of independent servers connected via a high-speed network. They form a group and are managed as a single system. When a client interacts with a cluster, the cluster behaves like a standalone server. Kubernetes clusters are a leading container orchestration and management technology. In this application, servers within a cluster project can also be represented as containers. Automating the operation and maintenance of a cluster project involves adjusting workloads that are unreasonably utilizing resources within the containers.

[0050] For example, in a Kubernetes cluster, K8S cluster monitoring data indicators can be obtained and stored on a daily basis. The monitoring data indicators include the indicator data of various container resources in the K8S cluster. When a new round of operation and maintenance cycle begins, when the system initiates automatic operation and maintenance request data, the daily data in the previous operation and maintenance cycle can be obtained for aggregation and adjustment to obtain the aggregated analysis data of each workload in each cluster project. The operation and maintenance cycle here can be set and adjusted according to the current cluster type and the configuration of the automatic operation and maintenance platform. It can be 7 days as an operation and maintenance cycle or 10 days as an operation and maintenance cycle. There is no restriction here.

[0051] Furthermore, the user can adjust the scope of the automatic operation and maintenance cluster project based on the aggregated analysis data of the previous operation and maintenance cycle to obtain an operation and maintenance project group. That is, after obtaining the aggregated analysis data of each workload in each cluster project in the previous operation and maintenance cycle, the aggregated analysis data can be returned to the user so that the user can adjust the scope of the automatic operation and maintenance cluster project based on the aggregated analysis data.

[0052] Furthermore, the aggregated analysis data can be returned to the user through a scheduled email sending task to send the aggregated analysis data obtained during the current operation and maintenance cycle to the user. After the user receives and views the aggregated analysis data, he can make adjustments on the automatic operation and maintenance platform based on the working status of each project in the cluster to determine whether the project can be automatically operated and maintained in the next operation and maintenance cycle, thereby determining the operation and maintenance project team for the next operation and maintenance cycle.

[0053] S120: Determine operation and maintenance data for each workload based on the aggregated analysis data for each workload.

[0054] In this step, the aggregated analysis data of the current operation and maintenance cycle obtained in step S110 can be integrated and analyzed with the aggregated analysis data of the previous operation and maintenance cycle to determine the operation and maintenance data of each workload of each cluster project, so as to screen out the workloads whose resource data need to be adjusted.

[0055] For example, the automatic operation and maintenance cycle of the K8S cluster by the automatic operation and maintenance platform is 7 days. At the beginning of a new round of operation and maintenance cycle, after obtaining the aggregated analysis data, the automatic operation and maintenance platform can perform a second aggregation with the aggregated analysis data of the previous operation and maintenance cycle to obtain the operation and maintenance data of each workload in the K8S cluster, so as to judge whether the container resource occupancy corresponding to the workload is reasonable based on the information contained in the operation and maintenance data.

[0056] S130: Analyze the operation and maintenance data to determine target workloads that need to be adjusted among the workloads, and the intermediate cluster project where each target workload is located.

[0057] In this step, after obtaining the operation and maintenance data of each workload through step S120, the operation and maintenance data of each workload can be further analyzed, and the target workloads that need to be adjusted in each workload and the intermediate cluster projects where each target workload is located can be determined based on the analysis results.

[0058] Specifically, after analyzing the operation and maintenance data, the CPU and memory resource usage of each container in the K8S cluster can be further determined based on the analysis results, and the resource initialization requests and limits values ​​of the CPU and memory that have been set can be checked to determine whether the resource usage of the container is reasonable. If it is unreasonable, it is confirmed that the workload of the container needs to be adjusted. Therefore, based on the analysis results, all target workloads that need to be adjusted and the intermediate cluster projects where each target workload is located can be obtained.

[0059] It should be noted that in order to effectively schedule and fully utilize resources in the K8S cluster, the K8S cluster uses two types of restrictions: requests and limits to allocate resources at the container granularity. Each container can independently set corresponding requests and limits. These two parameters are set through the resources field of each container's containerSpec. Requests define the minimum amount of resources required by the corresponding container, and limits define the maximum resource upper limit that the container can consume, preventing excessive resource consumption from leading to resource shortages or even downtime. Generally speaking, requests are more important during scheduling, while limits are more important during runtime.

[0060] For example, for a Spring Boot business container, the requests here must be the minimum resources required by the JVM virtual machine in the container image. If the Pod's memory requests are specified as 10Mi, the actual memory Xms occupied by the JVM exceeds the memory allocated to the Pod by the K8S cluster, resulting in Pod memory overflow, and the K8S cluster continuously restarts the Pod. Setting limits to 0 means there is no limit on the used resources. When limits are set but not requests, the K8S cluster defaults to making requests equal to limits. Here, Pod is the smallest atomic scheduling unit in the K8S cluster.

[0061] S140: According to the operation and maintenance project group, target cluster projects that need to be automatically operated and maintained are screened out from all intermediate cluster projects to form a set of projects to be operated and maintained.

[0062] In this step, after the intermediate cluster project is determined in step S130, the target cluster project that needs to be automatically operated and maintained can be screened out from all target projects according to the operation and maintenance project group set in advance by the user, and a set of projects to be operated and maintained is formed to automatically operate and maintain each target cluster project in the set of projects to be operated and maintained.

[0063] Specifically, after obtaining the intermediate operation and maintenance projects corresponding to the containers whose resource usage is unreasonable and needs to be adjusted, the automated operation and maintenance system can screen again according to the operation and maintenance project group set in advance by the user, determine the target cluster projects that need to be automatically operated and maintained, and form a set of projects to be operated and maintained, so as to avoid unnecessary problems caused by the overlap of the automatic operation and maintenance process with the key time of project operation.

[0064] Furthermore, the operation and maintenance project group can be modified after obtaining aggregated analysis data in each operation and maintenance cycle, which serves as the basis for screening cluster projects in the next operation and maintenance cycle. Users can make adjustments based on the working status of cluster projects in the current operation and maintenance cycle, and determine the projects that can be automatically operated and maintained in the next operation and maintenance cycle, thereby obtaining the operation and maintenance project group.

[0065] S150: Automatically adjust the target workload corresponding to each target cluster project in the set of operation and maintenance projects according to the operation and maintenance data of each workload, so as to realize automatic operation and maintenance of each cluster project.

[0066] In this step, after the set of projects to be operated and maintained is determined in step S140, the target workload corresponding to each target cluster project in the set of projects to be operated and maintained can be automatically adjusted in turn according to the operation and maintenance data to achieve automatic operation and maintenance of each cluster project.

[0067] Specifically, when performing automatic operation and maintenance of cluster projects, the CPU can be automatically adjusted or the value of requests can be adjusted for the workload corresponding to each target cluster project in the set of operation and maintenance projects based on the reference values ​​obtained from the operation and maintenance data analysis, thereby improving the stability of the overall cluster service and completing the automatic operation and maintenance of each cluster project.

[0068] It should be noted that to avoid modifying or restarting multiple services in the same cluster, when a single thread executes this loop operation and performs control, only a single automatic operation and maintenance project is allowed to restart in a single cluster within the preset time. Generally speaking, the restart time of a single automatic operation and maintenance project can be set to 30s. If the cluster resources are sufficient, 30s can meet the requirements of a single automatic operation and maintenance project to adjust and start resource allocation. If the automatic operation and maintenance platform allows, this preset time can be adjusted and there is no restriction here.

[0069] In the above embodiment, before the cluster project in the current operation and maintenance cycle is automatically operated and maintained, the aggregated analysis data of each workload in each cluster project can be obtained first, so as to determine the operation and maintenance data of each workload, and then the operation and maintenance data can be analyzed to determine the actual resource usage in the cluster project, and to determine whether the performance indicators of each workload and the current service resource occupancy are reasonable. If not, adjustments need to be made. After determining the target workloads that need to be adjusted in each workload, and the intermediate cluster projects where each target workload is located, the present application can obtain the operation and maintenance project group obtained by the user adjusting the range of cluster projects for automatic operation and maintenance in the previous operation and maintenance cycle, and filter out the target cluster projects that need to be automatically operated and maintained from all intermediate cluster projects to form a set of projects to be operated and maintained, and automatically adjust the target workloads corresponding to each target cluster project in the set of projects to be operated and maintained in turn according to the operation and maintenance data of each workload, so as to improve the stability of the overall cluster service and realize the automatic operation and maintenance of each cluster project.

[0070] In one embodiment, in step S110, obtaining aggregated analysis data of each workload in each cluster project during the current operation and maintenance cycle may include:

[0071] S111: In the current operation and maintenance cycle, the average CPU resource value, the maximum CPU resource value, the average memory resource value, and the maximum memory resource value of each workload in each cluster project are obtained.

[0072] S112: According to the preset adjustment coefficient, the CPU resource average value, the CPU resource maximum value, the memory resource average value and the memory resource maximum value of each workload are adjusted to obtain the CPU resource request value corresponding to the CPU resource average value, the CPU resource limit value corresponding to the CPU resource maximum value, the memory resource request value corresponding to the memory resource average value and the memory resource limit value corresponding to the memory resource maximum value, and form aggregated analysis data for each workload.

[0073] In this embodiment, before the cluster project in the current operation and maintenance cycle is automatically operated and maintained, the CPU resource average value, CPU resource maximum value, memory resource average value and memory resource maximum value of each workload in each cluster project in the current operation and maintenance cycle can be obtained first, and then the CPU resource average value, CPU resource maximum value, memory resource average value and memory resource maximum value of each workload are adjusted according to the preset adjustment coefficient to obtain the CPU resource request value corresponding to the CPU resource average value, the CPU resource limit value corresponding to the CPU resource maximum value, the memory resource request value corresponding to the memory resource average value and the memory resource limit value corresponding to the memory resource maximum value, and form aggregated analysis data for each workload.

[0074] Specifically, when determining the aggregated analysis data, you can first determine the adjustment coefficients corresponding to the average values ​​of the CPU and memory resources and the adjustment coefficients corresponding to the maximum values ​​of the CPU and memory resources, and then calculate based on the average and maximum values ​​of the CPU and memory resources and the corresponding adjustment coefficients to obtain the CPU resource request value corresponding to the average value of the CPU resources, the CPU resource limit value corresponding to the maximum value of the CPU resources, the memory resource request value corresponding to the average value of the memory resources, and the memory resource limit value corresponding to the maximum value of the memory resources, and form the aggregated analysis data.

[0075] Furthermore, the adjustment coefficient of this application can first obtain an initial value by investigating each cluster project, and then adjust it according to the cluster stability and the historical data of CPU and memory resource occupancy in the cluster project to obtain the adjustment coefficient. For example, the adjustment coefficient used in this application defaults to an average value adjustment coefficient of 0.8 for CPU and memory resources, and a maximum value of 1.2.

[0076] In one embodiment, in step S111, obtaining the average CPU resource value, the maximum CPU resource value, the average memory resource value, and the maximum memory resource value of each workload in each cluster project during the current operation and maintenance cycle may include:

[0077] S1111: In the current operation and maintenance cycle, performance indicators of each workload in each cluster project are obtained, where the performance indicators include CPU resource indicators and memory resource indicators corresponding to each workload.

[0078] S1112: Convert the data format of the CPU resource indicator of each workload to obtain the average CPU resource value and the maximum CPU resource value of each workload.

[0079] S1113: Convert the data format of the memory resource indicator of each workload to obtain the average memory resource value and the maximum memory resource value of each workload.

[0080] In this embodiment, before the cluster project in the current operation and maintenance cycle is automatically operated and maintained, the performance indicators of each workload in each cluster project in the current operation and maintenance cycle can be obtained first, and then the CPU resource indicator and memory resource indicator in each performance indicator can be converted into data format to obtain the indicator data corresponding to each performance indicator. The indicator data includes the average CPU resource value, maximum CPU resource value, average memory resource value and maximum memory resource value of each workload.

[0081] Specifically, the performance indicators of each workload in the cluster project can be obtained at regular intervals, and the indicators can be converted into indicator data in order to visualize the resource occupancy in the cluster project and serve as the basis for automatic adjustment of the cluster project. The automated operation and maintenance platform can then obtain the indicator data from the cluster for analysis and calculation, thereby obtaining the average CPU resource value, maximum CPU resource value, average memory resource value, and maximum memory resource value for each workload.

[0082] Furthermore, to obtain the performance indicators of each workload in the cluster project, you can use Prometheus. Prometheus is an open source system monitoring and alarm system. In the K8S cluster automatic operation and maintenance platform, Prometheus can usually be used to monitor and obtain the performance indicators of each container. The automated operation and maintenance platform can obtain indicator data from Prometheus. PromQL can be used to obtain it. PromQL is a query language built into the Prometheus monitoring system. It can flexibly select, aggregate, and convert and calculate time series data in other ways. This language is only used to read data.

[0083] It should be noted that the automatic operation and maintenance platform can obtain indicator data and store it in the database in each operation and maintenance cycle. To avoid storage pressure on the database, the automatic operation and maintenance platform can clear historical data at regular intervals.

[0084] In one embodiment, in step S120, in the current operation and maintenance cycle, determining the operation and maintenance data of each workload based on the aggregated analysis data of each workload may include:

[0085] S121: Obtain the persistent data of each workload in the previous operation and maintenance cycle.

[0086] S122: Integrate the persistent data of each workload with the aggregated analysis data of each workload to obtain operation and maintenance data of each workload.

[0087] In this embodiment, after obtaining the aggregated analysis data of each workload in the current operation and maintenance cycle, the persistent data of each workload in the previous operation and maintenance cycle can be obtained and integrated with the aggregated analysis data of each workload in the current operation and maintenance cycle to obtain the operation and maintenance data of each workload, so as to screen out the workloads whose resource data need to be adjusted.

[0088] It can be understood that the persistent data here is the data obtained by secondary aggregation of the aggregated analysis data of each workload in the previous operation and maintenance cycle. The four indicator data that require automatic operation and maintenance for each workload are aggregated and optimally aggregated into one piece of data.

[0089] For example, after calculating and adjusting the indicator data of each workload to obtain the aggregated analysis data, the four indicator data of the CPU resource request value, CPU resource limit value, memory resource request value and memory resource limit value of each workload in the aggregated analysis data are aggregated for a second time, and the data obtained by the second aggregation is integrated and analyzed with the newly obtained aggregated analysis data in the next operation and maintenance cycle to obtain the workload data that has not yet been automatically operated and maintained.

[0090] In one embodiment, in step S140, target cluster projects that require automatic operation and maintenance are screened from all intermediate cluster projects according to the operation and maintenance project group to form a set of projects to be operated and maintained, which may include:

[0091] S141: determining intermediate cluster projects that are identical to the cluster projects in the operation and maintenance project group among all intermediate cluster projects, and taking the determined intermediate cluster projects as target cluster projects that require automatic operation and maintenance to form a set of projects to be operated and maintained.

[0092] In this embodiment, based on the operation and maintenance project group set by the user on the automatic operation and maintenance platform, the intermediate cluster project that is the same as the cluster project in the operation and maintenance project group is determined among all intermediate cluster projects, and the determined intermediate cluster project is used as the target cluster project that needs to be automatically operated and maintained to form a set of projects to be operated and maintained.

[0093] It should be noted that before starting the automatic operation and maintenance of the cluster project for the first time, the operation and maintenance project group needs to be set up. After that, in each operation and maintenance cycle, the user can be notified by email after obtaining the aggregated analysis data. This is used to give the user a data notification and a self-adjustable range of time to adjust the automatic operation and maintenance range of the next operation and maintenance cycle, so as to avoid unnecessary problems caused by the overlap of the critical operation time of the cluster project and the automatic operation and maintenance time when automatic operation and maintenance is performed in the next operation and maintenance cycle.

[0094] In one embodiment, in step S150, automatically adjusting the target workload corresponding to each target cluster project in the set of projects to be operated and maintained according to the operation and maintenance data of each workload may include:

[0095] S151: For the target workload corresponding to each target cluster project in the set of projects to be maintained, obtain the operation and maintenance data of the target workload, analyze the operation and maintenance data, determine the performance indicator to be adjusted for the target workload, and automatically adjust the performance indicator.

[0096] In this embodiment, for the target workload corresponding to each target cluster project in the set of projects to be operated and maintained, the operation and maintenance data corresponding to the target workload can be obtained, and the operation and maintenance data can be analyzed to obtain the actual resource usage of the CPU and memory of the target workload, and based on the resource initialization requests and limits values ​​of the CPU and memory set for the target workload, the performance indicators to be adjusted of the target workload are determined, and the performance indicators are automatically adjusted.

[0097] It should be noted that in each cluster project, the requests value of the workload and the current service resource occupancy form an unreasonable state. If it is too large, it will cause resource waste and affect the resource utilization of the cluster. If it is too small, it will affect the stability of various services in the cluster project and the current service. Similarly, if the limits value is set too large, it will cause too much over-provisioning and affect the stability of the overall cluster service. If it is too small, it may cause OOM (Out Of Memory) or repeated restarts. Therefore, it is necessary to automatically adjust the requests value or limits value of the target workload.

[0098] In one embodiment, the method may further include:

[0099] S160: After the automatic operation and maintenance of each cluster project in the current operation and maintenance cycle is completed, the performance indicator of the target workload corresponding to each target cluster project in the set of projects to be operated and maintained is updated.

[0100] In this embodiment, after the automatic operation and maintenance of each cluster project in the current operation and maintenance cycle is completed, the performance indicators of the target workload corresponding to each target cluster project in the set of projects to be operated and maintained are updated so that Prometheus can obtain the latest performance indicators of each workload in the cluster in a timely manner.

[0101] The following describes an automatic operation and maintenance device for a cluster project provided in an embodiment of the present application. The automatic operation and maintenance device for a cluster project described below and the automatic operation and maintenance method for a cluster project described above may refer to each other.

[0102] In one embodiment, Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of a cluster project automatic operation and maintenance device provided in an embodiment of the present application. The present application also provides a cluster project automatic operation and maintenance device, including a converged analysis data acquisition module 210, an operation and maintenance data acquisition module 220, a data analysis module 230, a data screening module 240, and an automatic operation and maintenance module 250, specifically including the following:

[0103] The aggregate analysis data acquisition module 210 is used to obtain the aggregate analysis data of each workload of each cluster project in the current operation and maintenance cycle, as well as the operation and maintenance project group obtained by the user adjusting the automatic operation and maintenance cluster project scope based on the aggregate analysis data of the previous operation and maintenance cycle.

[0104] The operation and maintenance data acquisition module 220 is configured to determine the operation and maintenance data of each workload based on the aggregated analysis data of each workload.

[0105] The data analysis module 230 is configured to analyze the operation and maintenance data of each workload, determine multiple target workloads that need to be adjusted in each workload, and the intermediate cluster project where each target workload is located.

[0106] The data screening module 240 is configured to screen target cluster projects that require automatic maintenance from all intermediate cluster projects according to the maintenance project group, to form a set of projects to be maintained.

[0107] The automatic operation and maintenance module 250 is configured to automatically adjust the target workload corresponding to each target cluster project in the set of projects to be operated and maintained according to the operation and maintenance data of each workload, so as to realize automatic operation and maintenance of each cluster project.

[0108] In the above embodiment, before the cluster project in the current operation and maintenance cycle is automatically operated and maintained, the aggregated analysis data of each workload in each cluster project can be obtained first, so as to determine the operation and maintenance data of each workload, and then the operation and maintenance data can be analyzed to determine the actual resource usage in the cluster project, and to determine whether the performance indicators of each workload and the current service resource occupancy are reasonable. If not, adjustments need to be made. After determining the target workloads that need to be adjusted in each workload, and the intermediate cluster projects where each target workload is located, the present application can obtain the operation and maintenance project group obtained by the user adjusting the range of cluster projects for automatic operation and maintenance in the previous operation and maintenance cycle, and filter out the target cluster projects that need to be automatically operated and maintained from all intermediate cluster projects to form a set of projects to be operated and maintained, and automatically adjust the target workloads corresponding to each target cluster project in the set of projects to be operated and maintained in turn according to the operation and maintenance data of each workload, so as to improve the stability of the overall cluster service and realize the automatic operation and maintenance of each cluster project.

[0109] In one embodiment, the aggregate analysis data acquisition module 210 may include:

[0110] The resource data acquisition submodule is used to obtain the average CPU resource, maximum CPU resource, average memory resource, and maximum memory resource of each workload in each cluster project during the current operation and maintenance cycle.

[0111] The resource data adjustment submodule is used to adjust the CPU resource average value, CPU resource maximum value, memory resource average value and memory resource maximum value of each workload according to the preset adjustment coefficient, obtain the CPU resource request value corresponding to the CPU resource average value, the CPU resource limit value corresponding to the CPU resource maximum value, the memory resource request value corresponding to the memory resource average value and the memory resource limit value corresponding to the memory resource maximum value, and form the aggregated analysis data of each workload.

[0112] In one embodiment, the resource data acquisition submodule may include:

[0113] The performance indicator acquisition unit is used to acquire the performance indicators corresponding to each workload in each cluster project during the current operation and maintenance cycle, wherein the performance indicators include the CPU resource indicators and memory resource indicators corresponding to each workload.

[0114] The CPU resource data acquisition unit is used to convert the data format of the CPU resource indicator of each workload to obtain the average CPU resource value and the maximum CPU resource value of each workload.

[0115] The memory resource data acquisition unit is used to convert the data format of the memory resource indicator of each workload to obtain the average memory resource value and the maximum memory resource value of each workload.

[0116] In one embodiment, the operation and maintenance data acquisition module 220 may include:

[0117] The persistent data acquisition submodule is used to obtain the persistent data of each workload in the previous operation and maintenance cycle. The persistent data is the data obtained by secondary aggregation of the aggregated analysis data of each workload in the previous operation and maintenance cycle.

[0118] The data integration module is used to integrate the persistent data of each workload with the aggregated analysis data of each workload to obtain the operation and maintenance data of each workload.

[0119] In one embodiment, the data screening module 240 may include:

[0120] The data screening submodule is used to determine the intermediate cluster projects in all intermediate cluster projects that are the same as the cluster projects in the operation and maintenance project group, and use the determined intermediate cluster projects as target cluster projects that need automatic operation and maintenance to form a set of projects to be operated and maintained.

[0121] In one embodiment, the automatic operation and maintenance module 250 may include:

[0122] The automatic operation and maintenance submodule is used to obtain the operation and maintenance data of the target workload corresponding to each target cluster project in the set of projects to be operated and maintained, analyze the operation and maintenance data, determine the performance indicators to be adjusted for the target workload, and automatically adjust the performance indicators.

[0123] In one embodiment, the apparatus may further include:

[0124] The performance indicator update module is used to update the performance indicator of the target workload corresponding to each target cluster project in the set of projects to be operated and maintained after the automatic operation and maintenance of each cluster project in the current operation and maintenance cycle is completed.

[0125] In one embodiment, the present application further provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the cluster project automatic operation and maintenance method as described in any of the above embodiments.

[0126] In one embodiment, the present application further provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the cluster project automatic operation and maintenance method as described in any of the above embodiments.

[0127] Schematically, as Figure 3 As shown, Figure 3 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 300 can be provided as a server. Figure 3 Computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by memory 301 for storing instructions executable by processing component 302, such as application programs. The application programs stored in memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, processing component 302 is configured to execute the instructions to perform the cluster project automatic operation and maintenance method according to any of the above-described embodiments.

[0128] The computer device 300 may further include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate based on an operating system stored in the memory 301, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or the like.

[0129] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0130] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0131] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referenced to each other.

[0132] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cluster project automatic operation and maintenance method, characterized in that: The method comprises: During the current maintenance cycle, the aggregated analysis data for each workload in each cluster project is obtained, as well as the maintenance project group obtained by adjusting the automatic maintenance cluster project scope based on the aggregated analysis data of the previous maintenance cycle; Determine the operation and maintenance data for each workload based on the aggregated analytical data for each workload; Analyze the operation and maintenance data of each workload to determine multiple target workloads that need to be adjusted within each workload, as well as the intermediate cluster projects where each target workload is located; According to the operation and maintenance project group, target cluster projects that need automatic operation and maintenance are screened out from all intermediate cluster projects to form a set of projects to be operated and maintained; Automatically adjust the target workload corresponding to each target cluster project in the set of projects to be maintained according to the operation and maintenance data of each workload, so as to realize automatic operation and maintenance of each cluster project; The step of determining the operation and maintenance data of each workload based on the aggregated analysis data of each workload includes: Obtaining persistent data for each workload in the previous operation and maintenance cycle, wherein the persistent data is data obtained by secondary aggregation of aggregated analysis data for each workload in the previous operation and maintenance cycle; Integrate the persistent data of each workload with the aggregated analytical data of each workload to obtain the operational data of each workload; The automatically adjusting the target workload corresponding to each target cluster project in the set of projects to be operated and maintained according to the operation and maintenance data of each workload includes: For the target workload corresponding to each target cluster project in the set of projects to be maintained, the operation and maintenance data of the target workload is obtained, and the operation and maintenance data is analyzed to determine the performance index to be adjusted for the target workload, and the performance index is automatically adjusted.

2. The automatic operation and maintenance method for cluster projects according to claim 1, characterized in that: During the current operation and maintenance cycle, aggregated analytical data for each workload in each cluster project is obtained, including: During the current operation and maintenance cycle, obtain the average CPU resource, maximum CPU resource, average memory resource, and maximum memory resource of each workload in each cluster project; According to the preset adjustment coefficient, the CPU resource average value, CPU resource maximum value, memory resource average value and memory resource maximum value of each workload are adjusted to obtain the CPU resource request value corresponding to the CPU resource average value, the CPU resource limit value corresponding to the CPU resource maximum value, the memory resource request value corresponding to the memory resource average value and the memory resource limit value corresponding to the memory resource maximum value, and form the aggregated analysis data of each workload.

3. The automatic operation and maintenance method for cluster projects according to claim 2, characterized in that: The CPU resource average, CPU resource maximum, memory resource average, and memory resource maximum values ​​for each workload in each cluster project are obtained during the current operation and maintenance cycle, including: During the current operation and maintenance cycle, the performance indicators of each workload in each cluster project are obtained, including the CPU resource indicators and memory resource indicators corresponding to each workload; Convert the data format of the CPU resource indicators of each workload to obtain the average CPU resource and maximum CPU resource of each workload; The data format of the memory resource indicator of each workload is converted to obtain the average memory resource and the maximum memory resource of each workload.

4. The automatic operation and maintenance method for cluster projects according to claim 1, characterized in that: According to the operation and maintenance project group, target cluster projects that need to be automatically operated and maintained are screened from all intermediate cluster projects to form a set of projects to be operated and maintained, including: After determining the intermediate cluster projects that are identical to the cluster projects in the operation and maintenance project group among all the intermediate cluster projects, and taking the determined intermediate cluster projects as target cluster projects that require automatic operation and maintenance, a set of projects to be operated and maintained is formed.

5. The automatic operation and maintenance method for cluster projects according to claim 1, characterized in that: The method further comprises: After the automatic operation and maintenance of each cluster project in the current operation and maintenance cycle is completed, the performance indicator of the target workload corresponding to each target cluster project in the set of projects to be operated and maintained is updated.

6. A cluster project automatic operation and maintenance device, characterized in that: include: The aggregated analysis data acquisition module is used to obtain the aggregated analysis data of each workload in each cluster project during the current operation and maintenance cycle, as well as the operation and maintenance project group obtained by adjusting the scope of the automatic operation and maintenance cluster project based on the aggregated analysis data of the previous operation and maintenance cycle; an operation and maintenance data acquisition module, configured to determine operation and maintenance data for each workload based on the aggregated analysis data; The data analysis module is used to analyze the operation and maintenance data of each workload, determine the multiple target workloads that need to be adjusted in each workload, and the intermediate cluster project where each target workload is located; A data screening module is used to screen target cluster projects that need automatic operation and maintenance from all intermediate cluster projects according to the operation and maintenance project group to form a set of projects to be operated and maintained; An automatic operation and maintenance module, configured to automatically adjust the target workload corresponding to each target cluster project in the set of projects to be operated and maintained according to the operation and maintenance data of each workload, so as to realize automatic operation and maintenance of each cluster project; The operation and maintenance data acquisition module includes: The persistent data acquisition submodule is used to obtain the persistent data of each workload in the previous operation and maintenance cycle. The persistent data is the data obtained by secondary aggregation of the aggregated analysis data of each workload in the previous operation and maintenance cycle; The data integration module is used to integrate the persistent data of each workload with the aggregated analysis data of each workload to obtain the operation and maintenance data of each workload; The automatic operation and maintenance module includes: The automatic operation and maintenance submodule is used to obtain the operation and maintenance data of the target workload corresponding to each target cluster project in the set of projects to be operated and maintained, analyze the operation and maintenance data, determine the performance indicators to be adjusted for the target workload, and automatically adjust the performance indicators.

7. A storage medium, characterized in that: The storage medium stores computer-readable instructions, which, when executed by one or more processors, enable the one or more processors to perform the steps of the cluster project automatic operation and maintenance method according to any one of claims 1 to 5.

8. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, execute the steps of the cluster project automatic operation and maintenance method according to any one of claims 1 to 5.

Images