CPU and memory utilization rate prediction algorithm based on clustering and Prophet

Abstract

The invention discloses a CPU (Central Processing Unit) and memory utilization rate prediction method based on clustering and Prophet, which comprises the following steps of: after collecting historical data of a server, carrying out preprocessing including noise elimination and missing value filling, automatically dividing equipment groups in similar load modes through a K-Means clustering algorithm, training and sharing a Prophet prediction model for each equipment group, and outputting a prediction result. According to the method, the problems of poor load difference adaptation and model redundancy of multiple devices are solved, accurate prediction of the multiple devices is realized, the operation and maintenance cost is reduced, the resource scheduling rationality is improved, and the service life of the server is prolonged.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a CPU and memory usage prediction algorithm based on clustering and Prophet. Background Technology

[0002] With the rapid development of cloud computing and big data technologies, the number of data center servers is growing exponentially. As core hardware resources, CPU and memory utilization prediction is crucial for resource scheduling and operation and maintenance management. Current technologies for multi-device CPU and memory utilization prediction have significant drawbacks: most solutions employ a single, one-size-fits-all model or model each device individually. The former ignores the differences in load patterns across different devices, leading to insufficient prediction accuracy; the latter requires maintaining numerous independent models, resulting in high model redundancy, soaring operation and maintenance costs, and difficulty in adapting to the prediction needs of expanded device scale. This situation prevents operation and maintenance personnel from efficiently managing the resource status of multiple devices, resulting in both resource waste and service interruption risks.

[0003] Based on the above problems, there is an urgent need for an accurate prediction solution that can adapt to the load differences of multiple devices and reduce model redundancy. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method for predicting CPU and memory utilization based on clustering and Prophet. This method includes collecting historical CPU and memory utilization data from servers, and further includes: performing preprocessing on the collected historical data, including noise removal and missing value imputation; automatically dividing devices into groups with similar load patterns based on the K-Means clustering algorithm to solve the prediction adaptation problem caused by load differences among multiple devices; training a shared Prophet prediction model for each device group to adapt to the load cycle characteristics of the device group, thus solving the problems of model redundancy and high maintenance costs in multi-device prediction; and outputting the predicted results of future CPU and memory utilization of the server through the Prophet prediction model.

[0005] Preferably, the preprocessing further includes equal-interval resampling, using Pandas' .resample() method to group data at a frequency of 5 minutes or 15 minutes, and averaging the data within each time window to generate a regular time series.

[0006] More preferably, the noise removal specifically involves: marking individual data points with CPU or memory usage rates greater than 80% as potential outliers; counting the proportion of data points whose values ​​exceed 80%, and if the proportion exceeds 60%, it is determined to be a normal business peak and no removal is performed; if the proportion does not exceed 60%, the potential outlier is set as a missing value.

[0007] More preferably, the missing value imputation adopts a two-level strategy: the first level uses Pandas' .interpolate(method='time') method to perform time-weighted interpolation; if there are still missing values ​​after the first level processing, forward imputation is used, and if there are still missing values ​​after forward imputation, backward imputation is used.

[0008] More preferably, the K-Means clustering algorithm execution process includes: extracting overall statistical features and refined time-granular features to construct a multi-dimensional feature vector; using the Z-Score standardization method to convert all features into a distribution with a mean of 0 and a standard deviation of 1; determining the optimal number of clusters by evaluating the silhouette coefficient, traversing all possible values ​​from 2 to the maximum number of clusters, and selecting the number of clusters with the largest silhouette coefficient.

[0009] Further preferably, the overall statistical features include the arithmetic mean of CPU utilization, the standard deviation of CPU utilization, the difference between the maximum and minimum CPU utilization, the maximum value of the average CPU utilization per hour during the day, and the four types of features corresponding to memory utilization; the refined time granular features include 24-hour granular features and 7-day granular features. The 24-hour granular features are the average CPU utilization, standard deviation of CPU utilization, average memory utilization, and standard deviation of memory utilization for each hour during the day. The 7-day granular features are the average CPU utilization and average memory utilization for each day during the week.

[0010] More preferably, the K-Means clustering algorithm uses the K-Means++ algorithm to initialize the cluster centers, and sets the n_init=10 parameter for multiple initialization optimizations; the iteration termination condition is that the relative change in the position of the cluster centers is less than 1e-4 or the number of iterations reaches 300.

[0011] Further preferably, the training of the Prophet prediction model includes: sorting the preprocessed historical data in chronological order, selecting the data from the last day as the test set, and the remaining data as the training set; customizing parameters for different device groups, using a multiplication mode for CPU metrics for some device groups, and a uniform addition mode for memory metrics, with the changepoint_prior_scale parameter ranging from 0.1 to 0.4, the seasonality_prior_scale parameter ranging from 0.6 to 15.0, and the n_changepoints parameter ranging from 10 to 22.

[0012] Further preferred features include an anomaly handling mechanism: when data is missing, interpolation, prediction based on regression models, or data replacement are used to fill the gaps; when the predicted value of CPU utilization or memory utilization exceeds the upper bound of the confidence interval for three consecutive time points, the difference between the predicted value and the upper bound is greater than a preset threshold, and the abnormal state lasts for at least 45 minutes, an early warning is triggered.

[0013] More preferably, the upper bound of the confidence interval has a CPU utilization threshold of 90% and a memory utilization threshold of 90%; the warning includes a severe warning and a warning warning, with a resource utilization threshold of 95% for a severe warning and a resource utilization threshold of 85% for a warning warning.

[0014] Technical Effects: This invention automatically groups devices with similar load patterns using the K-Means clustering algorithm, and trains a shared Prophet prediction model for each group. It creatively solves the core problems of poor load variation adaptation and model redundancy in existing technologies when predicting multiple devices. It achieves accurate prediction of CPU and memory usage across multiple devices, simplifies model maintenance processes, reduces operational costs, and provides reliable support for intelligent resource scheduling. Attached Figure Description

[0015] Figure 1 The flowchart shows the CPU and memory utilization prediction method based on clustering and Prophet in this application. Figure 2 This is a structural diagram of the CPU and memory utilization prediction algorithms for clustering and Prophet in this application. Figure 3 This is a flowchart of the profile coefficient evaluation process for this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0017] Traditional technical solutions have the following technical problems: In multi-device CPU and memory utilization prediction, existing solutions either ignore the differences in device load patterns and use a single model, resulting in insufficient prediction accuracy; or they model each device separately, resulting in high model redundancy, high operation and maintenance costs, and inability to adapt to the prediction needs of multi-device scale.

[0018] Based on this, please refer to Figures 1-3 This embodiment provides a method for predicting CPU and memory usage based on clustering and Prophet, including collecting historical CPU and memory usage data of the server, and also including: S1: Perform preprocessing on the collected historical data, including noise removal and missing value imputation; S2: Automatically divides devices with similar load patterns based on the K-Means clustering algorithm to solve the prediction and adaptation problem caused by the load differences of multiple devices; S3: Train a shared Prophet prediction model for each device group to adapt to the load cycle characteristics of the device group and solve the problems of model redundancy and high operation and maintenance costs when predicting multiple devices. S4: Output the predicted CPU and memory usage of the server using the Prophet prediction model.

[0019] The technical solution is described as follows: First, a server resource data collection process is initiated. The collection targets are the historical CPU and memory usage data of all servers within the target range for the past three months. The collected data includes the unique identifier of the device, the corresponding collection time, and the usage value, ensuring that the data covers the load status of the devices at different times. In the preprocessing stage, noise removal is performed first, setting 80% as the benchmark for judging abnormal usage. Individual abnormal data points with sudden increases or decreases in usage are identified. Simultaneously, the proportion of high usage data is statistically analyzed to distinguish between normal business peaks and noise, avoiding the accidental deletion of valid data. Then, for data gaps, a strategy of primarily using time-weighted interpolation for forward filling and secondarily for backward filling is adopted to fill data gaps and ensure sequence continuity. After preprocessing, the K-Means clustering process is initiated. First, two core features are extracted to construct a multi-dimensional feature vector. The overall statistical features include the difference between the arithmetic mean and standard deviation of CPU and memory usage, the maximum and minimum values, and the maximum value of the hourly average over a day. Refined time-granular features include the hourly mean and standard deviation over 24 hours and the daily average over a week, comprehensively characterizing device load patterns. Then, the Z-Score normalization method is used to process all features, transforming them into a distribution with a mean of 0 and a standard deviation of 1, eliminating the impact of dimensional differences on the clustering results. The optimal number of clusters is determined through silhouette coefficient evaluation. All possible values ​​from 2 to the preset maximum number of clusters are iterated, and K-Means clustering is performed on each cluster number, calculating the silhouette coefficient. The cluster number with the largest silhouette coefficient is selected as the final number of clusters, ensuring that the clustering results can effectively distinguish devices with different load patterns. Finally, K-Means clustering is performed to group devices with similar load patterns into the same device group. For each cluster, a shared Prophet prediction model is trained based on the common load cycle characteristics, trends, and fluctuation patterns of the devices within the group. The model parameters are customized and optimized according to the load characteristics of the devices within the group. For example, the trend flexibility parameter and seasonal effect strength parameter are adjusted for high-fluctuation load groups, so that the model can accurately capture the load change patterns of the devices within the group. After the model is trained, the preprocessed historical data is input to output the prediction results of CPU utilization and memory utilization for future periods, realizing intensive and accurate prediction of multiple devices.

[0020] The technical effects achieved by the above embodiments include: adapting to the load differences of multiple devices, avoiding prediction bias of a single model, reducing the number of independent models, reducing operation and maintenance complexity, and achieving accurate and efficient prediction of resource utilization of multiple devices.

[0021] Traditional technical solutions have the following technical problems: preprocessing only focuses on a single data cleaning step and does not form a complete and regular process, resulting in irregular data time dimension and affecting the consistency of subsequent model input.

[0022] Based on this, the preprocessing also includes equal-interval resampling, using Pandas' resample method to group data at a frequency of 5 minutes or 15 minutes, and averaging the data within each time window to generate a regular time series.

[0023] The technical solution is described below: After noise removal and missing value imputation, to address the issue of non-uniform timestamps in the original monitoring data, an equal-interval resampling step is introduced to construct a complete preprocessing workflow. Pandas' resample method is used as the resampling tool. The resampling frequency is selected based on the actual business's prediction granularity requirements. For example, if short-term prediction with a 5-minute granularity is required, a 5-minute resampling frequency is chosen; if medium-term prediction with a 15-minute granularity is required, a 15-minute resampling frequency is chosen. During resampling, all data points within the time window corresponding to each frequency are grouped together. The arithmetic mean of all data points within the group is calculated to obtain the representative data value for that time window. This process is repeated for all time windows to generate a continuous, regular time series of data points with regular time dimensions. This process not only converts non-uniform timestamp data into a format that meets the model's input requirements but also smooths short-term random fluctuations through window mean calculation, preserving long-term trend characteristics and ensuring that the data input to the K-Means clustering algorithm and the Prophet prediction model maintains consistency and regularity in the time dimension.

[0024] The technical effects achieved by the above embodiments include: eliminating the irregularity of the data time dimension, meeting the model's requirements for the input data format, reducing data fluctuation interference, and improving the stability of subsequent modeling.

[0025] Traditional technical solutions have the following problems: noise removal is based solely on a fixed threshold, failing to distinguish between normal high load and abnormal noise, and is prone to accidentally deleting valid data or retaining interfering data.

[0026] Based on this, the noise removal specifically involves: marking individual data points with CPU or memory usage exceeding 80% as potential outliers; counting the proportion of data points whose values ​​exceed 80%, and if the proportion exceeds 60%, it is determined to be a normal business peak and no removal is performed; if the proportion does not exceed 60%, the potential outlier is set as a missing value.

[0027] The technical solution is described below: A dual-standard judgment mechanism is employed to achieve accurate noise removal. First, 80% is set as the anomaly judgment threshold for CPU and memory usage. All collected historical data points are iterated through, and individual data points exceeding this threshold are marked as potential outliers, initially filtering out possible noisy data. Then, a proportional statistics process is initiated to calculate the proportion of data points in the entire dataset whose values ​​exceed 80%. This proportion distinguishes between normal business peaks and abnormal noise. If the statistical proportion exceeds 60%, it indicates that the equipment was under continuous high load during that period, belonging to a normal business peak scenario. The marked potential outliers are valid data and no removal operation is performed. If the statistical proportion does not exceed 60%, it indicates that the marked potential outliers are not caused by normal business load but are noise generated during data collection, transmission, or storage. These potential outliers are set as missing values ​​to exclude them from subsequent data processing and modeling, eliminating noise interference with the prediction results at the source.

[0028] The technical effects achieved by the above embodiments include: accurately distinguishing between normal high load and abnormal noise, avoiding the accidental deletion of valid data or noise residue, and ensuring the purity and validity of the preprocessed data.

[0029] Traditional technical solutions have the following technical problems: missing value imputation uses a single method, which cannot adapt to missing value scenarios caused by different locations and reasons, and the imputation results deviate greatly from the actual data trend.

[0030] Based on this, the missing value imputation adopts a two-level strategy: the first level uses Pandas' interpolatemethod=time method for time-weighted interpolation; if there are still missing values ​​after the first level, forward imputation is used, and if there are still missing values ​​after forward imputation, backward imputation is used.

[0031] The technical solution is described below: A two-level progressive missing value imputation strategy is constructed to ensure imputation effectiveness under different missing value scenarios. The first level uses a time-weighted interpolation method, calling Pandas' `interpolate` function and setting the `method` parameter to `time`. This method performs linear weighted calculations based on the time distance between valid data points before and after the missing data. Valid data points with closer time distances have higher weights, calculating the imputation value for the missing position. This method can accurately adapt to scenarios with irregular data time intervals, making the imputation results more consistent with the actual trend of data changes. If missing values ​​still exist in the data sequence after the first-level time-weighted interpolation, especially at the beginning or end of the sequence, the second-level imputation strategy is activated. First, a forward imputation method is used, propagating the nearest valid data value before the missing value backward to fill the gap in the middle. If missing values ​​still exist at the beginning of the sequence after forward imputation, a backward imputation method is used, propagating the nearest valid data value after the missing value forward to fill the gap at the beginning of the sequence. Through the combined application of the two-level imputation strategy, various scenarios such as missing values ​​in the middle of the data, missing values ​​at the beginning of the sequence, and missing values ​​at the end of the sequence can be handled efficiently, ultimately outputting a continuous and complete regular data sequence.

[0032] The technical effects achieved by the above embodiments include: adapting to various missing scenarios, filling in results that conform to the actual change patterns of data, generating continuous and complete regular sequences, and providing high-quality input for modeling.

[0033] Traditional technical solutions have the following technical problems: K-Means clustering lacks systematic feature support and a mechanism for determining the optimal number of clusters, resulting in clustering results that cannot accurately reflect differences in device load patterns.

[0034] Based on this, the K-Means clustering algorithm execution process includes: extracting overall statistical features and refined time-granular features to construct a multi-dimensional feature vector; using the Z-Score standardization method to convert all features into a distribution with a mean of 0 and a standard deviation of 1; determining the optimal number of clusters through silhouette coefficient evaluation, traversing all possible values ​​from 2 to the maximum number of clusters, and selecting the number of clusters with the largest silhouette coefficient.

[0035] The technical solution is described below: A complete K-Means clustering execution process is constructed to ensure that the clustering results accurately distinguish device load patterns. First, feature extraction is performed to construct a comprehensive multi-dimensional feature vector. Overall statistical features characterize device load characteristics from a global perspective, including the arithmetic mean of CPU utilization, the standard deviation of CPU utilization, the difference between the maximum and minimum CPU utilization, the maximum value of the hourly average CPU utilization throughout the day, and the four features corresponding to memory utilization, reflecting the device's average load level, load fluctuation range, load span, and peak period patterns, respectively. Refined time-granular features are used to capture the periodic patterns of load, including 24-hour and 7-day granular features. The 24-hour granular features are the hourly average CPU utilization, the standard deviation of CPU utilization, and the hourly average memory utilization, accurately describing the daily load change pattern. The 7-day granular features are the daily average CPU utilization and the average memory utilization throughout the week, revealing the weekly load change pattern. After feature extraction, Z-Score normalization is used to process all features, calculating the mean and standard deviation of each feature using the formula... The eigenvalues ​​are converted into a distribution with a mean of 0 and a standard deviation of 1 to eliminate the influence of differences in the dimensions of different eigenvalues ​​on the clustering results. Then, the optimal number of clusters is determined. The range of the number of clusters is set from 2 to a preset maximum number. K-Means clustering is performed on each cluster, and the corresponding silhouette coefficient is calculated. The silhouette coefficient comprehensively measures the cohesion of samples within a cluster and the separation of samples between clusters, with a value ranging from -1 to 1. The closer the value is to 1, the better the clustering quality. The cluster with the largest silhouette coefficient is selected as the optimal number of clusters to ensure that the clustering results can effectively distinguish different device load patterns.

[0036] The technical effects achieved by the above embodiments include: constructing a comprehensive feature system, eliminating dimensional interference, accurately determining the optimal number of clusters, and enabling the clustering results to effectively distinguish the load modes of different devices.

[0037] Traditional technical solutions have the following technical problems: cluster feature extraction lacks clear dimensional definitions, the feature coverage is incomplete, and it cannot fully characterize the global characteristics and periodic patterns of device load.

[0038] Based on this, the overall statistical features include the arithmetic mean of CPU utilization, the standard deviation of CPU utilization, the difference between the maximum and minimum CPU utilization, the maximum value of the average CPU utilization per hour of the day, and the four types of features corresponding to memory utilization; the refined time-granular features include 24-hour granular features and 7-day granular features. The 24-hour granular features are the average CPU utilization, the standard deviation of CPU utilization, and the average memory utilization and standard deviation of memory utilization per hour of the day. The 7-day granular features are the average CPU utilization and the average memory utilization per day of the week.

[0039] The technical solution is described as follows: A two-dimensional feature system is constructed to comprehensively depict the global characteristics and periodic patterns of device load. Overall statistical features are extracted from a global perspective. The arithmetic mean of CPU utilization is obtained by summing all CPU utilization data points and dividing by the number of data points, reflecting the average CPU load level of the device. The standard deviation of CPU utilization is obtained by taking the square root of the average of the squared differences between each CPU utilization data point and the mean, quantifying the fluctuation range of CPU load. The difference between the maximum and minimum CPU utilization values ​​is obtained by finding the maximum and minimum values ​​among all CPU utilization data points and calculating their difference, reflecting the range of CPU load. The maximum value of the hourly average CPU utilization is obtained by first calculating the average CPU utilization for each hourly group, and then finding the maximum value from these averages, revealing the peak period pattern of CPU load. The four types of features corresponding to memory utilization are calculated using the same method, comprehensively reflecting the global characteristics of memory load. Refined time-granular features are extracted from a periodic perspective. 24-hour granular features are grouped by the 24 hours of a day. The mean and standard deviation of CPU and memory usage data within each hourly group are calculated, resulting in 48-dimensional features that accurately describe the device's load variation patterns at different times of the day. 7-day granular features are grouped by the 7 days of a week. The mean of CPU and memory usage data within each daily group is calculated, resulting in 7-dimensional features that clearly reveal the device's load variation patterns on different days within a week. These two types of features complement each other, forming a comprehensive load description system.

[0040] The technical effects achieved by the above embodiments include: comprehensively covering the global characteristics and multi-dimensional periodic patterns of the load, providing high-information-density feature support for the clustering algorithm, and improving the accuracy of the clustering results.

[0041] Traditional technical solutions have the following problems: the initial center selection of K-Means clustering is randomized and the termination condition of the iteration is ambiguous, resulting in poor stability and low convergence efficiency of the clustering results.

[0042] Based on this, the K-Means clustering algorithm uses the K-Means++ algorithm to initialize the cluster centers, and sets the n_init=10 parameter to perform multiple initialization optimizations; the iteration termination condition is that the relative change in the position of the cluster center is less than 1e-4 or the number of iterations reaches 300.

[0043] The technical solution is described below: The initialization and iteration termination mechanism of K-Means clustering is optimized to improve the stability and convergence efficiency of the clustering results. The initial center selection uses the K-Means++ algorithm. This algorithm first randomly selects a data point from all sample data as the first initial cluster center. Then, it calculates the distance from each remaining data point to the selected initial cluster center and selects the next initial cluster center based on a probability distribution proportional to the square of the distance. This process is repeated until a preset number of initial cluster centers are selected, ensuring that the initial cluster centers are evenly distributed in the data space and avoiding clustering bias caused by the concentration of initial centers. The parameter n_init=10 is set, meaning that 10 different groups of initial cluster centers are used to perform K-Means clustering operations. Each group of initial centers completes the clustering process independently, and the clustering result with the smallest sum of squares within the cluster is selected as the final output, significantly reducing the impact of the randomness of the initial centers on the clustering quality. The iteration termination is subject to two constraints. The first is a convergence tolerance threshold, which calculates the relative change in the cluster center position after each iteration. When the relative change is less than 1e-4, the clustering result is considered to have reached a stable state, and the iteration is terminated immediately. The second is a maximum iteration limit, which sets the maximum number of iterations to 300. If the convergence tolerance threshold is not met during the clustering process, but the number of iterations reaches 300, the iteration is forcibly terminated to avoid the algorithm from getting stuck in an infinite loop and to ensure that the clustering calculation is completed within a reasonable time.

[0044] The technical effects achieved by the above embodiments include: improving the stability and consistency of clustering results, accelerating the algorithm convergence speed, and ensuring that high-quality clustering results are obtained within a reasonable time.

[0045] Traditional technical solutions have the following technical problems: Prophet model training lacks targeted parameter configuration and data segmentation strategies, resulting in poor model adaptability and insufficient generalization ability.

[0046] Based on this, the training of the Prophet model includes: sorting the preprocessed historical data in chronological order, selecting the data from the last day as the test set, and using the remaining data as the training set; customizing parameters for different device groups, using multiplication mode for some device groups of CPU metrics, and uniformly using addition mode for memory metrics, with the changepoint_prior_scale parameter ranging from 0.1 to 0.4, the seasonality_prior_scale parameter ranging from 0.6 to 15.0, and the n_changepoints parameter ranging from 10 to 22.

[0047] The technical solution is described below: A targeted model training process is constructed to improve the adaptability and generalization ability of the Prophet model. Data segmentation strictly follows the temporal characteristics of time series data. Preprocessed historical data is sorted according to the chronological order of collection time. Data from the last day after sorting is selected as the test set to evaluate the model's generalization ability on unseen data. The remaining data is used as the training set for learning and optimizing model parameters, ensuring that the temporal order of the training and test sets is not disrupted, consistent with the logic of using historical data to predict the future in real-world prediction scenarios. Parameter configuration is customized based on the load characteristics and metric types of the cluster groups. For CPU metrics, device groups where load fluctuations significantly increase with utilization are treated using a multiplicative mode to better adapt to fluctuation characteristics. Memory metrics, due to their relatively stable fluctuation amplitude, are uniformly treated using an additive mode to ensure model stability. The `changepoint_prior_scale` parameter controls the flexibility of the trend, with a value range of 0.1-0.4. A higher value is configured for high-fluctuation load groups to sensitively capture trend changes, while a lower value is configured for low-fluctuation load groups to maintain a smooth trend and avoid overfitting. The `seasonality_prior_scale` parameter adjusts the strength of the seasonal effect, with a value range of 0.6-15.0. A higher value is configured for equipment groups with low-fluctuation sequences to highlight stable seasonal patterns, while a lower value is configured for equipment groups with high-fluctuation sequences to avoid seasonal noise interference. The `n_changepoints` parameter sets the maximum number of potential trend change points, with a value range of 10-22. This is adjusted according to the complexity of the load trend of the equipment group; equipment groups with frequent trend changes are configured with a larger number of change points, while equipment groups with stable trends are configured with a smaller number of change points, ensuring that the model can accurately adapt to the load characteristics of different equipment groups.

[0048] The technical effects achieved by the above embodiments include: improving the model's adaptability to different load modes, ensuring the rationality of training and testing, and enhancing the model's generalization ability and prediction accuracy.

[0049] Traditional technical solutions have the following technical problems: they lack a sound anomaly handling mechanism and cannot effectively deal with missing data or prediction anomalies, leading to prediction service interruption or misjudgment.

[0050] Based on this, an anomaly handling mechanism is also included: when data is missing, interpolation based on regression model prediction or data replacement is used to fill the missing data; when the predicted value of CPU utilization or memory utilization exceeds the upper limit of the confidence interval for 3 consecutive time points and the difference between the upper limit and the upper limit is greater than the preset threshold and the abnormal state lasts for at least 45 minutes, an early warning is triggered.

[0051] The technical solution is described below: A comprehensive anomaly handling mechanism is constructed to ensure the continuity and accuracy of the prediction service. Data missing anomaly handling employs a multi-layered imputation strategy. Interpolation utilizes valid data points before and after the missing data, calculating and filling in the missing values ​​based on time distance or numerical trends. Regression-based prediction imputation trains a specialized regression model, using other relevant features to predict the missing usage rate. Data replacement methods include internal and external replacement. Internal replacement extracts replaceable historical data from other system logs or caches, while external replacement temporarily calls backup data interfaces or uses historical data from the most recent period as a substitute, ensuring the prediction service can continue to operate even when data is missing. The system sets up a three-tiered judgment condition for predictive anomaly alerts. The first tier is the continuous exceedance condition, which monitors the predicted values ​​of CPU or memory usage. When the predicted value exceeds the upper limit of the confidence interval for three consecutive time points, a preliminary alert is triggered. The second tier is the significant deviation condition, which calculates the difference between the predicted value and the upper limit of the confidence interval. When the difference is greater than a preset threshold, the degree of anomaly is further confirmed. The third tier is the duration condition. When the first two conditions are met and the abnormal state lasts for at least 45 minutes, the system automatically triggers the alert mechanism, promptly sending alert information to operations and maintenance personnel to pay attention to the resource usage status of the corresponding equipment and avoid service interruption.

[0052] The technical effects achieved by the above embodiments include: ensuring the continuity of prediction services when data is missing, timely identifying abnormal resource usage, reducing the risk of service interruption, and improving operation and maintenance response efficiency.

[0053] Traditional technical solutions have the following technical problems: the warning threshold is set in a single way and does not distinguish between anomalies of different urgency levels, resulting in low alarm accuracy and insufficient targeted operation and maintenance response.

[0054] Based on this, the upper bound of the confidence interval has a CPU utilization threshold of 90% and a memory utilization threshold of 90%; the warning includes critical alarms and warning alarms, with a resource utilization threshold of 95% for critical alarms and 85% for warning alarms.

[0055] The technical solution is described below: A tiered early warning threshold system is established to improve alarm accuracy and the targeted nature of operational responses. First, a basic threshold is set for the upper bound of the confidence interval. The upper bound of the confidence interval for both CPU utilization and memory utilization is set to 90%, serving as the basic standard for anomaly detection. When the predicted value approaches this threshold, it indicates high resource load. Based on this, early warnings are divided into two levels: a critical alarm with a resource utilization threshold of 95%. When the predicted value of CPU or memory utilization reaches this threshold, it indicates that the device resources are severely overloaded and may face service interruption risks, triggering the highest level critical alarm; and a warning alarm with a resource utilization threshold of 85%. When the predicted value reaches this threshold but not 95%, it indicates that the device resource load is high but not severely overloaded, triggering a warning alarm. Operations personnel can take targeted measures according to different alarm levels. Critical alarms are prioritized for rapid reduction of resource load, while warning alarms are optimized by allocating time reasonably to avoid excessive maintenance or untimely responses.

[0056] The technical effects achieved by the above embodiments include: refining the anomaly alarm levels, improving alarm accuracy, making operation and maintenance responses more targeted, and effectively avoiding resource overload or excessive operation and maintenance.

[0057] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for predicting CPU and memory usage based on clustering and Prophet, comprising collecting historical CPU usage data and historical memory usage data of a server, characterized in that, It also includes: performing preprocessing on the collected historical data, including noise removal and missing value imputation; automatically dividing the data into device groups with similar load patterns based on the K-Means clustering algorithm to solve the prediction adaptation problem caused by the load differences of multiple devices; training a shared Prophet prediction model for each device group to adapt to the load cycle characteristics of the device group and solve the problems of model redundancy and high operation and maintenance costs when predicting multiple devices; and outputting the prediction results of the server's future CPU utilization and memory utilization through the Prophet prediction model.

2. The CPU and memory utilization prediction method based on clustering and Prophet according to claim 1, characterized in that, The preprocessing also includes equal-interval resampling, using Pandas' .resample() method to group data at 5-minute or 15-minute frequencies, and averaging the data within each time window to generate a regular time series.

3. The CPU and memory utilization prediction method based on clustering and Prophet according to claim 1, characterized in that, The noise removal process specifically involves: marking individual data points with CPU or memory usage exceeding 80% as potential outliers; and determining that if the percentage of data points exceeds 80%, or exceeds 60%, it is considered a normal business peak and no removal is performed. If the proportion does not exceed 60%, the potential outlier will be set as a missing value.

4. The CPU and memory utilization prediction method based on clustering and Prophet according to claim 1, characterized in that, The missing value imputation adopts a two-level strategy: the first level uses Pandas' .interpolate(method='time') method to perform time-weighted interpolation; if there are still missing values ​​after the first level, forward imputation is used, and if there are still missing values ​​after forward imputation, backward imputation is used.

5. The CPU and memory utilization prediction method based on clustering and Prophet according to claim 1, characterized in that, The K-Means clustering algorithm execution process includes: extracting overall statistical features and refined time-granular features to construct a multi-dimensional feature vector; using the Z-Score standardization method to convert all features into a distribution with a mean of 0 and a standard deviation of 1; determining the optimal number of clusters by evaluating the silhouette coefficient, traversing all possible values ​​from 2 to the maximum number of clusters, and selecting the number of clusters with the largest silhouette coefficient.

6. The CPU and memory utilization prediction method based on clustering and Prophet according to claim 5, characterized in that, The overall statistical features include the arithmetic mean of CPU utilization, the standard deviation of CPU utilization, the difference between the maximum and minimum CPU utilization, the maximum value of the average CPU utilization per hour during the day, and the four features mentioned above corresponding to memory utilization. The refined time granularity features include 24-hour granularity features and 7-day granularity features. The 24-hour granularity features are the average CPU utilization, standard deviation of CPU utilization, average memory utilization, and standard deviation of memory utilization for each hour of the day. The 7-day granularity features are the average CPU utilization and average memory utilization for each day of the week.

7. The CPU and memory utilization prediction method based on clustering and Prophet according to claim 1, characterized in that, The K-Means clustering algorithm uses the K-Means++ algorithm to initialize the cluster centers, and sets the n_init=10 parameter for multiple initialization optimizations; the iteration termination condition is that the relative change in the position of the cluster center is less than 1e-4 or the number of iterations reaches 300.

8. The CPU and memory utilization prediction method based on clustering and Prophet according to claim 1, characterized in that, The training of the Prophet prediction model includes: sorting the preprocessed historical data in chronological order, selecting the data from the last day as the test set, and using the remaining data as the training set; customizing parameters for different device groups, using multiplication mode for CPU metrics for some device groups, and uniformly using addition mode for memory metrics; the changepoint_prior_scale parameter has a value range of 0.1-0.4, the seasonality_prior_scale parameter has a value range of 0.6-15.0, and the n_changepoints parameter has a value range of 10-22.

9. The CPU and memory utilization prediction method based on clustering and Prophet according to claim 1, characterized in that, It also includes anomaly handling mechanisms: when data is missing, interpolation, prediction based on regression models, or data replacement are used to fill the gaps; when the predicted value of CPU utilization or memory utilization exceeds the upper limit of the confidence interval for three consecutive time points, the difference between the predicted value and the upper limit is greater than the preset threshold, and the abnormal state lasts for at least 45 minutes, an early warning is triggered.

10. The CPU and memory utilization prediction method based on clustering and Prophet according to claim 1, characterized in that, The upper bound of the confidence interval has a CPU utilization threshold of 90% and a memory utilization threshold of 90%. The warning includes critical warnings and warning warnings. The resource utilization threshold for critical warnings is 95%, and the resource utilization threshold for warning warnings is 85%.

Images