Time series analysis for predicting computational workloads

Abstract

A time series analysis technique for predicting computational workloads is disclosed. A resource management system monitors the computing system and acquires metric data from it. This metric data is stored as a historical data point set of the dataset. A first portion of this historical data point set includes outliers that do not correspond to seasonal patterns associated with a second portion of the historical data point set. The resource management system tests a first time series model incorporating a first exogenous variable corresponding to a first exogenous factor to determine if the first time series model fits both the first and second portions of the historical data point set within an error threshold. The resource management system then selects the first time series model to predict future data points in the dataset.

Description

Technical Field

[0001] This disclosure relates to the analysis of time series data. In particular, this disclosure relates to techniques for performing time series analysis for predicting computational workloads.

[0002] Claims; related applications; incorporation by reference

[0003] The subject matter of this application relates to the subject matter of a co-pending non-provisional application entitled "System for Detecting and Characterizing Seasons" with serial number 15 / 057,065 and filed on February 29, 2016, which is incorporated herein by reference.

[0004] The subject matter of this application relates to the subject matter of a co-pending non-provisional application entitled “Supervised Method for Classifying Seasonal Patterns”, serial number 15 / 057,060, filed on February 29, 2016, which is incorporated herein by reference.

[0005] The subject matter of this application relates to the subject matter of a co-pending non-provisional application entitled “Unsupervised Method for Classifying Seasonal Patterns”, serial number 15 / 057,062, filed on February 29, 2016, which is incorporated herein by reference.

[0006] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 939,603, filed November 23, 2019, which is incorporated herein by reference.

[0007] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 901,088, filed September 16, 2019, which is incorporated herein by reference.

[0008] The applicant hereby withdraws any declaration of waiver of the scope of claims in the parent applications(one or more) or their examination history, and informs the United States Patent and Trademark Office (USPTO) that the claims in this application may be more extensive than any claims in the parent applications(one or more). Background Technology

[0009] Applications and data are increasingly migrating from on-premises systems to cloud-based Software as a Service (SaaS) systems. In such cloud-based systems, computing resources such as processors, memory, storage devices, networks, and / or disk input / output (I / O) can be consumed by entities and / or components such as physical machines, virtual machines, applications, application servers, databases, database servers, services, and / or transactions.

[0010] On the other hand, cloud service providers must ensure that cloud-based systems have sufficient resources to meet customer needs and requirements. For example, cloud service providers can perform capacity planning involving estimating the resources required to run customer applications, databases, services, and / or servers. Cloud service providers can also monitor the performance of customer systems to detect performance degradation, errors, and / or other problems. However, because such monitoring techniques are reactive, errors, failures, and / or outages may occur on the system before remedial measures are taken to correct or mitigate the problems.

[0011] The methods described in this section are feasible methods, but not necessarily methods that have been previously conceived or implemented. Therefore, unless otherwise stated, no method described in this section should be assumed to be prior art solely because it is included in this section. Attached Figure Description

[0012] Embodiments are illustrated in the accompanying drawings by way of example and not limitation. It should be noted that references to "a" or "one" embodiments in this disclosure do not necessarily refer to the same embodiment; they mean at least one. In the drawings:

[0013] Figure 1 The illustration shows a system according to one or more embodiments;

[0014] Figure 2 The illustration depicts a resource management system according to one or more embodiments;

[0015] Figure 3 The diagram illustrates a flowchart of performing time series analysis to predict computational workloads according to one or more embodiments;

[0016] Figure 4 The diagram illustrates a flowchart of training a time series model according to one or more embodiments;

[0017] Figure 5 The diagram illustrates a flowchart for selecting a model with exogenous variables according to one or more embodiments;

[0018] Figure 6 The illustration shows a flowchart of anomaly detection using predicted computational workloads according to one or more embodiments;

[0019] Figure 7 The illustration shows an embodiment of a time series model including exogenous variables according to one or more embodiments; and

[0020] Figure 8 A block diagram illustrating a computer system according to one or more embodiments is shown. Detailed Implementation

[0021] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding. One or more embodiments may be practiced without these specific details. Features described in one embodiment may be combined with features described in different embodiments. In some examples, well-known structures and devices are described in block diagram form to avoid unnecessarily obscuring the invention.

[0022] 1. General Overview

[0023] 2. System Architecture

[0024] 3. Time series analysis for predicting computational workloads

[0025] 4. Anomaly detection using predicted computational workloads

[0026] 5. Example implementation of time series models including exogenous variables

[0027] 6. Computer networks and cloud networks

[0028] 7. Miscellaneous; Extension

[0029] 8. Hardware Overview

[0030] 1. General Overview

[0031] One or more embodiments analyze historical system data and incorporate exogenous variables into a time series model trained with that historical system data. The system detects outlier data points within the dataset that do not match the seasonal patterns corresponding to other data points in the dataset. To generate a time series model that accurately represents the entire dataset (including outlier data points), the system tests the time series model using combinations of various exogenous variables or using exogenous variables with different parameters representing different exogenous factors. The system selects a time series model that incorporates one or more exogenous factors and represents outliers within an error threshold.

[0032] 2. System Architecture

[0033] Figure 1 A system 100 according to one or more embodiments is illustrated. Figure 1 As shown, system 100 includes computing system 110, application server 120, resource management system 130 and user interface 140.

[0034] Computing system 110 is a system managed by resource management system 130. Computing system 110 includes one or more data repositories 111 and one or more nodes 112, 113 configured to interact with data repositories 111, with each other and other nodes, and with application server 120 to perform workloads. The computing system may include components such as one or more data centers, collocation centers, cloud computing systems, local systems, clusters, content delivery networks, server racks, and / or other combinations of processing, storage, networking, input / output (I / O), and / or other resources.

[0035] like Figure 2 As shown, the resource management system 130 includes a monitoring module 131, which has the function of monitoring and / or managing the utilization or consumption of resources on the computing system 110. For example, the monitoring module 131 may collect and / or monitor metrics related to the utilization and / or workload of processors, memory, storage devices, networks, I / O, thread pools, and / or other types of hardware and / or software resources. The monitoring module 131 may also, or alternatively, collect and / or monitor performance metrics such as latency, queries per second (QPS), error count, garbage collection count, and / or garbage collection time related to resources.

[0036] Furthermore, the resource management system 130 can perform such monitoring and / or management at different granularity levels and / or for different entities. For example, the resource management system 130 can assess resource utilization and / or workload at the environment, cluster, host, virtual machine, database, database server, application, application server, transaction (e.g., a series of clicks on a website or a web application completing an online order) and / or data (e.g., database records, metadata, request / response attributes, etc.) level. The resource management system 130 can additionally use a set of entity attributes to define entities and perform monitoring and / or analysis based on metrics associated with the entity attributes. For example, the resource management system 130 can identify entities as combinations of customers, metric types (e.g., processor utilization, memory utilization, etc.), and / or granularity levels (e.g., virtual machines, applications, databases, application servers, database servers, transactions, etc.).

[0037] The monitoring module 131 stores metrics related to the workload of the computing system 110 in the data storage 200. The stored metrics constitute historical data 210. Historical data 210 includes time-series data and may include one or more of the following characteristics: seasonality 211, multi-seasonality 212, trend 213, shock, or outlier 214.

[0038] The resource management system 130 includes a training module 133 that uses machine learning techniques to generate time-series models for various entities associated with the monitored system. The training module 133 obtains historical time-series data 210 for a given entity (e.g., a combination of customers, metrics, and granularity levels) from a data repository 200. The training module 133 partitions the historical time-series data into a training dataset 133a, a test dataset 133b, and a validation dataset 133c. The training module 133 trains a set of time-series models using the training dataset 133a and tests these models using the test dataset 133b. The training module 133 validates the models using the validation set 133c. Based on training, testing, and validation, the training module 133 generates a selection of one or more time-series models for evaluating subsequent time-series metrics. The selection of time-series models can be obtained from the data repository 200.

[0039] The resource management system 130 includes a prediction module 134 that uses a time-series model generated by a training module 133 to generate predictions of metrics representing resource consumption and / or workload on a monitored system. In these embodiments, the time-series model analysis includes time-series data of metrics collected from the monitored system to estimate future values ​​in the time-series data based on previously observed values ​​in the time-series data.

[0040] In one or more embodiments, the filled time series model is stored in data repository 200 for later forecasting. Time series model 220 includes one or more of the following models: Holt-Winters exponential smoothing (HES) model and triangular seasonal Box-Cox ARMA trend and seasonality (TBATS) model 220a; autoregressive integral moving average (ARIMA) model 220b; seasonal ARIMA (SARIMAX) model 220c with exogenous variables having parameters (p, d, q, P, D, Q, frequency); or any combination of these models or alternative models.

[0041] Time series model 220 includes components 221a to 221d to account for seasonality 211, multi-seasonality 212, trend 213, and shocks or outliers 214 in historical time series data 210. Components of time series model 221 also include a Fourier term 221e, which is added as an external regressor to either ARIMA model 220b or SARIMAX model 220c when multi-seasonality 212 is present in the historical data 210. These components of time series model 221 improve the accuracy of the model and allow model 220 to be adapted to various types of time series data collected from the monitored system. In one embodiment, time series model 220 includes an exogenous variable 221d that accounts for outliers 214 in historical time series data 210, such that outliers 214 in the model generated from historical time series data 210 do not affect the metrics in the predictions of prediction module 134.

[0042] In one or more embodiments, the time series model 220 includes one or more variants of the autoregressive integral moving average (ARIMA) model 220b and / or the exponential smoothing model 220a.

[0043] In some embodiments, ARIMA model 220b is a generalization of an autoregressive moving average (ARMA) model having the following representation:

[0044]

[0045] The above representation can be simplified as follows:

[0046] φ p (B)Y i =θ q (B)a i

[0047] In the above representation, Y t This represents the values ​​Y, φ1, ..., φ in a time series indexed by time step t. p The autoregressive parameters to be estimated are θ1, θ2, θ3, θ4, θ5, θ6, θ7, θ8, θ9, θ1, θ1, θ9 ...9, θ1, θ9, θ9, θ9, q Let a1, a2 be the moving average parameters to be estimated. t It represents a series of unknown random errors (or residuals) that are assumed to follow a normal distribution.

[0048] In one embodiment, training module 133 utilizes the Box-Jenkins method to detect the presence or absence of stationarity and / or seasonality in historical time series data 210. For example, the Box-Jenkins method may utilize autocorrelation function (ACF), partial ACF, correlation plot, spectrogram, and / or another technique for evaluating stationarity and / or seasonality in time series.

[0049] When training module 133 determines that only nonstationarity is found, training module 133 can add the difference degree d to the ARMA model to produce an ARIMA model with the following form:

[0050] φ p (B)(1-B) d γ d =B q (B)a i

[0051] When training module 133 determines that seasonality has been discovered, training module 133 can add the seasonal component to the ARIMA model to produce a seasonal ARIMA (SARIMA) model with the following form:

[0052] φ P (B)Φ(P)(B * (1-B) d (1-B * ) D Y t θ q (B)θ Q (B * )a t

[0053] In the SARIMA model, parameters 120p, d, and q represent the trend elements of the autoregressive, difference, and moving average orders, respectively; parameters 120P, D, and Q represent the seasonal elements of the autoregressive, difference, and moving average orders, respectively; and parameter s represents the number of seasons in the time series (e.g., hourly, daily, weekly, monthly, yearly, etc.).

[0054] In one or more embodiments, training module 133 applies Fourier term 221e to time series model 220. For example, when multiple seasons are detected in the time series, seasonal patterns can be represented using Fourier term 221e, which is added to the ARIMA model as an external regressor.

[0055]

[0056] In the equation above, N t It is an ARIMA process, P1, ..., P M It represents a time period in a time series (e.g., hourly, daily, weekly, monthly, yearly, etc.), and the Fourier terms are included as a weighted sum of sine and cosine pairs.

[0057] The time series model 220 may include exogenous variables 224 that consider outliers 214 in historical data 210 and represent external influences and / or shocks. In one embodiment, the training module 133 adds the exogenous variables 224 to the ARMA model above to produce an autoregressive moving average (ARMAX) model with exogenous inputs having the following representation:

[0058]

[0059] In the above representation, β1, ..., β r These are parameters of the time-varying exogenous input X. In an additional embodiment, training module 133 includes exogenous components in ARIMA and / or SARIMA models. In computing system 110, exogenous variables may represent system backups, batch jobs, periodic failover, and / or other external factors affecting workload, resource utilization, and / or other metrics in the time series. These external factors may cause spikes in workload metrics that do not follow potential seasonal patterns in the historical time series data 210.

[0060] In one or more embodiments, the exponential smoothing model includes the Triangular Seasonal Box-Cox ARMA Trend Seasonal Component (TBATS) model. The TBATS model includes the following representation:

[0061]

[0062] l c =l i-1 +φ-b i-1 +α·d

[0063] b i =φ·b i-1 +S·d i

[0064]

[0065] In the above representation, T represents the number of seasons, and m i Let y be the length of the i-th seasonal period. t(λ) Let s be the time series after Box-Cox transformation at time t. t (i) For the i-th seasonal component, l t For level, b t To exhibit a damping effect, d t It is an ARMA(p,q) process, and e t It is Gaussian white noise with zero mean and constant variance. Additionally, Φ is the trend damping coefficient, and α and β are smoothing coefficients. θ and θ are ARMA(p,q) coefficients.

[0066] The seasonal component of the TBATS model is represented using the following:

[0067]

[0068]

[0069]

[0070]

[0071] In the equation above, k i is the harmonic number required for the i-th seasonal time period, λ is the Box-Cox transform, and γ1 (i) and γ2 (i) This represents the smoothing parameter.

[0072] Therefore, the TBATS model has parameters 120T and m i k i , λ, α, β, θ, γ1 (i) and γ2 (i) The final model can be selected from alternatives using the Akaike Information Standard (AIC), which includes (but is not limited to):

[0073] With and without Box-Cox transformation

[0074] With and without trends

[0075] With and without trend damping

[0076] • ARMA(p,q) processes with and without residual modeling

[0077] • With and without seasonality

[0078] • Changes in the number of harmonics used to model seasonal effects

[0079] The prediction module 134 obtains the time series of the most recently collected metrics for each entity from the data store 200 and inputs the data into the corresponding time series model generated by the training module 133. The time series model then outputs estimates 135 of future values ​​in the time series as estimated workload, resource utilization, and / or performance associated with the entity. These predictions can detect potential future anomalies, errors, interruptions, and / or failures in the operation of hardware and / or software resources associated with the entity.

[0080] When an anomaly is predicted in a metric for a given entity, the resource management system 130 transmits the predicted anomaly to one or more users involved in managing that entity's use of the monitored system. For example, the resource management system 130 may include a user interface 140 or a user interface 140 that can send information to a graphical user interface (GUI), a web-based user interface, a mobile user interface, a voice user interface, and / or other types of user interfaces that display a graph of metrics as a function of time. The graph also includes one or more thresholds for the metric and / or a representation of predicted values ​​of the metric from a time-series model of the corresponding entity. When a predicted value violates a given threshold, the user interface displays a highlight, coloring, shading, and / or other indication of a violation of the predicted future anomaly or problem in the entity's use of the monitored system.

[0081] In another example, the resource management system 130 may generate alerts, notifications, emails, and / or other communications of anticipated anomalies to the administrator of the monitored system, allowing the administrator to take preventative measures (e.g., allocating and / or providing additional resources for the entity to use before resource utilization leads to failure or disruption).

[0082] The prediction module 134 includes an staleness determination module 136, which performs a cyclical analysis of the selected model to determine whether the model is stale. For example, if the staleness determination module 136 determines that the model is more than a week old or has an error rate exceeding a predetermined threshold, the resource management system 130 can retrain the model using the most recent historical data 210 obtained by the training module 133 and the monitoring module 131.

[0083] In one or more embodiments, the resource management system 130 may include a ratio Figure 2 The components shown may have more or fewer components. For example, training module 133 and monitoring module 131 may include each other, be executed together, or be mutually exclusive. Figure 2 The components shown can be located locally or remotely to each other. Figure 2 The components shown can be implemented using software and / or hardware. Each component can be distributed across multiple applications and / or machines. Multiple components can be combined into one application and / or machine. Operations described with respect to one component can be performed alternatively by another component.

[0084] Additional embodiments and / or examples relating to computer networks are described below in Part 5, entitled “Computer Networks and Cloud Networks”.

[0085] In one or more embodiments, the data repository (e.g., data repository 200) is any type of storage unit and / or device (e.g., file system, database, table collection, or any other storage mechanism) used for storing data. The data repository may be implemented on or executed on the same computing system as the training module 133 and the monitoring module 131, or on a separate computing system from the training module 133 and the monitoring module 131. The data repository may be coupled to the training module 133 and the monitoring module 131 via a direct connection or via network communication. Furthermore, the data repository may include multiple different storage units and / or devices. These multiple different storage units and / or devices may or may not be of the same type or located at the same physical site.

[0086] In one or more embodiments, the resource management system 130 refers to hardware and / or software configured to perform the operations described herein for predicting computational workloads. Examples of such operations are described below.

[0087] In this embodiment, the resource management system 130 is implemented on one or more digital devices. The term "digital device" generally refers to any hardware device that includes a processor. A digital device can refer to a physical device that executes an application or a virtual machine. Examples of digital devices include computers, tablets, laptops, desktops, netbooks, servers, web servers, network policy servers, proxy servers, general-purpose machines, feature-specific hardware devices, hardware routers, hardware switches, hardware firewalls, hardware network address translation (NAT), hardware load balancers, mainframes, televisions, content receivers, set-top boxes, printers, mobile handheld terminals, smartphones, personal digital assistants ("PDAs"), wireless receivers and / or transmitters, base stations, communication management equipment, routers, switches, controllers, access points, and / or client devices.

[0088] In one or more embodiments, a user interface refers to hardware and / or software configured to facilitate communication between a user and the resource management system 130. The user interface presents user interface elements and receives input via these elements. Examples of interfaces include graphical user interfaces (GUIs), command-line interfaces (CLIs), haptic interfaces, and voice command interfaces. Examples of user interface elements include checkboxes, radio buttons, drop-down lists, list boxes, buttons, toggle keys, text fields, date and time pickers, command lines, sliders, pages, and forms.

[0089] In this embodiment, different components of the user interface are specified in different languages. The behavior of user interface elements is specified in a dynamic programming language such as JavaScript. The content of user interface elements is specified in a markup language such as Hypertext Markup Language (HTML) or XML User Interface Language (XUL). The layout of user interface elements is specified in a stylesheet language such as Cascading Style Sheets (CSS). Alternatively, the user interface may be specified in one or more other languages ​​such as Java, C, or C++.

[0090] 3. Time series analysis for predicting computational workloads

[0091] Figure 3 The diagram illustrates a flowchart of performing time-series analysis to predict computational workloads according to one or more embodiments. In one or more embodiments, one or more steps may be omitted, repeated, and / or performed in a different order. Therefore, Figure 3 The specific arrangement of the steps shown should not be construed as limiting the scope of the embodiments.

[0092] Initially, the resource management system of the monitored system obtains historical time-series data containing metrics collected from the monitored system (Operation 301). The resource management system can obtain historical time-series data for a given entity (e.g., a combination of customers, metrics, and granularity levels) from a data repository. For example, the resource management system can match an entity's entity attributes with records in the database that store historical time-series data for that entity (e.g., metrics collected from that entity over the past week, month, year, and / or other time periods). Each record may include a metric value, a timestamp indicating the time when the value was generated, and / or an index indicating the value's position in the time series.

[0093] The resource management system trains at least one time series model on historical data (Operation 302).

[0094] Figure 4 The process of a resource management system training a time series model on historical data is illustrated. In one or more embodiments, one or more steps may be omitted, repeated, and / or performed in a different order. Therefore, Figure 4 The specific arrangement of the steps shown should not be construed as limiting the scope of the embodiments.

[0095] The resource management system divides historical time-series data into training and testing datasets to train a set of time-series patterns (Operation 401). For example, the resource management system can populate the training dataset with the majority of the time-series data (e.g., 70-80%) and the testing dataset 116 with the remaining time-series data. In some embodiments, the resource management system chooses the size of the testing dataset to represent the prediction range of each time-series model, depending on the granularity of the time-series data. For example, the resource management system may include in the testing dataset 24 observations across a day for data collected hourly; seven observations across a week for data collected daily; and / or four observations across approximately a month for data collected weekly. The resource management system may optionally use cross-validation techniques to generate multiple training and testing datasets from the same time-series data.

[0096] The resource management system performs adjustments to reduce the number of models to be analyzed (operation 402). In one embodiment, the resource management system utilizes an autocorrelation function (ACF) or a partial autocorrelation function (PACF) to find multiple autoregressive terms to use in the time series model. For example, by using an autocorrelation function, a set of time series data is replicated and the replicas are adjusted to lag the original time series dataset. By comparing the original time series dataset with multiple replicas having different lag times, patterns in the historical data, such as seasonality, can be identified.

[0097] By performing adjustment operations, the resource management system determines whether historical data includes seasonal patterns (operation 403), multi-seasonal patterns (operation 404), trends (operation 405), and outliers or shocks (operation 406). Based on the identified characteristics of the historical time series data, the resource management system selects a specific time series model that is likely to fit the historical data very well. For example, in an embodiment where the resource management system calculates the ACF / PACF and identifies multi-seasonal patterns in historical data, the resource management system can select a model such as an ARIMA-type model to be trained on the historical data. In an embodiment where the ACF / PACF calculation identifies outliers or shocks in historical data, the resource management system can select a model such as a SARIMAX-type model to be trained on the historical data. The resource management system can choose from several different types of models to be trained on historical data, and different types of models can fit the training dataset for evaluation. For example, the resource management system can calculate the ACF / PACF and determine that both the ARIMA-type model and the SARIMAX-type model have similar likelihood of fitting the historical data.

[0098] When identifying one or more types of models as potential fits to historical data, the resource management system fits multiple versions of the selected time series models to the training dataset (operation 407). Specifically, the resource management system uses the training dataset to train a set of time series models with different parameters (operation 408). For example, the resource management system can use the Box-Jenkins method and / or other methods to generate a search space of parameters for various ARIMA-type models and / or TBATS-type models. In the embodiment where an ARIMA-type model is selected to fit the training dataset, there are parameters (p, d, q), and the resource management system uses different parameter values ​​to identify the search space of parameters. The resource management system then uses maximum likelihood estimation (MLE), ordinary least squares (OLS), and / or another technique to fit each model to the training dataset.

[0099] After the resource management system creates a set of time series models from the training dataset, it uses a test dataset to evaluate the performance of each model (Operation 409). Specifically, the resource management system uses the time series models to generate estimates of the values ​​in the test dataset based on previously observed values ​​in the time series data. The resource management system also determines the accuracy of the time series models based on comparisons between the estimates and corresponding values ​​in the test dataset. For example, the resource management system calculates the mean squared error (MSE), root mean squared error (RMSE), AIC, and / or another metric of model quality or accuracy for all time series models generated from the entity's historical time series data.

[0100] In one embodiment, the time series model includes exogenous variables to account for peaks or outliers in historical data. In one embodiment, the future data points predicted by the time series model do not incorporate any influence of the exogenous variables. In an alternative embodiment, the future data points predicted by the time series model incorporate the influence of the exogenous variables by accepting the values ​​of the exogenous variables as input. Additionally, or in an alternative, in one embodiment, the time series model incorporates the influence of the exogenous variables on the future data points predicted by the first time series model by reducing the weight given to the exogenous variables relative to other variables in a first time series model representing seasonal patterns in historical data.

[0101] In one embodiment, the resource management system utilizes the Fourier transform of the time series model to determine the accuracy of the time series model. The resource management system can apply the Fourier transform to the time series model to compare the time series model with a test dataset to determine the accuracy of the corresponding time series model.

[0102] Finally, the resource management system generates a selection of one or more time series models for evaluating subsequent time series metrics of the same entity (operation 410). For example, the resource management system selects one or more time series models that have the highest accuracy when including values ​​in the estimated test dataset.

[0103] After one or more best-performing time series models are selected for one or more entities, the resource management system stores the parameters of each model in a model repository. The resource management system also, or alternatively, provides representations of the models to monitoring modules, user interfaces, and / or other components of the resource management system. In one or more embodiments, the best-performing time series model includes components that consider seasonality, multi-seasonality, and shocks or outliers in historical time series data. These components of the time series model improve the model's accuracy and allow the model to adapt to various types of time series data collected from the monitored system. In one embodiment, the time series model includes exogenous variables that consider outliers in historical time series data, such that outliers in the model generated using historical time series data do not affect the metrics in the resource management system's forecasts.

[0104] In one embodiment, the resource management system applies Fourier transforms to a time series model. For example, when multiple seasons are detected in a time series, seasonal patterns can be represented using Fourier terms.

[0105] return Figure 3 In one or more embodiments, the resource management system uses a time series model selected by the resource management system to generate predictions of time series metrics (operation 303). For example, the resource management system may predict workloads and / or utilization related to processors, memory, storage devices, networks, I / O, thread pools, and / or other types of resources in the monitored system.

[0106] To generate forecasts, the resource management system feeds the time series of the most recently collected metrics for each entity into the corresponding time series model. The time series model then outputs estimates of future values ​​in the time series as estimated workload, resource utilization, and / or performance associated with the entity.

[0107] The resource management system may additionally include the ability to predict anomalies based on a comparison of predictions with corresponding thresholds. For example, a threshold may represent a limitation on an entity's resource utilization and / or a service level target for performance metrics associated with the entity. When a predicted metric violates (e.g., exceeds) a corresponding threshold, the resource management system can detect potential future anomalies, errors, interruptions, and / or failures in the operation of hardware and / or software resources associated with the entity.

[0108] When an anomaly is predicted in a metric for a given entity, the resource management system communicates the predicted anomaly to one or more users involved in managing that entity's use of the monitored system. For example, the resource management system may include a graphical user interface (GUI), a web-based user interface, a mobile user interface, a voice user interface, and / or other types of user interfaces that display a graph of the metric as a function of time. The graph may also include one or more thresholds for the metric and / or a representation of predicted values ​​of the metric from a time-series model of the corresponding entity. When a predicted value violates a given threshold, the user interface displays a highlight, coloring, shading, and / or other indication of a violation of the predicted future anomaly or problem in the entity's use of the monitored system. In another example, the monitoring module may generate alerts, notifications, emails, and / or other communications regarding the predicted anomaly to the administrator of the monitored system, allowing the administrator to take preventative measures (e.g., allocating and / or providing additional resources for the entity's use before resource utilization leads to failure or disruption).

[0109] The resource management system continuously monitors the time series model used to predict future metrics of entities to determine if the model is outdated (Operation 304). The resource management system determines the time series model is outdated if the error rate of the time series model exceeds a predetermined threshold or if a predetermined time period has elapsed. According to one embodiment, the resource management system determines the time series model is outdated if the root mean square error (RMSE) is below 95% accuracy. Alternative embodiments cover time series models at any desired level of accuracy. Furthermore, or alternatively, the resource management system may determine the time series model is outdated if more than one week has elapsed since the time series model was trained. While one week is provided as an example of a timeframe for determining whether a time series model is outdated, the embodiments cover any time period that can be adjusted based on historical data and the granularity of the forecast.

[0110] After a period of time has elapsed since a given time series model was trained to generate predictions and / or estimates of anomalies, the resource management system retrains the time series model using more recent time series data from the corresponding entities (Operation 301). For example, the resource management system may periodically acquire and / or generate new training and test datasets from metrics collected over recent days, weeks, months, and / or other durations. The resource management system can use the new training dataset to generate a set of time series models with different combinations of parameter values ​​and use the new test dataset to evaluate the accuracy of the generated time series models. The resource management system can then select one or more of the most accurate and / or best-performing time series models to include in the model repository and / or for use by the monitoring module to generate predictions and / or estimates of anomalies for entities in subsequent time periods.

[0111] If the resource management system determines that the time series model is not outdated, the resource management system obtains the time series of newly collected metrics for each entity (operation 305) and provides the newly collected metrics to the time series model to predict new future values ​​(operation 306).

[0112] Figure 5 The process of a resource management system selecting a model that includes one or more exogenous variables is illustrated in more detail. In one or more embodiments, one or more steps may be omitted, repeated, and / or performed in a different order. Therefore, Figure 4 The specific arrangement of the steps shown should not be construed as limiting the scope of the embodiments.

[0113] The resource management system identifies seasonal patterns in a set of historical data points (operation 501). The resource management system determines whether at least a portion of the historical data points include outliers or shocks (operation 502). In one embodiment, the resource management system utilizes an autocorrelation function (ACF) or a partial autocorrelation function (PACF) to find multiple autoregressive terms to be used in the time series model and to identify whether the historical data includes outliers or shocks. However, embodiments of the invention are not limited to these types of functions. Embodiments cover any function that can be applied to historical data to identify outliers in the historical data.

[0114] If the resource management system identifies one or more outliers in the historical data, it tests a time series model incorporating a seasonal pattern and one or more exogenous variables (operation 503). The resource management system determines whether the time series model fits the historical data within a predetermined threshold (operation 504). If the time series model fits the historical data within the predetermined threshold, the resource management system selects the time series model as a candidate for predicting future metrics (operation 505). For example, refer to... Figure 4 The resource management system compares and selects candidate models to determine which has the best performance, and chooses the best time series model to predict future metrics. Figure 4 (Operation 410).

[0115] If the resource management system determines that historical data does not include outliers or shocks (Operation 502), then the resource management system tests a time series model that does not include exogenous variables (Operation 506).

[0116] 4. Anomaly detection using predicted computational workloads

[0117] Figure 6 A flowchart illustrating anomaly detection using predicted computational workloads according to one or more embodiments is shown. In one or more embodiments, one or more steps may be omitted, repeated, and / or performed in a different order. Therefore, Figure 6The specific arrangement of the steps shown should not be construed as limiting the scope of the embodiments.

[0118] Initially, the resource management system selects the version of the time series model that has the best performance in predicting metrics from multiple versions of the time series model fitted to historical time series data containing metrics collected from the monitored system (Operation 601). For example, a version can be selected from multiple versions with different combinations of parameters used to create the time series model.

[0119] Next, the resource management system applies the selected version to additional time-series data collected from the monitored system to generate estimates of future values ​​from the metrics (operation 602). For example, the selected version generates estimates based on previously observed metric values.

[0120] The resource management system monitors projected metrics and detects when a projected metric violates a predetermined threshold (operation 603). When a projected metric violates a predetermined threshold associated with it, the resource management system generates an indication of an anomaly in the projected metric within the monitored system (operation 604). For example, the projected future value is compared to a threshold representing an upper limit for the metric (e.g., 80% resource utilization). When some or all of the projected future values ​​exceed the threshold, alarms, notifications, and / or other communications regarding the violated threshold are generated and sent to the administrator of the monitored system.

[0121] 5. Example implementation of time series models including exogenous variables

[0122] Figure 7 The illustration shows a graph illustrating the use of a time series model to predict a metric according to an embodiment of the present invention. Figure 7 In the diagram, line 710 represents the measured metric, which can constitute historical data used by the resource management system to train time series models for predicting the metric values ​​of the monitored system. Line 720, shown as a dashed line, represents the prediction generated by the resource management system, while line 730, shown as a dotted line, represents the cumulative average and is displayed for comparison purposes.

[0123] In embodiments of the invention, the time series model may include variables or components to take into account seasonality (such as recurring peaks or troughs at specific hours of the day), multi-seasonality (such as recurring peaks or troughs at specific dates of the week, and recurring peaks and troughs at specific hours of the day), and trends (such as increasing or decreasing over time, exhibiting both seasonality and multi-seasonality).

[0124] The embodiments also include time series models with exogenous variables to account for spikes or outliers indicated by reference numerals 711 and 712. Spikes or outliers may correspond to events such as irregularly scheduled backups of systems requiring significant resources, or any external event that causes spikes in metrics that occur infrequently or irregularly in historical data. In embodiments where the time series model includes exogenous variables, these spikes or outliers can be considered in the model, and the values ​​of the outliers are not incorporated into future estimates or predictions. In other words, while spikes or outliers in the historical data used to train the time series model may generally have a tendency to bias estimates toward values ​​that are spikes or outliers, embodiments where the time series model includes exogenous variables are able to account for spikes or outliers when the model is trained on historical data. Because exogenous variables are included in the time series model, the predictions generated by the time series model are not skewed by spikes or outliers in the historical data used to train the time series model.

[0125] 6. Computer networks and cloud networks

[0126] In one or more embodiments, a computer network provides connectivity between a set of nodes. Nodes may be local to each other and / or geographically distant. Nodes are connected by a set of links. Examples of links include coaxial cable, unshielded twisted pair, copper cable, fiber optic cable, and virtual links.

[0127] A subset of nodes implements a computer network. Examples of such nodes include switches, routers, firewalls, and Network Address Translation (NAT). Another subset of nodes uses a computer network. Such nodes (also referred to as "hosts") can execute client processes and / or server processes. Client processes request computing services (such as the execution of a specific application and / or the storage of a specific amount of data). Server processes respond by performing the requested service and / or returning the corresponding data.

[0128] A computer network can be a physical network, including physical nodes connected by physical links. A physical node is any digital device. A physical node can be a function-specific hardware device, such as a hardware switch, hardware router, hardware firewall, and hardware NAT. Additionally or alternatively, a physical node can be a general-purpose machine configured to run various virtual machines and / or applications performing corresponding functions. A physical link is the physical medium connecting two or more physical nodes. Examples of links include coaxial cable, unshielded twisted-pair cable, copper cable, and fiber optic cable.

[0129] Computer networks can be overlay networks. An overlay network is a logical network implemented on top of another network (such as a physical network). Each node in the overlay network corresponds to a corresponding node in the underlying network. Therefore, each node in the overlay network is associated with an overlay address (to address the overlay node) and an underlying address (to address the underlying node that implements the overlay node). Overlay nodes can be digital devices and / or software processes (such as virtual machines, application instances, or threads). The links connecting overlay nodes are implemented as tunnels through the underlying network. Overlay nodes at either end of the tunnel treat the underlying multi-hop path between them as a single logical link. Tunneling is performed through encapsulation and decapsulation.

[0130] In this embodiment, the client may be local to the computer network and / or remote from the computer network. The client may access the computer network via other computer networks, such as a private network or the Internet. The client may use a communication protocol such as Hypertext Transfer Protocol (HTTP) to transmit requests to the computer network. Requests are transmitted through interfaces such as client interfaces (such as web browsers), program interfaces, or application programming interfaces (APIs).

[0131] In this embodiment, a computer network provides connectivity between clients and network resources. Network resources include hardware and / or software configured to execute server processes. Examples of network resources include processors, data storage devices, virtual machines, containers, and / or software applications. Network resources are shared among multiple clients. Clients independently request computing services from the computer network. Network resources are dynamically assigned to requesting and / or clients on an on-demand basis. The network resources assigned to each requesting and / or client may be scaled up or down based on, for example, (a) computing services requested by a specific client, (b) aggregated computing services requested by a specific tenant, and / or (c) aggregated computing services requested by the computer network. Such a computer network may be referred to as a "cloud network."

[0132] In this embodiment, the service provider offers a cloud network to one or more end users. The cloud network can implement various service models, including but not limited to Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). In SaaS, the service provider provides end users with the ability to use the service provider's applications running on network resources. In PaaS, the service provider provides end users with the ability to deploy custom applications onto network resources. Custom applications can be created using programming languages, libraries, services, and tools supported by the service provider. In IaaS, the service provider provides end users with the ability to provision processing, storage, networking, and other basic computing resources provided by network resources. Any application, including operating systems, can be deployed on network resources.

[0133] In embodiments, various deployment models can be implemented via computer networks, including but not limited to private clouds, public clouds, and hybrid clouds. In a private cloud, network resources are provisioned for exclusive use by a specific group of one or more entities (as used herein, "entity" refers to a company, organization, individual, or other entity). Network resources may be located locally or remotely from the premises of the specific entity group. In a public cloud, cloud resources are provisioned for multiple entities (also referred to as "tenants" or "customers") that are independent of each other. The computer network and its network resources are accessed by clients corresponding to different tenants. Such a computer network may be referred to as a "multi-tenant computer network." Several tenants may use the same specific network resources at different times and / or simultaneously. Network resources may be located locally or remotely from the tenant's premises. In a hybrid cloud, the computer network includes both private and public clouds. The interface between the private and public clouds allows for the portability of data and applications. Data stored in the private cloud and data stored in the public cloud can be exchanged through the interface. Applications implemented in the private cloud and applications implemented in the public cloud may have dependencies on each other. You can use an interface to make calls from an application in the private cloud to an application in the public cloud (and vice versa).

[0134] In this embodiment, the tenants of a multi-tenant computer network are independent of each other. For example, one tenant's business or operations may be separate from those of another tenant. Different tenants may have different network requirements for the computer network. Examples of network requirements include processing speed, data storage capacity, security requirements, performance requirements, throughput requirements, latency requirements, resilience requirements, quality of service (QoS) requirements, tenant isolation, and / or consistency. The same computer network may need to meet the different network requirements of different tenants.

[0135] In one or more embodiments, tenant isolation is implemented in a multi-tenant computer network to ensure that applications and / or data from different tenants are not shared with each other. Various tenant isolation methods can be used.

[0136] In this embodiment, each tenant is associated with a tenant ID. Each network resource in a multi-tenant computer network is tagged with a tenant ID. A tenant is only allowed to access a specific network resource if the tenant and the specific network resource are associated with the same tenant ID.

[0137] In this embodiment, each tenant is associated with a tenant ID. Each application implemented by the computer network is tagged with a tenant ID. Additionally or alternatively, each data structure and / or dataset stored by the computer network is tagged with a tenant ID. A tenant is only allowed access to a specific application, data structure, and / or dataset if the tenant and the specific application, data structure, and / or dataset are associated with the same tenant ID.

[0138] For example, each database implemented in a multi-tenant computer network can be tagged with a tenant ID. Only the tenant associated with the corresponding tenant ID can access the data in a specific database. As another example, each entry in a database implemented in a multi-tenant computer network can be tagged with a tenant ID. Only the tenant associated with the corresponding tenant ID can access the data in that specific entry. However, the database can be shared by multiple tenants.

[0139] In this embodiment, the subscription list indicates which tenants are authorized to access which applications. For each application, a list of tenant IDs of tenants authorized to access that application is stored. A tenant is only allowed to access a specific application if its tenant ID is included in the subscription list corresponding to that specific application.

[0140] In this embodiment, network resources corresponding to different tenants (such as digital devices, virtual machines, application instances, and threads) are isolated from tenant-specific overlay networks maintained by the multi-tenant computer network. As an example, packets from any source device within a tenant overlay network can be sent only to other devices within the same tenant overlay network. Encapsulation tunneling is used to prevent any transmission from a source device on one tenant overlay network to devices in other tenant overlay networks. Specifically, packets received from the source device are encapsulated within an outer packet. The outer packet is sent from a first encapsulation tunnel endpoint (communicating with the source device in the tenant overlay network) to a second encapsulation tunnel endpoint (communicating with the destination device in the tenant overlay network). The second encapsulation tunnel endpoint decapsulates the outer packet to obtain the original packet sent by the source device. The original packet is then sent from the second encapsulation tunnel endpoint to the destination device within the same specific overlay network.

[0141] 7. Miscellaneous; Extension

[0142] The embodiments relate to a system having one or more devices, the one or more devices including a hardware processor and configured to perform any of the operations described herein and / or recited in any of the following claims.

[0143] In an embodiment, a non-transitory computer-readable storage medium includes instructions that, when executed by one or more hardware processors, cause to perform any operation described herein and / or recited in any of the claims.

[0144] Any combination of the features and functions described herein may be used according to one or more embodiments. In the foregoing specification, embodiments have been described with reference to numerous specific details that may vary with implementation. Therefore, the specification and drawings are to be considered illustrative rather than restrictive. The sole and exclusive indication of the scope of the invention, and what the applicant intends to define as the scope of the invention, is the literal and equivalent scope of the set of claims published in this application, including any subsequent corrections, in the specific form of such claims.

[0145] 8. Hardware Overview

[0146] According to one embodiment, the technology described herein is implemented by one or more dedicated computing devices. The dedicated computing device may be hardwired to execute the technology, or may include digital electronic devices permanently programmed to execute the technology, such as one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or network processing units (NPUs), or may include one or more general-purpose hardware processors programmed to execute the technology according to program instructions in firmware, memory, other storage devices, or combinations thereof. Such a dedicated computing device may also combine custom hardwired logic, ASICs, FPGAs, or NPUs with custom programming to implement the technology. The dedicated computing device may be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device incorporating hardwired and / or program logic to implement the technology.

[0147] For example, Figure 8 This is a block diagram illustrating a computer system 800 on which embodiments of the present invention may be implemented. The computer system 800 includes a bus 802 or other communication mechanism for transmitting information, and a hardware processor 804 coupled to the bus 802 for processing information. The hardware processor 804 may be, for example, a general-purpose microprocessor.

[0148] Computer system 800 also includes main memory 806, such as random access memory (RAM) or other dynamic storage device, coupled to bus 802, for storing information and instructions to be executed by processor 804. Main memory 806 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 804. When such instructions are stored in a non-transitory storage medium accessible to processor 804, computer system 800 presents itself as a dedicated machine customized to perform the operations specified in the instructions.

[0149] The computer system 800 also includes a read-only memory (ROM) 808 or other static storage device coupled to the bus 802 for storing static information and instructions for the processor 804. A storage device 810, such as a disk or optical disk, is provided and coupled to the bus 802 for storing information and instructions.

[0150] Computer system 800 can be coupled to display 812, such as a cathode ray tube (CRT), via bus 802 for displaying information to the computer user. Input device 814, including alphanumeric keys and other keys, is coupled to bus 802 for transmitting information and command selections to processor 804. Another type of user input device is cursor control 816, such as a mouse, trackball, or cursor direction keys for transmitting directional information and command selections to processor 804 and for controlling cursor movement on display 812. This input device typically has two degrees of freedom in two axes (a first axis (e.g., x) and a second axis (e.g., y)), which allows the device to specify a position in a plane.

[0151] Computer system 800 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware and / or program logic (which, in combination with the computer system, cause computer system 800 to become a special-purpose machine or program computer system 800 as a special-purpose machine). According to one embodiment, the techniques herein are executed by computer system 800 in response to processor 804 executing one or more sequences of one or more instructions contained in main memory 806. Such instructions may be read into main memory 806 from another storage medium (such as storage device 810). Execution of the sequence of instructions contained in main memory 806 causes processor 804 to perform the processing steps described herein. In alternative embodiments, hardwired circuitry may be used instead of or in combination with software instructions.

[0152] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs or magnetic disks, such as storage device 810. Volatile media include dynamic memory, such as main memory 806. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips or cassette disks, content-addressable memory (CAM), and tri-state content-addressable memory (TCAM).

[0153] Storage media differ from transmission media but can be used in conjunction with them. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including wires containing bus 802. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0154] Carrying one or more sequences of instructions to processor 804 for execution may involve various forms of media. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit them over a telephone line using a modem. A modem local to computer system 800 may receive data over the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on bus 802. Bus 802 carries the data to main memory 806, from which processor 804 retrieves and executes the instructions. The instructions received by main memory 806 may optionally be stored on storage device 810 before or after execution by processor 804.

[0155] Computer system 800 also includes a communication interface 818 coupled to bus 802. Communication interface 818 provides bidirectional data communication coupling to a network link 820 connected to a local network 822. For example, communication interface 818 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem providing a data communication connection to a corresponding type of telephone line. As another example, communication interface 818 may be a local area network (LAN) card providing a data communication connection to a LAN-compatible network. A wireless link may also be implemented. In any such implementation, communication interface 818 transmits and receives electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information.

[0156] Network link 820 typically provides data communication to other data devices via one or more networks. For example, network link 820 may provide a connection via local network 822 to host computer 824 or to data devices operated by Internet Service Provider (ISP) 826. ISP 826 then provides data communication services via a global packet data communication network now commonly referred to as the “Internet” 828. Both local network 822 and Internet 828 use electrical, electromagnetic, or optical signals that carry streams of digital data. Signals through various networks and on network link 820 and through communication interface 818 (which carry digital data to and from computer system 800) are example forms of transmission media.

[0157] Computer system 800 can send messages and receive data, including program code, through one or more networks, network links 820, and communication interfaces 818. In the Internet example, server 830 can send application request codes through the Internet 828, ISP 826, local network 822, and communication interface 818.

[0158] The received code may be executed by processor 804 upon receipt and / or stored in storage device 810 or other non-volatile storage device for later execution.

[0159] In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary with implementation. Therefore, the specification and drawings are to be considered illustrative rather than restrictive. The unique and exclusive indications of the scope of the invention, and what the applicant intends to define as the scope of the invention, are the literal and equivalent scope of the set of claims published in this application, including any subsequent corrections, in the specific form of such claims.

Claims

1. One or more non-transitory machine-readable media, the one or more non-transitory machine-readable media storing instructions, which, when executed by one or more processors, cause to perform operations including: Receive a historical data point set of a dataset, wherein the dataset includes metrics related to the utilization and / or workload of hardware and / or software resources on the computing system; The first portion of the historical data point set is determined to include at least one outlier that does not correspond to at least one seasonal pattern associated with the second portion of the historical data point set; The test incorporates the first time series model corresponding to the first exogenous variable of the first exogenous factor to determine whether the first time series model fits the first part of the historical data point set and the second part of the historical data point set within the error threshold. The first time series model is selected to predict future data points in the dataset; The staleness of a first-time-series model is determined by identifying the following: Is the first time series model older than the threshold expiration value? Or Does the first time series model have a root mean square error (RMSE) below a threshold percentage? as well as Based on the determination that the first time series model is outdated. Receive the first new set of historical data points from the second dataset; The test incorporates the second time series model corresponding to the second exogenous variable of the second exogenous factor; as well as A second time series model was chosen to predict future data points in the second dataset.

2. One or more non-transitory machine-readable media as claimed in claim 1, wherein the future data points predicted by the first time series model do not incorporate any influence of the first exogenous variable.

3. One or more non-transitory machine-readable media as claimed in claim 1, wherein the future data points predicted by the first time series model incorporate the influence of the exogenous variable by accepting the value of the first exogenous variable as input.

4. One or more non-transitory machine-readable media as claimed in claim 1 or 3, wherein the first time series model incorporates the influence of the first exogenous variable on future data points predicted by the first time series model by reducing the weight given to the first exogenous variable relative to a first variable in the first time series model representing the at least one seasonal pattern.

5. One or more non-transitory machine-readable media as claimed in any one of claims 1-4, wherein the operation further comprises: The test incorporates a third time series model with a third exogenous variable corresponding to the third exogenous factor to determine whether the third time series model fits both the first and second portions of the historical data set within the stated error threshold; and Avoid selecting a third time series model based on testing a third time series model.

6. One or more non-transitory machine-readable media as claimed in any one of claims 1-4, wherein the operation further comprises: The first score of the first time series model is determined based on the fit between the first time series model and both the first part and the second part of the historical data point set. as well as The second score of the third time series model is determined based on the fit between the third time series model and both the first and second parts of the historical data point set. In response to determining that a first time series model fits the dataset better than a third time series model based on a first score and a second score, the first time series model is selected to predict future data points of the dataset.

7. One or more non-transitory machine-readable media as claimed in any one of claims 1-4, wherein the operation further comprises: The staleness of a first-time-series model is also determined by identifying the following: The first time series model has a root mean square error (RMSE) of less than 95%.

8. One or more non-transitory machine-readable media as claimed in any one of claims 1-4, wherein the operation further comprises: The staleness of a second time series model is determined by identifying the following: Is the second time series model older than the threshold expiration value? or Does the second time series model have a root mean square error (RMSE) of less than 95%? Based on the premise that the second time series model is not outdated Receive the second new set of historical data points from the third dataset; as well as The first time series model is used to predict future data points in the third dataset.

9. One or more non-transitory machine-readable media as described in any one of claims 1-8, wherein testing the first time series model comprises: Perform a Fourier transform on the first time series model.

10. One or more non-transitory machine-readable media as claimed in any one of claims 1-9, wherein the first time series model includes the first exogenous variable and variables in the historical data corresponding to the at least one seasonal pattern and at least one multi-seasonal pattern.

11. One or more non-transitory machine-readable media as claimed in any one of claims 1-10, wherein the dataset is obtained from the workload of a computing system, and The operation also includes: Based on the estimated future data points, suggestions for modifying the computing system are generated.

12. A method comprising: Receive a historical data point set of a dataset, wherein the dataset includes metrics related to the utilization and / or workload of hardware and / or software resources on the computing system; The first portion of the historical data point set is determined to include at least one outlier that does not correspond to at least one seasonal pattern associated with the second portion of the historical data point set; The test incorporates the first time series model corresponding to the first exogenous variable of the first exogenous factor to determine whether the first time series model fits the first part of the historical data point set and the second part of the historical data point set within the error threshold. The first time series model is selected to predict future data points in the dataset; The staleness of a first-time-series model is determined by identifying the following: Is the first time series model older than the threshold expiration value? Or Does the first time series model have a root mean square error (RMSE) below a threshold percentage? as well as Based on the determination that the first time series model is outdated. Receive the first new set of historical data points from the second dataset; The test incorporates the second time series model corresponding to the second exogenous variable of the second exogenous factor; as well as A second time series model was chosen to predict future data points in the second dataset.

13. The method of claim 12, wherein the future data points predicted by the first time series model do not incorporate any influence of the first exogenous variable.

14. The method of claim 12 or 13, further comprising: The staleness of a first-time-series model is determined by identifying the following: The first time series model has a root mean square error (RMSE) of less than 95%.

15. The method of claim 12 or 13, further comprising: The staleness of a second time series model is determined by identifying the following: Is the second time series model older than the threshold expiration value? or Does the second time series model have a root mean square error (RMSE) of less than 95%? Based on the premise that the second time series model is not outdated Receive the second new set of historical data points from the third dataset; as well as The second time series model is used to predict future data points in the third dataset.

16. The method of any one of claims 12-15, wherein testing the first time series model comprises: Perform a Fourier transform on the first time series model.

17. The method of any one of claims 12-16, wherein the first time series model comprises the first exogenous variable and variables in the historical data corresponding to the at least one seasonal pattern and the at least one multi-seasonal pattern.

18. The method of any one of claims 12-17, wherein the dataset is obtained from the workload of the computing system, and The method further includes: Based on the estimated future data points, suggestions for modifying the computing system are generated.

19. An apparatus comprising: The monitoring module is configured to monitor the system to obtain measurement data from the system and store the measurement data as historical data in a data storage repository; The training module is configured as follows: Receive a historical data point set of a dataset, wherein the dataset includes metrics related to the utilization and / or workload of hardware and / or software resources on the computing system; The first portion of the historical data point set is determined to include at least one outlier that does not correspond to at least one seasonal pattern associated with the second portion of the historical data point set; The test incorporates the first time series model corresponding to the first exogenous variable of the first exogenous factor to determine whether the first time series model fits the first part of the historical data point set and the second part of the historical data point set within the error threshold. The first time series model is selected to predict future data points in the dataset; The staleness of a first-time-series model is determined by identifying the following: Is the first time series model older than the threshold expiration value? Or Does the first time series model have a root mean square error (RMSE) below a threshold percentage? as well as Based on the determination that the first time series model is outdated. Receive the first new set of historical data points from the second dataset; The test incorporates the second time series model corresponding to the second exogenous variable of the second exogenous factor; as well as A second time series model was chosen to predict future data points in the second dataset.

20. An apparatus comprising: A monitoring component is used to monitor the system to obtain measurement data from the system and to store the measurement data as historical data in a data storage repository. The training component is configured as follows: Receive a historical data point set of a dataset, wherein the dataset includes metrics related to the utilization and / or workload of hardware and / or software resources on the computing system; The first portion of the historical data point set is determined to include at least one outlier that does not correspond to at least one seasonal pattern associated with the second portion of the historical data point set; The test incorporates the first time series model corresponding to the first exogenous variable of the first exogenous factor to determine whether the first time series model fits the first part of the historical data point set and the second part of the historical data point set within the error threshold. The first time series model is selected to predict future data points in the dataset; The staleness of a first-time-series model is determined by identifying the following: Is the first time series model older than the threshold expiration value? Or Does the first time series model have a root mean square error (RMSE) below a threshold percentage? as well as Based on the determination that the first time series model is outdated. Receive the first new set of historical data points from the second dataset; The test incorporates the second time series model corresponding to the second exogenous variable of the second exogenous factor; as well as A second time series model was chosen to predict future data points in the second dataset.

21. A system comprising: One or more processors; as well as One or more non-transitory machine-readable media storing instructions that, when executed by the one or more processors, cause the method as described in any one of claims 12-18 to be performed.

22. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 12-18.

Images