A method for analyzing the predictability of metrics from monitoring agents in computer networks.

Abstract

We provide systems, methods, apparatus, and computer program products for predicting network metrics. [Solution] The method includes the steps of: a computing device receiving metric data; extracting at least one time series characteristic from the metric data; calculating at least one statistical measurement based on the at least one time series characteristic; performing ensemble scoring based on the at least one statistical measurement; generating a predictability score based on the ensemble scoring; and determining the predictability of the metric data based on a comparison of the predictability score with a pre-set threshold.

Description

Technical Field

[0001] Some exemplary embodiments generally relate to predicting network metrics between monitoring agents within a computer network.

Background Art

[0002] Metric prediction is an important aspect of modern information technology (IT) environments. Such metric predictions allow IT systems to take proactive measures against future consumption patterns of computer resources. However, predictions may be unreliable due to various factors including uncertainties in operation patterns and external influences.

Summary of the Invention

Problems to be Solved by the Invention

[0003] According to some exemplary embodiments, one method includes receiving metric data by a computing device. The method further includes extracting at least one time series characteristic from the metric data by the computing device. The method further includes calculating at least one statistical measurement based on the at least one time series characteristic by the computing device. The method further includes performing ensemble scoring based on the at least one statistical measurement by the computing device. The method further includes generating a predictability score based on the ensemble scoring by the computing device. The method further includes determining the predictability of the metric data based on a comparison between the predictability score and a pre-set threshold by the computing device.

[0004] According to a particular exemplary embodiment, the apparatus may include means for receiving metric data. The apparatus may further include means for extracting at least one time-series characteristic from the metric data. The apparatus may further include means for calculating at least one statistical measure based on the at least one time-series characteristic. The apparatus may further include means for performing ensemble scoring based on the at least one statistical measure. The apparatus may further include means for generating a predictability score based on the ensemble scoring. The apparatus may further include means for determining the predictability of the metric data based on a comparison of the predictability score with a pre-set threshold.

[0005] According to various exemplary embodiments, a non-transient computer-readable medium may include program instructions that, when executed by a device, cause the device to perform at least one method. The method may include the step of receiving metric data. The method may further include the step of extracting at least one time-series characteristic from the metric data. The method may further include the step of calculating at least one statistical measure based on the at least one time-series characteristic. The method may include the step of performing ensemble scoring based on the at least one statistical measure. The method may further include the step of generating a predictability score based on the ensemble scoring. The method may further include the step of determining the predictability of the metric data based on a comparison of the predictability score with a pre-set threshold.

[0006] According to some exemplary embodiments, a computer program product can perform a method. The method may include the step of receiving metric data. The method may further include the step of extracting at least one time-series characteristic from the metric data. The method may further include the step of calculating at least one statistical measure based on the at least one time-series characteristic. The method may include the step of performing ensemble scoring based on the at least one statistical measure. The method may further include the step of generating a predictability score based on the ensemble scoring. The method may further include the step of determining the predictability of the metric data based on a comparison of the predictability score with a pre-set threshold.

[0007] According to a particular exemplary embodiment, the apparatus may include at least one processor and at least one memory for storing instructions that, when executed by the at least one processor, cause the apparatus to receive at least metric data. The at least one memory and instructions may, when executed by the at least one processor, cause the apparatus to receive at least further metric data. The at least one memory and instructions may, when executed by the at least one processor, cause the apparatus to calculate at least one statistical measure based on at least one time-series characteristic. The at least one memory and instructions may, when executed by the at least one processor, cause the apparatus to perform ensemble scoring based on at least one statistical measure. The at least one memory and instructions may, when executed by the at least one processor, cause the apparatus to further generate a predictability score based on at least the ensemble scoring. The at least one memory and instructions may, further, when executed by the at least one processor, cause the apparatus to determine the predictability of the metric data based on at least a comparison of the predictability score with a preset threshold.

[0008] According to various exemplary embodiments, the apparatus may include a receiving circuit configured to perform the receiving of metric data. The apparatus may further include an extraction circuit configured to perform the step of extracting at least one time-series characteristic from the metric data. The apparatus may further include a calculation circuit configured to perform the step of calculating at least one statistical measurement based on the at least one time-series characteristic. The apparatus may further include an execution circuit configured to perform ensemble scoring based on the at least one statistical measurement. The apparatus may further include a generation circuit configured to perform the step of generating a predictability score based on the ensemble scoring. The apparatus may further include a determination circuit configured to perform the step of determining the predictability of the metric data based on a comparison of the predictability score with a preset threshold. [Brief explanation of the drawing]

[0009] To properly understand the exemplary embodiments, it is recommended to refer to the attached drawings. [Figure 1] An example of a flowchart of the method according to a specific exemplary embodiment is shown. [Figure 2] Based on several exemplary embodiments, we show the calculation of the spectrum |X(ωi)|2 of a signal computed using the Fourier transform, which can transform a time-based signal into the frequency domain by estimating the frequency of the original signal. [Figure 3] The coefficient of variation and the quartile coefficient are shown as measures of variability / variance in various exemplary embodiments. [Figure 4] The coefficient of variation and the quartile coefficient are shown as measures of variability / variance in various exemplary embodiments. [Figure 5] The coefficient of variation and the quartile coefficient are shown as measures of variability / variance in various exemplary embodiments. [Figure 6]This shows a shift in Power Spectral Density (PSD) that affects spectral entropy, according to a specific exemplary embodiment. [Figure 7] This shows a shift in the PSD that affects the Omega score, according to a specific exemplary embodiment. [Figure 8] The following are noisy plots of the average CPU load from the machine in several exemplary embodiments. [Figure 9] Seasonality plots of average disk reads from the machine are shown for various exemplary embodiments. [Figure 10] Examples of various network devices according to specific exemplary embodiments are shown. [Modes for carrying out the invention]

[0010] It will be readily apparent that the components of a particular exemplary embodiment, as described and illustrated in whole in the figures of this specification, can be arranged and designed in a wide variety of configurations. Therefore, the following detailed descriptions of some exemplary embodiments of systems, methods, apparatus, and computer program products for predicting network metrics are not intended to limit the scope of any particular exemplary embodiment, but rather to represent representative examples of selected exemplary embodiments.

[0011] Artificial intelligence operation (AIOp) service providers can offer metric forecasting services. However, these forecasting services may not provide end-users with guidance on which metrics to forecast, the quality of the forecasts may be uncertain, and computing resources may be unnecessarily consumed to run the forecasting service even when the metrics are unpredictable.

[0012] Certain exemplary embodiments described herein offer various advantages and / or benefits for overcoming the aforementioned shortcomings. For example, certain exemplary embodiments can determine whether a computer-based metric is predictable. Thus, the specific exemplary embodiments described below address improvements in computer-related technologies.

[0013] To build confidence in the model, predictability can be assessed using time series quality metrics. Such quality metrics can be based on a score indicating the predictability of the time series and can distinguish between different types of time series (e.g., smooth, spiky, or regime-changing series). Quality metrics can be aggregated across multiple time series (disks / volumes) within a metric (e.g., percentage of disks used, average node volume, etc.) to provide an accurate estimate of the metric's predictability. The time series can be sampled for each metric.

[0014] Figure 1 shows an example of a flowchart of Method 100 performed by a computing device such as the NE 1010 shown in Figure 10, according to various exemplary embodiments.

[0015] In step 101, the method includes the step of receiving metric data by a computing device. In a particular exemplary embodiment, the metric data may include at least one of the following: out-percentage utilization (i.e., current utilization of the transmitting bandwidth of the network interface), average load, average percentage of memory used, average response time, total average input / output operations per second, average read input / output operations per second, average write input / output operations per second, average total latency, average read latency, average write latency, percentage of disk used, and any other type of metric data.

[0016] In step 102, the method can further include the step of extracting at least one time series characteristic from the metric data by the computing device using various statistical and mathematical techniques, such as change point detection for identifying trends and seasonality, and peak detection algorithms based on statistical measures for identifying spikes and dips (temporary drops).

[0017] In some exemplary embodiments, the at least one time series characteristic can include at least one of a trend, seasonality, periodicity, spike, and dip.

[0018] In step 103, the method can further include the step of calculating at least one statistical measurement by the computing device based on the at least one time series characteristic. In various exemplary embodiments, the at least one statistical measurement can include at least one of variance, spectral density, residual variability, and omega score.

[0019] According to various exemplary embodiments, residual variability can be determined by training a baseline model on training window data (e.g., profit prediction by Python or R) and performing predictions on the same training window (e.g., overfitting the model to determine how profit responds to the trends and seasonality of the data). The residuals can be calculated as JPEG2026108528000002.jpg516 (where JPEG2026108528000003.jpg44 is the actual value, and JPEG2026108528000004.jpg44 is the predicted value of a given timestamp). The residuals can potentially capture additional variability in the time series that may not be captured by the baseline model. The standard deviation of the residuals is a measure of the noise present in the time series and can be scaled using the mean of the actual values and the decay factor. Therefore, JPEG2026108528000005.jpg571). Next, the final forecastability score FCST (Forecastability) is JPEG2026108528000006.jpg424. FCST can also be calculated as a weighted combination of any of the aforementioned scores.

[0020] In certain exemplary embodiments, the step of calculating at least one statistical measurement can further include the step of calculating the power spectral density of at least one signal by the computing device by taking the spectrum of at least one signal and squaring the amplitude of the at least one signal. Next, the computing device can normalize the at least one signal by a number of intervals and make the calculated power spectral density a probability density function. The method can further include the step of the computing device calculating the power spectral entropy according to a standard entropy calculation formula and the step of determining whether the normalized power spectral density represents a predictable series or a non - predictable noisy series.

[0021] Spectral Entropy (SE) can include a non - linear procedure for generalizing the irregularity of signal power over the measured frequencies and can be used as a measure to describe the complexity of a time series. Spectral JPEG2026108528000007.jpg514 converts a time-based signal to the frequency domain by estimating the frequencies that may constitute the original signal, as shown in Figure 2. The power spectral density of the signal is given by the formula According to JPEG2026108528000008.jpg626, the amplitude of the spectrum can be calculated by squaring it and normalizing it by the number of intervals. The calculated Power Spectral Density (PSD) is: It can be considered as a probability density function according to JPEG2026108528000009.jpg718. Next, the power spectral entropy (PSE) can be expressed using the standard formula for entropy calculation, for example. The calculation can be performed using JPEG2026108528000010.jpg526. Normalized PSE can be calculated as a value of 0 for predictable sequences and a value of 1 for unpredictable and noisy sequences. JPEG2026108528000011.jpg411 can be given a predictability score of Omega. Therefore, The image is JPEG2026108528000012.jpg420, where smaller omega values ​​indicate lower predictability, and larger omega values ​​indicate higher predictability.

[0022] According to some exemplary embodiments, the method may further include the step of calculating the omega score by subtracting the power spectral entropy from 1. As an example, predictability can be demonstrated by calculating the omega score as less than 1.

[0023] In step 104, the method may further include the step of having the computing device perform ensemble scoring based on at least one statistical measurement.

[0024] In some exemplary embodiments, the ensemble scoring may include the step of selecting a suitable method for generating a predictability score from candidate procedures, including any combination of FCST, spectral entropy, and / or coefficient of variation (i.e., variance). In particular, FCST can assess the predictability of a metric by fitting Prophet's model to the metric data and evaluating the residuals. FCST can efficiently capture seasonal and cyclical trends and may be more reliable for metrics with practical data (e.g., data spanning multiple days). Furthermore, spectral entropy may be preferable for metrics with relatively few data points. Spectral entropy can decompose a time series into individual time and frequency components, separate noise, and evaluate the resulting component for predictability. Spectral entropy may be particularly useful when FCST is undesirable due to noisy (i.e., irregular patterns) and / or sparse (i.e., data collected infrequently or data that changes little over time) data points. Furthermore, variance can be used to calculate the variability of metric data within the range of variability that determines the predictability measured by FCST and / or spectral entropy, as described above.

[0025] In step 105, the method may further include the step of generating a predictability score based on the ensemble scoring by the computing device. For example, the predictability score may be the result of the ensemble scoring in step 104. The predictability score may be based on one selected method (e.g., FCST, spectral entropy, coefficient of variation, etc.) or an aggregated score (e.g., mean) of several selected methods based on any combination of seasonal patterns, noise, and a large number of observed data points.

[0026] In step 106, the method may further include the step of determining the predictability of metric data based on a comparison between the predictability score and a pre-configured threshold (the pre-configured threshold is determined by statistically aggregating the predictability scores stored as history) by the computing device.

[0027] In step 107, if the method determines that the metric data is predictable, the computing device may further include the step of enabling metric forecasting. For example, metric forecasting may include the step of the computing device using time-series data of the metric to predict metric values ​​for future periods. Enabling metric forecasting may include the step of enabling the computing device to perform metric forecasting for a selected set of metrics that are considered predictable.

[0028] In step 108, the method may further include the computing device stopping metric forecasting if it determines that the metric data is unpredictable. Disabling metric forecasting may include the computing device stopping performing metric forecasting for a selected set of metrics that are deemed unpredictable.

[0029] In step 109, the method may further include the step of having a computing device perform at least one periodic predictability check.

[0030] As described above, the predictability metric (i.e., the Omega Score) can indicate the uncertainty of a given time series. The Omega Score can be aggregated over a given time series within the metric and can indicate the predictability of the metric. To determine the Omega Score, the Omega Score of a metric can be calculated using three days of minute / hour data. Three days of data sufficient to generalize the predictability of a metric can be demonstrated by the Pearson correlation score between the three days of data and all days (e.g., 10 days) of data. A high correlation score may occur when the metric has more predictable time series, such as flat trends and logistic growth trends. A low correlation score may indicate that the metric has data with varying trends, and that instability may not be constant across all days.

[0031] Tables 1-6 below provide a set of metrics that assess predictability with the best and minimum scores along the Omega score range, in order to improve the interpretation of predictability. These metrics cover a variety of signal types, from flat-trending signals to highly volatile signals. [Table 1] CPULoad_CS-AvgLoad has a useful mean omega score due to multiple mostly flat time series with small, intermittent spikes. Upon manual inspection, some of these time series may be periodic and / or predictable after transformations on the data. [Table 2] The VIM_CloudInstanceStatistics_Detail range is useful, but 92% of the data has an Omega Score of 1 due to a flat trend. The remaining 8% have a maximum Omega Score of 0.139, which is due to a highly volatile graph and makes this metric unpredictable. SRM_LUNStatistics_Detail has only 28 hashes, 16 of which have a flat trend. The remaining data has intermittent spikes in addition to a flat trend, resulting in a low Omega Score (<0.3). SRM_VolumeStatistics_Detail has only 15 hashes, all of which have a flat trend with intermittent spikes, and therefore the given Omega Score is even lower. [Table 3] SEUM_ResponseTime_Detail-Duration has only 17 data points, 10 of which are flat. Therefore, its high score of 0.603 may not provide much of an indicator. The remaining 7 hashes have low omega scores of less than 0.1, due to a flat trend with many intermittent spikes. SEUM_ResponseTime_Daily consists of many volatile time series, covering a wide range of values, all of which achieve omega scores of less than 0.1. [Table 4] CPULoad_CS_Daily and CPULoad_CS_Hourly have a distribution of Omega scores similar to the DiskCapacity metric and are very likely to be predictable. VIM_ClusterStatistics_CS_Detail_hist has a highly volatile graph that is unpredictable. The signal within the time series frequently drops to zero, and therefore decreases the Omega score. MemoryMultiLoad_Detail may also be unpredictable in a similar way. [Table 5] VIM_ClusterStatistics_CS_Detail_hist only shows a flat trend and is therefore safe to ignore. All other metrics contain signals that fluctuate intermittently between 0 and 100. Therefore, these metrics can be unpredictable. [Table 6] The metric has only nine hashes that exhibit a non-flat trend. These may have low omega scores due to their irregularity, thus making the metric unpredictable.

[0032] Thus, the Omega Score provides a general assessment of the predictability of a metric as a measure of randomness in the time series of each hashed ID. However, given the characteristics of time series modeling, this measurement does not provide a complete picture. It indicates that one metric is more volatile, but does not indicate whether this volatility is suitable for using a model like Prophet. Predictable metrics based on Omega Score comparison include CPULoad and MemoryUsage. Other metrics with intermittent spikes and flat trends can be made predictable using several data transformations, such as the use of scaling and filtering.

[0033] In some exemplary embodiments, the predictability of data can be based on variance, which can be determined by the coefficient of variation of time series data, a measure of relative variability as the ratio of the standard deviation to the mean. Specifically, the predictability of data can be interpreted as the amount of variability in the data, and more variability can be correlated with less predictability of the series. The coefficient of variation and the interquartile coefficient can be measures of variability / variance of data, as shown in Figures 3–5. In particular, Figures 3–5 show the coefficient of variation and the interquartile coefficient as measures of variability / variance by specific exemplary embodiments. In Figure 3, the values ​​in the different plot (a method used to remove the effects of trend and seasonality in time series) indicate high variability (e.g., 194.3). In Figure 4, the different plot is much smoother and has a single peak, resulting in a low coefficient of variation (e.g., 2.237). In Figure 5, the different plot shows a very large spike, resulting in a very high absolute value of the coefficient of variation (e.g., 3528). The reason we choose absolute value scores is that while the mean can be negative, we are only interested in the degree of variability.

[0034] To mitigate the errors that arise from measuring only the data distribution, a difference operation can be performed on a series of values ​​(i.e., the difference between consecutive values), and then the coefficient of variation can be measured. This will result in a very large value because the mean is close to zero when subtracting a series of values. [Table 7] The measure of variability does not fully reflect the predictability of the time series. It is beneficial to further acquire and measure the trends, seasonality, and noise components of the time series, to determine predictability using the fit and predictability uncertainties in Prophet, and to redefine predictability as the minimum error that our model can acquire and measure using a baseline method for all forecasts across multiple time periods.

[0035] To demonstrate the relationship between PSD and omega score, random spikes can be added to the signal, and the modified metric can be compared to the original metric. As shown in Figures 6 and 7, the shift in PSD affects spectral entropy and omega score, resulting in a decrease in the omega score. The table below compares the omega score and FCST score. [Table 8]

[0036] When predicting data with a 70% capacity threshold, the following table shows the deviation accuracy at 1, 5, 10, 15, 20, and 20+ (OOF: Out Of Fold) days during prediction. The distribution of Omega scores can be seen as different from the FCST scores for each precision interval. [Table 9] [Table 10] [Table 11]

[0037] Figure 8 shows a plot of time-series data of average CPU load from a machine, with random, non-seasonal spikes and dips. This time-series data yields a low predictability score.

[0038] Figure 9 shows a seasonality plot of average disk reads from the machine, with uniform spikes and dips that have a high predictability score.

[0039] Figure 10 shows an example of a system according to a specific exemplary embodiment. In one exemplary embodiment, the system may include multiple devices, such as NE 1010.

[0040] NE 1010 can be one or more of the following: mobile devices such as mobile phones, smartphones, personal digital assistants (PDAs), tablets, or portable media players; navigation devices such as digital cameras, pocket video cameras, video game consoles, and Global Positioning System (GPS) devices; desktop or laptop computers; standalone devices such as sensors or smart meters; or any combination thereof.

[0041] NE 1010 may include at least one processor, designated 1011. Processor 1011 can be implemented by any computing or data processing device, such as a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a similar device. The processor may be implemented as a single controller or as multiple controllers or processors.

[0042] At least one memory, as indicated by 1012, is provided in one or more of the aforementioned devices. The memory may be fixed memory or removable memory. The memory may contain computer program instructions or computer code internally. Memory 1012 can independently be any suitable storage device, such as a non-temporary computer-readable medium. The term “non-temporary” as used herein corresponds to a limitation of the medium itself (i.e., tangible rather than signaling) in relation to data storage persistence (e.g., random access memory (RAM) versus read-only memory (ROM)). A hard disk drive (HDD), random access memory (RAM), flash memory, or other suitable memory may be used. The memory may be combined with one or more processors on a single integrated circuit as a processor, or it may be separate. Furthermore, computer program instructions stored in memory and processed by the processors may be any suitable form of computer program code, for example, a compiled or interpreted computer program written in any suitable programming language.

[0043] The processor 1011, memory 1012, and any subset thereof can be configured to provide means corresponding to various blocks in Figure 1. Although not shown, the device may also include positioning hardware, such as GPS or micro-electromechanical system (MEMS) hardware, used to determine the device's position. Other sensors, such as a barometer and compass, may also be housed and configured to determine position, altitude, speed, orientation, etc.

[0044] As shown in Figure 10, a transceiver 1013 may also be provided, and one or more devices may include at least one antenna, indicated as 1014. The device may have many antennas, such as an array of antennas configured for multiple input multiple output (MIMO) communication, or multiple antennas for multiple RATs (Radio Access Technology). Other configurations of these devices may also be provided, for example. The transceiver 1013 may be a transmitter, a receiver, both a transmitter and a receiver, or a unit or device configured for both transmission and reception.

[0045] Memory and computer program instructions, along with a processor for a particular device, can be configured to cause a hardware device such as a UE (User Equipment) to execute one of the processes described above (i.e., Figure 1). Therefore, in a particular exemplary embodiment, a non-temporary computer-readable medium, when executed in hardware, can be encoded using computer instructions that execute one of the processes described herein, etc. Alternatively, in a particular exemplary embodiment, the entire process may be executed in hardware.

[0046] In certain exemplary embodiments, the apparatus may include a circuit configured to perform any of the processes or functions shown in Figure 1. As used in this application, the term “circuit” means one or more or all of the following: (a) a hardware-only circuit implementation (such as an implementation of analog and / or digital circuits only), (b) a combination of hardware circuits and software, for example (where applicable), (i) a combination of analog and / or digital hardware circuits having software / firmware, and (ii) any part of a hardware processor (including a digital signal processor), software, and memory having software that works together to cause a device such as a mobile phone or server to perform various functions, and (c) a hardware circuit and / or processor such as a microprocessor or part of a microprocessor that requires software (e.g., firmware) for operation but may not be present when the software is not required for operation. This definition of circuit applies to all use of this term in this application, including any claim. As further examples, as used in this application, the term circuit also includes a hardware circuit or processor (or more processors), or a part of a hardware circuit or processor, and the implementation of its (or their) accompanying software and / or firmware. The term circuit also includes, for example, a baseband integrated circuit or processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device, where applicable to the components of a particular claim.

[0047] According to certain exemplary embodiments, the processor 1011 and memory 1012 may be included in or form part of a processing circuit or control circuit. In addition, in some exemplary embodiments, the transceiver 1013 may be included in or form part of a transceiver circuit.

[0048] In some exemplary embodiments, the apparatus (e.g., NE 1010) may include means for performing any of the methods, processes, or modifications described herein. Examples of such means may include one or more processors, memory, controllers, transmitters, receivers, and / or computer program code for performing the operation.

[0049] In various exemplary embodiments, the device 1010, controlled by memory 1012 and processor 1011, can extract at least one time-series characteristic from metric data, calculate at least one statistical measurement based on the at least one time-series characteristic, perform ensemble scoring based on the at least one statistical measurement, generate a predictability score based on the ensemble scoring, and determine the predictability of the metric data based on a comparison of the predictability score with a pre-set threshold.

[0050] Specific exemplary embodiments include, for example, an apparatus comprising means for performing any of the methods described herein, which include means for receiving metric data; means for extracting at least one time-series characteristic from the metric data; means for calculating at least one statistical measurement based on the at least one time-series characteristic; means for performing ensemble scoring based on the at least one statistical measurement; means for generating a predictability score based on the ensemble scoring; and means for determining the predictability of the metric data based on a comparison of the predictability score with a pre-set threshold.

[0051] The features, structures, or characteristics of the exemplary embodiments described herein can be combined in any suitable way in one or more exemplary embodiments. For example, the use of the phrases “various embodiments,” “specific embodiments,” “several embodiments,” or other similar terms throughout this specification refers to the fact that certain features, structures, or characteristics described in relation to the exemplary embodiments are included in at least one exemplary embodiment. Thus, the appearance of the phrases “various embodiments,” “specific embodiments,” “several embodiments,” or other similar terms throughout this specification does not necessarily refer to the same group of exemplary embodiments, and the features, structures, or characteristics described can be combined in any suitable way in one or more exemplary embodiments.

[0052] As used herein, “at least one of the following lists of two or more elements,” and “at least one of the lists of two or more elements,” and similar phrases in which lists of two or more elements are joined by “and” or “or,” mean at least one of the multiple elements, at least two or more of the multiple elements, or at least all of the elements.

[0053] Furthermore, the various functions or procedures described above can be performed in various orders and / or simultaneously with each other, as needed. Additionally, if necessary, one or more of the described functions or procedures can be arbitrarily selected or combined. Therefore, the above description should be considered as a description of the principles and teachings of specific exemplary embodiments, and not as an limitation thereof. Those skilled in the art will readily understand that the above-described exemplary embodiments can be implemented in a different sequence of steps and / or with hardware elements of a different configuration than those disclosed. Therefore, while some embodiments are described based on these exemplary embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative configurations are obvious, while remaining within the spirit and scope of the exemplary embodiments. (Explanation of some terms)

[0054] AIOp (Artificial Intelligence Operation) CPU: Central Processing Unit FCST Forecastability Score GPS (Global Positioning System) PDA (Personal Digital Assistant) ROM (Read-Only Memory) SE (Spectral Entropy)

Claims

1. The computing device receives metric data, The computing device performs the steps of extracting at least one time-series characteristic from the metric data, The computing device performs the steps of calculating at least one statistical measurement based on the at least one time-series characteristic, The computing device performs ensemble scoring based on the at least one statistical measurement; The computing device generates a predictability score based on the ensemble scoring, The computing device performs the steps of determining the predictability of the metric data based on a comparison of the predictability score with a pre-set threshold, A method characterized by including the following.

2. If the computing device determines that the metric data is unpredictable, the further step includes stopping the metric prediction. The method according to claim 1, characterized by the above.

3. If the computing device determines that the metric data is predictable, the computing device further includes the step of enabling metric prediction. The method according to claim 1, characterized by the above.

4. The computing device further includes the step of performing at least one periodic predictability check. The method according to claim 1, characterized by the above.

5. The aforementioned metric data, Out-percentage utilization rate, average load, The average percentage of memory used. Average response time, The sum of the average input / output operations per second, Average input / output read operation per second, Average input / output write operation per second, Average total latency, Average read latency, Average write latency, The percentage of disks used, and, Any other type of metric data, must include at least one of the following The method according to claim 1, characterized by the above.

6. The aforementioned at least one time series characteristic is tendency, seasonality, periodicity, Spikes, and, dip must include at least one of the following The method according to claim 1, characterized by the above.

7. The aforementioned at least one statistical measurement is, distributed, spectral density, Residual variability, and Omega score, must include at least one of the following The method according to claim 1, characterized by the above.

8. The step of calculating the at least one statistical measurement is: The computing device performs the steps of calculating the spectrum of at least one signal, The computing device performs the steps of calculating the power spectral density of at least one signal by squaring the amplitude of at least one signal, The computing device performs the steps of normalizing the at least one signal by a number of intervals, The steps include: converting the power spectral density calculated by the computing device into a probability density function; The computing device performs the steps of calculating the power spectral entropy according to a standard entropy calculation formula, The further step includes determining whether the normalized power spectral density represents a predictable series or an unpredictable, noisy series. The method according to claim 1, characterized by the above.

9. The computing device further includes the step of calculating the omega score by subtracting the power spectral entropy from 1, Predictability is demonstrated by calculating the aforementioned omega score as less than 1. The method according to claim 7, characterized by the following.

10. It is a device, At least one processor, Includes at least one memory for storing instructions, When the at least one memory and the instruction are executed by the at least one processor, the device has at least, To receive metric data, Extracting at least one time-series characteristic from the aforementioned metric data, Calculating at least one statistical measurement based on the at least one time series characteristic, Performing ensemble scoring based on at least one of the aforementioned statistical measurements, To generate a predictability score based on the aforementioned ensemble scoring, The predictability of the metric data is determined based on a comparison of the predictability score with a pre-set threshold. A device characterized by the following.

11. When the at least one memory and the instruction are executed by the at least one processor, the device has at least If it is determined that the aforementioned metric data is unpredictable, the system will stop predicting the metric. The apparatus according to claim 10, characterized in that

12. When the at least one memory and the instruction are executed by the at least one processor, the device has at least Once it is determined that the metric data is predictable, enable metric prediction. The apparatus according to claim 10, characterized in that

13. When the at least one memory and the instruction are executed by the at least one processor, the device has at least Perform at least one regular predictability check. The apparatus according to claim 10, characterized in that

14. The aforementioned metric data, Out-percentage utilization rate, average load, The average percentage of memory used. Average response time, The sum of the average input / output operations per second, Average input / output read operation per second, Average input / output write operation per second, Average total latency, Average read latency, Average write latency, The percentage of disks used, and, Any other type of metric data, must include at least one of the following The apparatus according to claim 10, characterized in that

15. The aforementioned at least one time series characteristic is tendency, seasonality, periodicity, Spikes, and, dip must include at least one of the following The apparatus according to claim 10, characterized in that

16. The aforementioned at least one statistical measurement is, distributed, spectral density, Residual variability, and Omega score, must include at least one of the following The apparatus according to claim 10, characterized in that

17. Calculating the aforementioned at least one statistical measurement means Calculate the spectrum of at least one signal, The power spectral density of at least one signal is calculated by squaring the amplitude of at least one of the signals, The above-mentioned at least one signal is normalized by a number of intervals, The calculated power spectral density is normalized to a probability density function, Calculating the power spectral entropy according to the standard entropy calculation formula, The further includes determining whether the normalized power spectral density represents a predictable series or an unpredictable, noisy series. The apparatus according to claim 10, characterized in that

18. When the at least one memory and the instruction are executed by the at least one processor, the device has at least The omega score is calculated by subtracting the power spectral entropy from 1. The Omega score is calculated as less than 1 to demonstrate predictability. The apparatus according to claim 16, characterized by the following:

19. A non-temporary computer-readable medium containing program instructions, When the program instruction is executed by the device, it causes the device to perform at least one method. The method is, Steps to receive metric data, A step of extracting at least one time-series characteristic from the aforementioned metric data, A step of calculating at least one statistical measurement based on the at least one time series characteristic, The steps include performing ensemble scoring based on at least one of the aforementioned statistical measurements, The steps include generating a predictability score based on the aforementioned ensemble scoring, The step of determining the predictability of the metric data based on a comparison of the predictability score with a pre-set threshold is included. A non-temporary computer-readable medium characterized by the following:

20. The aforementioned device is If it is determined that the aforementioned metric data is unpredictable, the process will be further modified to stop predicting the metric. A non-temporary computer-readable medium according to claim 19, characterized in that

Images