An incremental change-point detection method considering dependencies

Abstract

To provide a method and device for targeting change detection and analysis of power efficiency of a data center.SOLUTION: A method includes: collecting first data of a first period and second data of a second period related to energy consumption of a data center and related characteristics; generating a first machine trained model obtained by modeling a relation between the energy consumption and the related characteristics on the basis of the first data of the first period; identifying a change in the relation between the energy consumption and the related characteristics on the basis of either a first prediction error larger than a first value or the first prediction error smaller than a second value; displaying an indication of the identified change, and collecting third data of a third period on the basis of the identified change; and generating a second machine trained model on the basis of the third data.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present disclosure is generally directed to change detection and analysis of power efficiency in data centers. [Background technology]

[0002] Data centers (DCs) are heavy users of electricity, and continuous efforts are being made to improve their energy efficiency. One important aspect of energy efficiency is the reduction of the amount of cooling power used in a data center for a certain amount of equipment power used by information technology (IT) infrastructure or devices (e.g., storage and servers). This equipment power may be referred to as IT power in some aspects, to distinguish the power consumed by the infrastructure providing the data maintained in the DC (e.g., IT power) from the power consumed by the facility (e.g., DC) for other related functions, such as cooling power, lighting power, or other power consumed in maintaining the facility. A common metric to express this relationship is the power usage efficiency (PUE) metric, or its inverse, Data Center infrastructure efficiency (DCiE). The resulting metric value may change throughout the day and over the year, because system settings are updated, the required cooling power changes due to hotter / colder weather, or adjustments are made by the DC operator. However, not all changes lead to an improvement in the overall power efficiency of the DC.

[0003] Therefore, we provide a power efficiency change detection and analysis tool that can find changes in the power efficiency of DCs and changes in related factors, and provide feedback to DC operators regarding these changes. Summary of the Invention

[0004] Exemplary implementations described herein include an innovative method for detecting and analyzing changes in power efficiency in a DC. In some aspects, the method may identify changes in the power efficiency and related factors of the DC. The method may further, in some aspects, provide (e.g., display) feedback to the DC operator regarding the identified changes.

[0005] Aspects of the disclosure include a method for collecting first data for a first time period and second data for a second time period regarding energy usage and related characteristics of a data center. The method, in some aspects, further includes generating a first machine-trained model that models a relationship between the energy usage and the related characteristics based on the first data for the first time period. If a change in the relationship between the energy usage and the related characteristics is identified based on a difference between a first predicted energy usage for the second time period based on the first machine-trained model and a first actual energy usage for the second time period shown in the second data being one of greater than a first value or less than a second value, the method may include displaying an indication of the identified change, collecting third data for a third time period based on the identified change, and generating a second machine-trained model based on the third data.

[0006] Aspects of the disclosure include a non-transitory computer-readable medium storing instructions for execution by a processor, the instructions may include instructions for collecting first data for a first time period and second data for a second time period relating to energy usage and related characteristics of a data center. The non-transitory computer-readable medium may in some aspects further store instructions for generating a first machine-trained model that models a relationship between energy usage and the related characteristics based on the first data for the first time period. If a change in the relationship between energy usage and the related characteristics is identified based on a difference between a first predicted energy usage for the second time period based on the first machine-trained model and a first actual energy usage for the second time period shown in the second data being one of a first value or a second value, the non-transitory computer-readable medium may in some aspects further store instructions for displaying an indication of the identified change, and based on the identified change, collecting third data for a third time period and generating a second machine-trained model based on the third data.

[0007] Aspects of the present disclosure include a system that may include means for collecting first data for a first time period and second data for a second time period relating to energy usage and related characteristics of a data center. The system in some aspects further includes means for generating a first machine-trained model that models a relationship between the energy usage and the related characteristics based on the first data for the first time period. The system may include means for displaying an indication of the identified change if a change in the relationship between the energy usage and the related characteristics is identified based on a difference between a first predicted energy usage for the second time period based on the first machine-trained model and a first actual energy usage for the second time period shown in the second data being one of greater than a first value or less than a second value, and for collecting third data for a third time period and generating a second machine-trained model based on the third data based on the identified change.

[0008] Aspects of the disclosure include an apparatus that can include a processor configured to collect first data for a first time period and second data for a second time period related to energy usage and associated characteristics of a data center. The processor can in some aspects be further configured to generate a first machine-trained model that models a relationship between the energy usage and the associated characteristics based on the first data for the first time period. The processor can be further configured to display an indication of the identified change if a change in the relationship between the energy usage and the associated characteristics is identified based on a difference between a first predicted energy usage for the second time period based on the first machine-trained model and a first actual energy usage for the second time period depicted in the second data being one of a greater than a first value or a less than a second value, and to collect third data for a third time period and generate a second machine-trained model based on the third data based on the identified change. [Brief description of the drawings]

[0009] [Figure 1] 1 is a first graph of PUE as a function of OAT and a second graph of cooling power usage as a function of IT power usage in accordance with certain aspects of the disclosure. [Diagram 2] 1 is a set of diagrams showing a PUE index over time for two different periods that may lead to an erroneous identification of changes in the PUE index. [Diagram 3] 1 is a diagram showing the PUE indicator over time subject to changes in PUE. [Figure 4] 1 is a diagram illustrating a set of components of a system for implementing a change detection method according to some aspects of the present disclosure. [Diagram 5] 5 illustrates data (and associated data structures) that may be stored by one or more of the internal or external data storages of FIG. 4. [Figure 6] 1 is a flow diagram illustrating operations associated with incremental change point detection by a dependency modeling system according to some aspects of the present disclosure. [Figure 7] 1 is a flow diagram illustrating a set of operations related to data preparation and data processing according to some aspects of the present disclosure. [Figure 8] 1 is a diagram showing a set of windows at a set of steps (or times) for two approaches to training and testing window generation. [Figure 9] 1 is a diagram illustrating the use of a set of test windows (including a first validation window and a second change detection window) according to some embodiments of the present disclosure. [Figure 10] 1 is a diagram illustrating an example user interface for setting change detection operations and displaying results of change detection in accordance with some aspects of the present disclosure. [Figure 11] 1 is a flow diagram illustrating a method according to some aspects of the present disclosure. [Figure 12] 1 is a flow diagram illustrating a method according to some aspects of the present disclosure. [Figure 13] FIG. 1 illustrates an example computing environment including an example computing device suitable for use in some example implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The following detailed description provides details about the figures and exemplary implementations of the present application. For clarity, reference numbers and descriptions of overlapping elements between figures are omitted. The terms used throughout this description are provided by way of example and are not intended to be limiting. For example, the use of the term "automatic" may include a fully automatic or semi-automatic implementation that includes user or administrator control over certain aspects of the implementation, depending on the desired implementation of one of ordinary skill in the art practicing the implementation of the present application. The selection may be made by a user via a user interface or other input means, or may be implemented through a desired algorithm. The exemplary implementations described herein may be utilized alone or in combination, and the functionality of the exemplary implementations may be implemented via any means depending on the desired implementation.

[0011] In some aspects, a dependency between cooling power usage (or total non-IT power) and IT power usage in a DC may be modeled to identify significant changes in PUE. The dependency may include a dependency on outside air temperature (OAT) in some aspects, as well as other factors related to the operation of DC resources. FIG. 1 illustrates a first graph 110 of PUE as a function of OAT and a second graph 120 of cooling power usage as a function of IT power usage according to some aspects of the disclosure. The diagram 110 includes a first function 111 assuming no use of free cooling based on a low outside temperature. The diagram 110 also includes a second function 112 related to free cooling operation based on a low outside temperature. The diagram 120 includes a first function 122 of cooling power usage as a function of IT power usage assuming a fixed OAT (i.e., not considering the effect of OAT). If the impact of IT power on PUE is not correctly or properly considered, erroneous (or unnecessary) "changes" in the PUE metric may be identified in the time series of the PUE metric. Therefore, this disclosure presents a suitable ML model to model the relationship between cooling or total power consumption and IT power consumption in DC for change detection.

[0012] FIG. 2 is a set of diagrams 200 and 220 showing PUE metrics over time in two different time periods that may lead to erroneous identification of changes in PUE metrics. For example, if a change point is identified based on a single time series, i.e., PUE metric 202 or PUE metric 222 (solid line), the impact of OAT 204 or OAT 224 (dashed line) on PUE may not be properly considered. Diagram 200 shows a graph of PUE metric 202 over time and OAT 204 over time in the same time period. At time 210, PUE metric 202 may experience a significant change (e.g., a jump), which may be simply interpreted as a meaningful change in PUE metric 202 (or a set of factors contributing to PUE) that should be identified to the DC manager. However, at time 210, OAT 204 also experiences a similar significant change, which may be the cause of most or all of the change in PUE metric 202. Thus, an ML model may be provided that takes into account the correlation between OAT 204 and PUE metric 202 in order to correctly identify when a change in PUE metric 202 is due to a change in OAT 204 (and other known factors) or when a change in PUE metric 202 is unexplainable based on known factors (e.g., OAT 204).

[0013] Similarly, diagram 220 shows a graph of PUE index 222 over time and OAT 224 over time for the same period. At time 230 (or throughout the period), PUE index 222 may undergo significant changes (e.g., a change in trajectory, or a steady increase and / or decrease), which may be simply interpreted as a meaningful change in PUE index 222 (or a set of factors contributing to PUE) to be identified for the DC manager. However, at time 230 (or throughout the period), OAT 224 also undergoes a similar change, which may be the cause of most or all of the change in PUE index 222. Thus, a ML model may be provided that considers the correlation between OAT 224 and PUE index 222 to correctly identify when the change in PUE index 222 is due to a change in OAT 224 (and other known factors) or when the change in PUE index 222 is unexplainable based on known factors (e.g., OAT 224).

[0014] To find significant changes in the power efficiency of a DC, a model that considers the dependency of PUE on OAT may be trained. In some aspects, the model may approximate a linear function of OAT such that a first set of PUE values ​​around a first linear function of OAT may be considered to be derived from a distribution of first PUE values ​​(e.g., a set of PUE values ​​based on a first set of underlying conditions related to PUE). Similarly, a second set of PUE values ​​around a second linear function of OAT may be considered to be derived from a distribution of second PUE values ​​(e.g., a set of PUE values ​​based on a second set of underlying conditions related to PUE) based on the difference between the first and second linear functions of OAT, and the model may identify changes in the set of underlying conditions related to PUE and display and / or present them to a DC administrator. FIG. 3 includes a diagram 300 illustrating a PUE metric over time subject to changes in PUE (e.g., based on changes in underlying factors). FIG. 3 further includes a first diagram 310 and a second diagram 320 illustrating the relationship between PUE and OAT in different time periods. Diagram 310 illustrates that during a first time period (e.g., period A), PUE may be generally related to OAT with a first linear relationship (e.g., associated with trend line 312), and during a second time period (e.g., period B), PUE may be generally related to OAT with a second linear relationship (e.g., associated with trend line 311) that differs from the first linear relationship (e.g., differs by a relatively small gap). Similarly, diagram 320 illustrates that during a third time period (e.g., period C), PUE may be generally related to OAT with a third linear relationship (e.g., associated with trend line 323) that differs from the first linear relationship (e.g., associated with trend line 322, corresponding to trend line 312) and / or the second linear relationship (e.g., associated with trend line 321, corresponding to trend line 311) (e.g., differs by a relatively large gap).The model may identify a relatively large change between the first and third linear relationships depicted in diagram 320 (e.g., between trendlines 322 and 323) but may not identify a relatively small change between the first and second linear relationships depicted in diagram 310 (e.g., between trendlines 312 and 311) as important enough to present to a DC manager. In some aspects, the model may further identify a relatively small change between the first and second linear relationships depicted in diagram 310 (e.g., between trendlines 312 and 311) as important enough to present to a DC manager, depending on a threshold set for detecting a change in PUE.

[0015] 4 is a diagram 400 illustrating a set of components of a system for implementing a change detection method according to some aspects of the disclosure. At a first level of abstraction, the system may include a data preparation component or subsystem 410, a dependency modeling component or subsystem 420, and a change detection component or subsystem 430. The data preparation, dependency modeling, and change detection components or subsystems 410, 420, and 430 may interact with a model management database 440 and / or a display 450 (e.g., an output interface).

[0016] In some aspects, the data preparation component or subsystem 410 may be associated with a set of sensors, measurement devices, or other inputs that provide data for a data stream 411. The data stream 411 may be processed to store internal data (e.g., in internal data storage 412) and external data (e.g., in external data storage 413) for later retrieval. FIG. 5 illustrates data (e.g., tables) that may be stored by one or more of the internal data storage 412 or external data storage 413 of FIG. 4. For example, a power grouping information table 500 may store information about multiple aggregation levels, including a grouping type (or grouping level) ID 502, a grouping granularity 504, and an indication of the location of further information associated with that aggregation level (e.g., a pointer, index, or other identifier to other information tables 506 or data sets). For example, entry 510 indicates that the grouping type ID 502 is 2, the grouping granularity 504 is room level, and the associated information table 506 is A-2 (as shown in power grouping relationship table 540). Data management table 520 may store information regarding sensor (or source) ID 522, sensor type 524, grouping ID 526, and associated data table ID 528, which may be stored by data preparation component or subsystem 410. For example, a first entry in data management table 520 may reference a data table 560 that indicates power usage for a set of times at a particular granularity (e.g., the finest granularity that may later be aggregated to a different level of granularity).

[0017] The data preparation component or subsystem 410 may, in some aspects, include a data prep 414 that may prepare and / or group data at different aggregation levels for different features associated with the DC (e.g., hourly, daily, weekly, etc. aggregation based on a room or floor of the DC). The data preparation component or subsystem 410 may receive an indication of one or more of a spatial aggregation level, a spatial extent, or a temporal aggregation level. The data preparation component or subsystem 410 may, in some aspects, prepare data at the indicated spatial and temporal aggregation levels. In some aspects, preparing the data may include dividing the data into windows that are processed using an incremental approach (e.g., using a rolling window). The prepared data, in some aspects, may then be provided to a dependency modeling component or subsystem 420 for change detection operations performed by the change detection component or subsystem 430. Further details of data preparation are discussed below in conjunction with FIGS. 5-9.

[0018] The dependency modeling component or subsystem 420 may include a data correlation analysis component 421 and a dependency-based modeling component 423 in some aspects. The components of the dependency modeling component or subsystem 420 may use the data provided by the data preparation component or subsystem 410 to generate and / or train a model of PUE as a function of a set of factors in the provided data. The data provided by the data preparation component or subsystem 410 may be divided into time windows (e.g., “training windows” representing separate periods of equal length or overlapping and / or rolling time windows) for model training and / or generation in some aspects. The model training may include known machine learning algorithms and / or methods to train a machine-trained model of PUE as a function of one or more input factors (e.g., OAT, measured IT power, temperature set points of a DC cooling system, etc.). The machine-trained model may be a linear regression (LR) model that represents an approximate linear relationship between OAT and PUE in some aspects.

[0019] The data provided by the data preparation component or subsystem 410 to the dependency modeling component or subsystem 420 may also include a set of time windows for testing (e.g., "test windows") in some aspects. The test windows may include the most recent time window associated with the change detection operation (e.g., a change detection window) and a second most recent time window used to determine parameters related to the change detection operation (e.g., a validation window). The test windows may be a fixed number of the most recent time windows in some aspects, which may then be used to train and / or generate models as new data is collected (e.g., become the training window). In some aspects, the model is updated each time a new time window becomes available for training until a change is detected. If a change is detected, the model may be retrained (e.g., a new machine-trained model may be generated rather than updating the current machine-trained model) after a sufficient amount of time has passed (e.g., one day, seven days, one month, etc.) to collect enough training data to train a new model of the changed system. In some aspects, at least one recent time window (e.g., a test window) may be used with the trained model to determine a set of control limits to be applied by the change detection component or subsystem 430. In some aspects, the amount of training data deemed sufficient to train a new model may be reduced by using difference data (e.g., data generated by calculating the difference between a set of input source data and a target data) instead of raw value data. The input source data, in some aspects, may be represented as a data feature vector associated with step (or time) t:

number

number

number

[0020] The dependency modeling component or subsystem 420 may, in some aspects, provide and / or store the machine trained model in a model management database 440. The stored machine trained model may then be used by a change detection component or subsystem 430 to detect changes in the relationship between the PUE and the set of factors considered by the machine trained model. In some aspects, the change detection component or subsystem 430 may include a control limit generator 431 that generates thresholds used to detect changes based on a recent test window (e.g., validation window). The thresholds may include upper and lower control limits based, for example, on a moving average (exponentially weighted moving average (EWMA) or other average value) and standard deviation (exponentially weighted moving standard deviation (EWMSTD) or other standard deviation measure) of the error between the PUE recorded and / or calculated during the recent test window (e.g., validation window) and the predicted PUE generated by the machine trained model based on the (input) data associated with the test window (e.g., change detection window). In some embodiments, the upper control limit (UCL) may be calculated as UCL=EWMA+k*EWMSTD and the lower control limit (LCL) may be calculated as LCL=EWMA-k*EWMSTD, where k may be a fixed value determined by an administrator based on the desired sensitivity of the associated change detection operation.

[0021] In some aspects, a threshold may be provided to the change point detector 432 used to detect a change in the underlying relationship between the PUE and the set of factors considered by the machine-trained model (e.g., to perform a change detection operation). For example, the change point detector 432 may determine whether the recorded PUE is outside a region defined by a threshold (e.g., UCL and LCL) a threshold number of times. The threshold number of times, in some aspects, may be determined by an administrator based on a desired sensitivity of the change detection operation (and may depend on the value of k selected by the administrator).

[0022] Upon detecting a change, the change point detector 432 may provide an indication to the model updater 433. The model updater 433 may indicate to the control limit generator 431 that a change has been detected in order to adjust the control limit generation (e.g., pause the control limit generation until an updated model is generated). The change detection component or subsystem 430 may further indicate to the data preparation component or subsystem 410 and / or the dependency modeling component or subsystem 420 to initiate a new model training or update operation. The change detection component or subsystem 430 may further output an indication of the detected change to an administrator via the display 450 or other output interface.

[0023] 6 is a flow diagram 600 illustrating operations associated with incremental change point detection by a dependency modeling system according to some aspects of the disclosure. The set of data preparation operations 610 may include a set of data management operations at 612 to store the incoming data in one or more data structures (e.g., databases or other data structures). The one or more data structures may include different data structures for different types of data, such as IT power consumption, OAT, or other factors at the lowest aggregation level of various components of the DC (which may be aggregated to a desired aggregation level).

[0024] The set of data preparation operations 610 may further include a data aggregation operation at 614. The data aggregation operation at 614 may, in some aspects, include a data aggregation operation to generate data at a desired spatial and temporal granularity level. The data aggregation operation at 614 may, in some aspects, be based on a granularity selected by an administrator via a user interface (e.g., the user interface shown in FIG. 10). In some aspects, the data aggregation operation at 614 may include a set of training and test data processing operations at 616. A set of training and test data processing operations may be performed at 616, which may include splitting the data generated by the data aggregation operation at 614 into training and test windows.

[0025] After the set of data preparation operations 610, a set of dependency modeling operations 620 may be performed. The set of dependency modeling operations 620 may include a determination at 622 of whether a model should be updated and / or generated. In some aspects, the determination at 622 may be based on whether a model has been previously generated (e.g., whether a model at the currently selected level of granularity has been generated), or based on whether a change has been detected and sufficient time has passed since the detected change to collect sufficient data. The determination at 622 may be based on whether a period of time (or a configured update frequency) associated with a training window and / or a testing window has passed such that a further training window is available to update the previously generated model. If it is determined not to update the model at 622, the system may proceed to a set of change detection operations 630, described below.

[0026] If it is determined at 622 that a model should be updated and / or generated, the dependency modeling operations 620 may include feature correlation analysis at 624. The feature correlation analysis at 624 may, in some aspects, include calculating correlations between data collected for different components and / or factors. The feature correlation analysis at 624 may, in some aspects, also include identifying and / or selecting data features based on the model to be trained. For example, the feature correlation analysis at 624 may identify a set of inputs, e.g., a type of input (e.g., OAT, time of day, server demand, or other data), or a granularity of inputs (e.g., hourly data, daily data, weekly data, or other granularity of data), and a set of outputs, e.g., a type of output (e.g., total energy used, IT power, PUE, DCiE, or other data), for a particular requested model. The calculated correlations may be used to train and / or generate a predictive model at 626 based on a set of training data (e.g., a training window). The calculation of correlations (e.g., 624) may, in some aspects, be part of (or included in) model training at 626. For example, using one or more of linear regression and / or other machine learning algorithms or operations, the model training may learn and / or identify correlations between the metric of interest (e.g., PUE or DCiE) and one or more factors (e.g., OAT, IT power, temperature control set points, or other factors) in the training data. The learned and / or identified correlations may then be incorporated and / or reflected in the trained predictive model (e.g., may be related to a set of weights or coefficients of a linear regression model for one or more nodes of the machine trained network). Once trained, the model may be stored in a model management database at 628. The system, in some aspects, may then perform a set of model management operations at 640 to manage multiple models that may be generated at different aggregation levels and / or for different components of the DC (e.g., areas, rooms, or floors, etc.).Model management at 640 may, in some aspects, include providing desired and / or appropriate models for the set of change detection operations 630.

[0027] In some aspects, the set of change detection operations 630 may begin at 631 by predicting a PUE (or other metric of interest) based on input data collected during a change detection period (e.g., test data collected during or associated with a change detection window in the set of test windows) based on the model generated by the set of dependency modeling operations 620 (and provided by the model management operations at 640). The predicted PUE may then be compared at 632 to a measured PUE to determine a prediction error. To determine whether the PUE is consistent with the current model, the system may in some aspects update a set of control limits (e.g., as described above in connection with the control limit generator 431 of FIG. 4 ) based on validation data (e.g., data associated with a validation window that is a previous change detection window in the set of test windows) at 633. In some aspects, the control limits may be updated prior to or in parallel with determining the prediction error.

[0028] Based on the generated set of control limits (e.g., the control limits updated at 633), the system, in some embodiments, may count the number of control limit violations at 634 (e.g., count the number of times the error exceeds the UCL or falls below the LCL). As discussed above in connection with FIG. 4, the UCL and LCL may be set based on the desired sensitivity of the set of change detection operations 630. The system, in some embodiments, may then determine at 635 whether the count generated at 634 exceeds a threshold number n set by an administrator. The number n, in some embodiments, may be set along with the control limits to determine the sensitivity of the change point detection operations. In some embodiments, the number n may be based on the total number of data points and / or samples within a test window (e.g., may be set as a percentage of the data points within the window).

[0029] If the system determines at 635 that the counts generated at 634 do not exceed the limit, the system may output at 636 an indication that a change was not detected and may return to the set of training and test data processing operations at 616 to generate another set of training and / or test windows (e.g., change detection and / or validation windows). The set of windows may be used to update the current model or to generate new control limits for a subsequent set of change detection operations 630 associated with the current test period and / or test window. If the system determines at 635 that the counts generated at 634 exceed the limit, the system may output at 637 an indication that a change was detected and may return to the set of training and test data processing operations at 616 to generate another set of training and / or test windows (e.g., change detection and / or validation windows). In some aspects, the set of training and test data processing operations at 616 may include generating at least a minimum number of new training windows (e.g., based on data collected after the detected change) to train a new model based on the indicated change. The set of windows may be used to update a current model, generate a new model based on the detected change, and / or generate new control limits for a subsequent set of change detection operations 630 associated with the current test period and / or test window. The indication of the detected change output at 637 may be considered at 622 in some aspects to determine that a model should be updated and / or generated. These operations may be performed for a fixed number of "loops", for a set amount of time, or until an input is received indicating the stop of the process. These operations may be performed in parallel for different aggregation levels or different areas and / or components of the DC.

[0030] FIG. 7 is a flow diagram 700 illustrating a set of operations related to data preparation and data processing according to some aspects of the disclosure. The set of operations for data preparation may be a set of data preparation operations 710 associated with a data preparer and a set of training and test data processing operations 720. The flow diagram 700 may be preceded by receiving an indication of a desired spatial and / or temporal granularity (and / or range) received via an input interface from an administrator of the DC. This set of data manipulations may begin at 712 by grouping the power data according to a user-selected grouping granularity (e.g., an indicated spatial granularity and / or range) based on grouping information available from a data management database. For example, with reference to FIGS. 4 and 5, the data preparation component or subsystem 410 may retrieve data from one or more of the internal data storage 412 and the external data storage 413 (e.g., from a data table 560 identified using the data management table 520) based on the power groupings identified in the information table 500 and / or the power grouping relationship table 540. Grouping may be aggregated at the DC-wide, building, floor, room group, room, or rack level in some aspects. Aggregation may be performed for a selected level of granularity rather than for a particular unit. For example, if floor-level granularity is selected, aggregation may be performed at 712 (1) for a particular floor, (2) for each of a set of floors in one or more buildings, or (3) for each floor in the DC.

[0031] At 714, the data preparer may resample the grouped data (data aggregated based on a user-selected aggregation level) based on a user-selected sampling frequency. For example, the stored data points may be those associated with a first highest frequency (e.g., every second, every minute, every hour, or other frequency sufficient to identify changes at a desired characteristic time or granularity). The first highest frequency may be set by an administrator in some aspects based on a shortest period of interest (e.g., based on a shortest time associated with a change that may be significant). For example, changes in the relationship between PUE and IT power over a time span of less than one day (or more in some aspects) may be based on transient factors that an administrator may not want to address. Thus, in some aspects, to minimize data storage size, composite (aggregated), summed, and / or average values ​​for the grouped data may be stored for each period that represents the smallest time useful for identifying changes at a meaningful and / or significant aggregation level (e.g., week, day, or hour). Sampling may aggregate and / or average the most frequently stored data (eg, grouped data) to produce less frequent data selected by the user for the aggregation level.

[0032] At 716, the data preparer may assign the resampled and grouped data to one or more windows. In some aspects, assigning the resampled and grouped data at 716 may include generating one or more windows based on a user-selected spatial and temporal granularity and / or range. The one or more windows may have the same temporal range, for example, covering a one-week or one-month time span, going back from the present, a set number of windows. For example, the past six months (or six weeks) of resampled and grouped data may in some aspects be divided into six one-month (or one-week) windows for training and testing. The set number of windows may in some aspects be a variable number based on the time since the last detected change, a minimum number of windows for accurate training of the model, and / or a maximum number of windows to conserve processing power or avoid overtraining the model.

[0033] After assigning the resampled and grouped data to windows in 716 (e.g., generating the windows), one or more windows may, in some aspects, be designated and / or identified as one of a training window and / or a test window (e.g., a change detection window and / or a validation window) in 722. In some aspects, the designation and / or identification in 722 may be an updated designation and / or identification of a training window or a test window based on the current time and / or index. FIG. 8 is a diagram 800 illustrating a set of windows in a set of steps (or times) for two approaches of window generation for training and testing. In some aspects, operations 712-722 may be performed during each step (or time) to generate (and / or update the designation as) a training window and a test window for the current index (or time). Operations 712-722 may be omitted for windows already designated and / or identified as training windows (e.g., if the maximum number of training windows has not been reached). In each approach, a first step (or time) associated with index (or time) t may be associated with a set number of training and test windows. As described in connection with Figure 6, the generated training windows may be used for a set of dependency modeling operations 620 and a set of change detection operations 630, at which point further data may be generated according to a set of data preparation operations 610 and / or operations 712-722.

[0034] For subsequent steps (or times), e.g., steps (or times) t+s or t+2s, in some aspects, one of the first sliding window approach 810 or the second sliding window approach 820 may be used. In either the first sliding window approach 810 or the second sliding window approach 820, the training window may be used to train an LR model or other machine-trained (MT) model. The training may be based in some aspects on direct data modeling (e.g., using measured raw data) or differential data modeling (e.g., using data generated by taking the difference between raw data of at least the first training window and data of a reference time window). In some aspects, the reference time window may be any one of a fixed (representative) time window, or a dynamic time window, e.g., the immediately preceding time window. In each of the first sliding window approach 810 and the second sliding window approach 820, a first set of windows (e.g., window 811, window 813, window 815, window 821, window 823, and window 825) may be designated as training windows, and a second subsequent set of windows (e.g., window 812, window 814, window 816, window 822, window 824, and window 826) may be designated as test windows (e.g., change detection and / or validation windows). In the first sliding window approach 810, the window at each step (or time) may be shifted by a set time s, which may be less than the length n of the window, so that at least operations 716 and 722 are performed to update the training windows 813 and / or 815 and the test / validation windows 814 and / or 816. For example, a window of one week (eg, n equals 7 days) may be used, and a new window may be generated each day (eg, s equals 1 day) for change detection.In some aspects (as shown for the set of training windows 821, 823, and 825), s may be set equal to n, such that the same training window may be reused between steps (or time instants) and new windows are generated for the test window based on data captured since the last step (or time instant).

[0035] As shown for both the first sliding window approach 810 and the second sliding window approach 820, after a detected change, there may be a minimum number of training windows (and associated minimum time elapsed) before the system can be used to train a new model and use the model to detect changes. For example, the system may be configured to train a model using at least four windows (e.g., window 811 or 821) and capture data for validation and / or change detection using at least two windows (e.g., windows 812 and 822). In some aspects, the first sliding window approach 810 may use a fixed number of most recent training windows, while the second sliding window approach 820 may use an increasing number of most recent training windows. The number of most recent training windows used in the second sliding window approach 820 may be constrained by a minimum number of training windows for model training accuracy and / or a maximum number of training windows to conserve processing power and / or avoid overtraining the model. In both approaches, data associated with a test window in a first step (or time) may be associated with a training window in a subsequent step (or time).

[0036] In some embodiments, the training window may be used to update an existing model. For example, the training window may be used to update an LR or other MT model. After a first step (or time) associated with initial model training, in some embodiments, the LR or MT model may be updated by retraining the LR or MT model based on the current set of training windows (in either the first sliding window approach 810 or the second sliding window approach 820). In some embodiments, the LR or MT model trained during the first step (or time) may be updated (e.g., modified without full retraining) based on data associated with a new training window (e.g., a window previously designated and / or identified as a test window) at each subsequent step (or time) until a change is detected.

[0037] 9 is a diagram 900 illustrating the use of a set of test windows (including a first validation window and a second change detection window) according to some aspects of the disclosure. Diagram 900 shows that a set of time windows associated with step (or time) t may include a first set of training windows 911 and a second set of test windows 912. The second set of test windows may, in some aspects, include a first validation window 912a and a second change detection window 912b. As discussed in connection with FIG. 8, the validation window 912a at the first time t may be used as a training window at a later time t+s.

[0038] Diagram 920 illustrates a first prediction error 922 calculated based on data captured during a training window and a second prediction error 926 calculated based on data captured during a test window (one of a validation window or a change detection window). The prediction error associated with a particular window (one of the first or second prediction errors 922 and 926) may, in some aspects, be calculated by generating a predicted PUE using a trained model based on a set of input data associated with the particular window (e.g., data used as input for the associated model) and subtracting the predicted PUE from the corresponding measured PUE during the particular window (e.g., PUE measured -PUE predicted ) (or vice versa). As shown, the prediction error 922 associated with the training window may be (or may be expected to be) smaller (e.g., have a smaller mean value) or less variable in some aspects than the prediction error 926 associated with the test window. In some aspects, this difference in prediction errors may be the result of a model being trained to fit data associated with the training data.

[0039] To detect the change, a set of upper and lower control limits may be defined to identify a change in the relationship between the cooling or total power consumption and the IT power consumption in the DC. If the UCL and LCL are set based on the average prediction error and the prediction error variance associated with the training window, in some embodiments, the UCL and LCL may define an overly restricted area and the system may produce false positives. Thus, in some embodiments, the UCL and LCL are defined based on the average prediction error and / or the prediction error variance associated with the test window and / or the validation window (e.g., a window not used to train the model) discussed in connection with FIG. 4. Diagram 950 illustrates UCL 955 and LCL 957 determined based on data associated with validation window 951 and used to detect a change during change detection window 953. Diagram 950 further illustrates change detection window 953 including the detected change. In the example illustrated in diagram 950, the change results in a jump in the prediction error between the first time t1 and the second time t2, resulting in the prediction error exceeding UCL 955. In some aspects, detecting the change further includes detecting that the change lasts for a threshold time or is associated with a threshold number of data points (e.g., a calculated prediction error) within a window to avoid updating a model based on a transient phenomenon.

[0040] FIG. 10 is a diagram 1000 illustrating an example user interface for setting change detection operations and displaying change detection results, according to some aspects of the disclosure. The example user interface of diagram 1000 includes a data preparation and change detection parameter setting area 1010 and a display area 1030. The data preparation parameter setting area 1010 may include a set of input elements for defining spatial granularity and / or range and temporal granularity and window length associated with the data preparation and change detection operations. The set of input elements may include a first input element 1011 for specifying and / or setting an index of interest (e.g., PUE, DCiE, etc.). In some aspects, a second input element 1012 and a third input element 1013 may be included in the set of input elements for specifying and / or setting a granularity (or aggregation) level and a spatial range, respectively. The set of input elements may further include a fourth input element 1014 and a fifth input element 1015, which are used in some aspects to specify and / or set a sampling frequency and a window length for the associated data preparation and change detection operations. In some aspects, the set of input elements may include a sixth input element 1016 for specifying and / or setting a change detection sensitivity for the change detection operation. The set of input elements may include the set of drop-down menus shown in diagram 1000, or other types of input elements (e.g., text entry areas, radio buttons, etc.).

[0041] Display area 1030 may, in some aspects, display data associated with the set of test windows. This data may, in some aspects, include one of PUE (or other indicated metric of interest) over time or a prediction error associated with PUE (or other indicated metric of interest). Display area 1030 may, in some aspects, include a change detection indication 1035 indicating that a change has been detected. In some aspects, the display of the data may further include an indication of the UCL and / or LCL (as shown in FIG. 9). In some aspects, display area 1030 may further include a recommendation to address the detected change 1037 and / or an indication of its predicted cause.

[0042] FIG. 11 is a flow diagram 1100 illustrating a method according to some aspects of the disclosure. In some aspects, the method is performed by a system or apparatus (e.g., a system described in connection with FIG. 4 or an apparatus described below in connection with FIG. 13 below). At 1102, the system may collect first data for a first time period and second data for a second time period related to energy usage and related characteristics of a data center. In some aspects, the first time period may include one or more training windows and the second time period may include a set of one or more test windows (e.g., validation windows and / or change detection windows). In some aspects, collecting data at 1102 may include collecting fourth data for a fourth time period following the first time period and preceding the second time period (e.g., collecting data for a validation window associated with a change detection window associated with the second time period). The related characteristics may in some aspects include at least OAT and any other external data deemed important to the metric of interest (e.g., PUE, DCiE, or other metric monitored by the administrator). In some aspects, the data regarding energy usage may include data regarding at least first energy usage data associated with a first power consumed by equipment providing IT functions in the data center (e.g., IT power) and second energy usage data associated with a second power consumed by the data center (e.g., total power consumption).

[0043] In some aspects, the first data and / or the second data may include energy usage data at one or more spatial and / or temporal granularity levels. The one or more levels may include a first highest granularity level representing the smallest spatial and / or temporal unit of interest, which may be aggregated to generate further (lower) granularity levels. For example, with reference to FIG. 5, a data structure (e.g., data table 560) may store energy usage data (e.g., server power) at hourly granularity, which may be aggregated to generate data at different (lower) granularity levels, such as, for example, daily, weekly, or monthly. In some aspects, the first data and / or the second data may include energy usage data at multiple spatial and / or temporal granularity levels. The multiple spatial granularity levels, in some aspects, may include one or more of IT device-level granularity (representing the highest level of granularity), rack-level granularity, rack group-level granularity, room-level granularity, room group-level granularity, floor-level granularity, building-level granularity, or data center-level granularity. For example, with reference to FIG. 5, data associated with different spatial granularity levels may be stored in a set of data structures (e.g., information table 500 and / or power grouping relationship table 540). The multiple temporal granularity levels, in some aspects, may include granularity levels at one or more of seconds, minutes, hours, days, weeks, months, quarters, or years. For example, with reference to FIG. 5, the system may include multiple data structures at different granularity levels (data table 560 is an example of one such table associated with an hourly granularity level).

[0044] At 1104, the system may, in some aspects, generate a machine-trained model that models a relationship between energy usage and associated characteristics based on the first data for the first time period. In some aspects, the relationship between energy usage and associated characteristics includes a specific relationship between the second (total consumption) power, the first (IT) power, and the associated characteristics (e.g., OAT or other input). The specific relationship between the second power, the first power, and the associated characteristics may, in some aspects, include a function for calculating a PUE based on the first power and the associated characteristics. In some aspects, the PUE may be calculated by dividing the second power by the first power. For example, with reference to FIGS. 4, 6, 9, and 10, the dependency modeling component or subsystem 420 may generate a machine-trained model (e.g., an LR model) based on the set of dependency modeling operations 620 and a set of parameters received from an administrator (e.g., via input elements 1011-1015).

[0045] At 1106, the system may identify a change in the relationship between the energy usage and the associated characteristic. The identification at 1106 may in some embodiments be based on a first prediction error associated with the second time period reflecting (e.g., being a measure of or being a difference between) a first predicted energy usage for the second time period based on the machine-trained model and a first actual energy usage for the second time period as indicated in the second data being either greater than a first value or less than a second value. In some embodiments, the first value and / or the second value may be determined based on fourth data collected during a fourth time period following the first time period and preceding the second time period. For example, the system may determine, as part of the identification at 1106, a mean and standard deviation of a second prediction error for the fourth time period based on the second predicted energy usage (or metric of interest, such as PUE or DCiE) for the fourth time period predicted by the machine-trained model and the second actual energy usage (or metric of interest, such as PUE or DCiE) for the fourth time period indicated in the fourth data. In some aspects, the second prediction error may be a difference between the second predicted energy usage and the second actual energy usage. The first value and the second value may be based on a mean of the second prediction error and a standard deviation of the second prediction error in some aspects. In some aspects, the mean of the second prediction error may be EWMA and the standard deviation of the second prediction error may be EWMSTD, as described above in connection with FIG. 4, and the first value may be EWMA plus the product of EWMSTD and a scaling factor, and the second value is EWMA minus the product of EWMSTD and a scaling factor. For example, with reference to Figures 4, 6, 9, and 10, the change detection component or subsystem 430 (including the control limit generator 431, the change point detector 432, and / or the model updater 433) may identify changes in relationships based on a set of change detection operations 630 that are based on a set of control limits (e.g., UCL 955 and LCL 957) generated based on sensitivities set by an administrator (e.g., via input element 1016).

[0046] Based on identifying the change at 1106, the system may display an indication of the identified change at 1108. The display at 1108 may, in some aspects, further include an indication of one or more of a determined probable cause of the change or a recommendation for remediation if the change is related to a decrease in energy efficiency in the DC. For example, referring to FIG. 10, based on identifying the detected change, the UI may display an indication of the detected change 1035 and an indication of a recommendation for remediating the detected change and / or its predicted cause 1037 in the display area 1030.

[0047] At 1110, the system may collect third data for a third time period based on the identified change. In some aspects, the third data and / or the third time period may be associated with a set of training windows after the change is identified. The set of training windows may be a minimum number of training windows over a minimum time period defined by an administrator (e.g., to achieve a desired level of accuracy of the machine trained model based on the collected data). For example, with reference to FIGS. 4 and 6, the data preparation component or subsystem 410 may receive an indication (e.g., an indication output at 637) from the change detection component or subsystem 430 that a change has been identified and / or detected, and may collect and / or identify updated data for a new set of training windows.

[0048] After collecting the third data, the system may, in some aspects, generate a new (second) machine-trained model based on the third data at 1112. The generation of the new (second) machine-trained model may be based on data collected after the identified and / or detected change to avoid using data related to a previous relationship between energy usage and related characteristics. In some aspects, the second data including the identified change may be used to update and / or generate the second machine-trained model along with the third data. For example, with reference to FIG. 4 and FIG. 6, the dependency modeling component or subsystem 420 may update the model with a set of dependency modeling operations 620 based on data collected for a new set of training windows. The system may continue to monitor the PUE based on one or more of the set of parameters (e.g., granularity and range parameters) received from the administrator until it receives a stop command, receives an updated set of parameters, or reaches a set number of periods from the start of the monitoring and / or change detection operations.

[0049] In some embodiments, the fourth time period may be a previously tested time period (e.g., a change detection window associated with a particular step preceding the step of using the change detection window associated with the second time period). The system may, in some embodiments, identify the absence and / or absence of a change in the relationship between the energy usage and the associated characteristic that exceeds a threshold. This identification may, in some embodiments, be based on the second prediction error (e.g., the difference between the fourth predicted energy usage for the fourth time period based on the machine trained model and the fourth actual energy usage for the fourth time period as indicated in the fourth data) being within a range between a third value and a fourth value. The third and fourth values ​​may, in some embodiments, be based on data collected in a fifth time period preceding the fourth time period (e.g., a validation window associated with the change detection operation for the fourth time period) because the data collected in the fourth time period was used to generate the first and second values ​​for identifying the change in the second time period in 1106. In some aspects, a set of thresholds (e.g., UCL and LCL values) for a current time window may be calculated based on multiple previous time windows (e.g., validation or test windows). EWMA and EWMSTD may provide a recency bias such that more recent time periods are given more weight (e.g., based on the assumption that more recent time periods are more relevant). In some aspects, the machine trained model generated in 1104 may be initially generated based on the first data and updated based on the fifth data (or may be generated with an updated set of training data including the first data and the fifth data).

[0050] FIG. 12 is a flow diagram 1200 illustrating a method according to some aspects of the disclosure. In some aspects, the method is performed by a system or device (e.g., a system described in connection with FIG. 4 or a device described below in connection with FIG. 13 below). The following operations and / or steps may be discussed, for purposes of explanation, in connection with an example scenario in which a first (training) period is followed by a second (change detection) period in which changes are detected based on a model trained using data from at least the first period. In some aspects, the second period may be followed by a third (training) period for training a new and / or updated model based on the changes detected in the second period. Also, the second period may be preceded by a fourth period in which no changes were detected, which is used as a validation period (or window) associated with the second period. A fifth time period, which follows the first time period and precedes the fourth time period, may in some aspects be used as a validation period associated with the fourth time period and as part of a training period associated with the second time period (e.g., a model trained using data from the first time period may be updated based on data from the fifth time period before being used to predict an indicator of interest for the second time period).

[0051] At 1202, the system may receive a selection of a first temporal granularity level and a second spatial granularity level for the change detection operation. In some aspects, the system may further receive a spatial scope of the change detection operation at 1202. The temporal granularity level may be selected in some aspects from one or more of granularity levels in seconds, minutes, hours, days, weeks, months, quarters, or years. Similarly, the spatial granularity level may be selected in some aspects from one or more of IT device-level granularity (representing the highest level of granularity), rack-level granularity, rack group-level granularity, room-level granularity, room group-level granularity, floor-level granularity, building-level granularity, or data center-level granularity. The spatial scope of the analysis may similarly be selected to identify the entire DC or one or more specific IT devices, racks, groups of racks, rooms, groups of rooms, floors, or buildings of the DC. For example, referring to FIG. 10, the system may receive a selection of parameters via one or more input elements 1011-1015.

[0052] At 1204, the system may collect and / or prepare training data for a training period (e.g., first data for a first time period or third data for a third time period) regarding energy usage and related characteristics of the data center. In some aspects, the training data may include one or more training windows. The related characteristics may in some aspects include at least OAT and any other external data deemed important to the metric of interest (e.g., PUE, DCiE, or other metric monitored by an administrator). In some aspects, the data regarding energy usage may include data related at least to first energy usage data associated with a first power consumed by equipment providing IT functions in the data center (e.g., IT power) and second energy usage data associated with a second power consumed by the data center (e.g., total power consumption). For example, with reference to Figures 4-10, data preparation component or subsystem 410 (or data prepper 414) may prepare training windows 811, 813, 815, 821, 823, 825, or 911 from information stored in a data structure such as that shown in Figure 5 using operations discussed in connection with one or more of sets of data preparation operations 610 or 710 according to spatial and temporal granularity input via one or more input elements, such as input elements 1011-1015 of Figure 10.

[0053] In some aspects, the training data may include energy usage data at one or more spatial and / or temporal granularity levels. The one or more levels may include a first highest granularity level representing the smallest spatial and / or temporal unit of interest, which may be aggregated to generate additional (lower) granularity levels. For example, with reference to FIG. 5, a data structure (e.g., data table 560) may store energy usage data (e.g., server power) at hourly granularity, which may be aggregated to generate data at different (lower) granularity levels, such as, for example, daily, weekly, or monthly. In some aspects, the training data may include energy usage data at multiple spatial and / or temporal granularity levels. The multiple spatial granularity levels may include, in some aspects, one or more of IT device-level granularity (representing the highest level of granularity), rack-level granularity, rack group-level granularity, room-level granularity, room group-level granularity, floor-level granularity, building-level granularity, or data center-level granularity. 5, data associated with different spatial granularity levels may be stored in a set of data structures (e.g., information table 500 and / or power grouping relationship table 540). The multiple temporal granularity levels may, in some aspects, include granularity levels at one or more of seconds, minutes, hours, days, weeks, months, quarters, or years. For example, with reference to FIG. 5, the system may include multiple data structures at different granularity levels (data table 560 is an example of one such table associated with an hourly granularity level).

[0054] At 1206, the system may generate (or update) a machine-trained model that models a relationship between energy usage and related characteristics based on the training data (e.g., the first data for the first time period, the updated training data including the first data and the fifth data for the fifth time period, or the third data for the third time period). In some aspects, the relationship between energy usage and related characteristics includes a specific relationship between the second (total consumption) power, the first (IT) power, and the related characteristics (e.g., OAT or other input). The specific relationship between the second power, the first power, and the related characteristics may include a function for calculating a PUE (or DCiE) based on the first power and the related characteristics in some aspects. In some aspects, the PUE may be calculated by dividing the second power by the first power. For example, with reference to Figures 4, 6, 9, and 10, the dependency modeling component or subsystem 420 may generate a machine-trained model (e.g., an LR model) based on the set of dependency modeling operations 620 and a set of parameters received from an administrator (e.g., via input elements 1011-1015).

[0055] At 1208, the system may, in some embodiments, collect, prepare, and / or identify validation data for a validation period. In some embodiments, when performing change detection for a second period, the validation data may be the fourth data for a fourth period. In some embodiments, when performing change detection for a fourth period, the validation may be the fifth data. In some embodiments, data may be collected for a recently initiated set of change detection operations (or after identifying a change in the relationship between energy usage and the associated characteristic). Data for an ongoing (or newly initiated) set of change detection operations may be prepared and / or identified from previously collected data stored in one or more data structures discussed in connection with FIG. 4 and FIG. 5. The validation data may, in some embodiments, be of the same spatial and temporal granularity level (as well as the same spatial extent) as the training data. For example, with reference to Figures 4-10, the data preparation component or subsystem 410 (or data preparer 414) may prepare validation data associated with validation window 912a (or validation data associated with test windows 812, 814, 816, 822, 824, or 826) from information stored in a data structure such as that shown in Figure 5 using operations discussed in connection with one or more of the sets of data preparation operations 610 or 710 according to spatial and temporal granularity input via one or more input elements, such as input elements 1011-1015 of Figure 10.

[0056] At 1210, the system may determine a mean and standard deviation of a second prediction error for the fourth time period based on the second predicted energy usage (or other metric of interest, such as PUE or DCiE) for the fourth time period predicted by the machine-trained model and the second actual energy usage for the fourth time period indicated in the fourth data. The first control limit (e.g., UCL) and the second control limit (e.g., LCL) may be based on the mean and standard deviation of the second prediction error in some aspects. In some aspects, the mean of the second prediction error may be EWMA and the standard deviation of the second prediction error may be EWMSTD, as described above in connection with FIG. 4, where the first value may be EWMA plus the product of EWMSTD and a scaling factor, and the second value is EWMA minus the product of EWMSTD and a scaling factor. For example, with reference to Figures 4, 6, 9, and 10, the change detection component or subsystem 430 (e.g., control limit generator 431) may identify, as part of the set of change detection actions 630, a set of control limits (e.g., UCL 955 and LCL 957) for identifying changes in relationships based on the set of change detection actions 630 based on a sensitivity set by an administrator (e.g., via input element 1016).

[0057] At 1212, the system may collect, prepare, and / or identify current (e.g., change detection) data for a current time period (or step). The current time period may be either the second time period or the fourth time period in some aspects. In some aspects, data may be collected for a recently initiated set of change detection operations (or after identifying a change in the relationship between energy usage and the associated characteristic). Data for an ongoing (or newly initiated) set of change detection operations (e.g., for an ongoing change detection operation associated with the second time period) may be prepared and / or identified from previously collected data stored in one or more data structures discussed in connection with FIG. 4 and FIG. 5. The current and / or change detection data may be of the same spatial and temporal granularity level (as well as the same spatial extent) as the training and validation data in some aspects. For example, with reference to Figures 4-10, data preparation component or subsystem 410 (or data prepper 414) may prepare change detection data associated with change detection window 912b (or change detection data associated with a portion of test windows 812, 814, 816, 822, 824, or 826) from information stored in a data structure such as that shown in Figure 5 using operations discussed in connection with one or more of sets of data preparation operations 610 or 710 according to spatial and temporal granularity input via one or more input elements, such as input elements 1011-1015 of Figure 10.

[0058] At 1214, the system may generate a prediction based on the machine trained model and the current (or change detection) data for the current time period for change detection. For example, when performing change detection in a second or fourth time period, a prediction may be generated based on data associated with the second or fourth time period, respectively. To predict a metric of interest (e.g., PUE or DCiE) based on or using the machine trained model, the prediction may be based on data regarding relevant characteristics (e.g., OAT) and energy usage (e.g., IT power) included in the current (or change detection) data. For example, referring to FIGS. 4-6, the dependency modeling component or subsystem 420 may generate a prediction at 631 based on input data (e.g., data stored in data table 560).

[0059] In some aspects, the system may determine at 1216 whether it detects a change in the relationship between the energy usage and the associated characteristic based on the generated prediction. The identification at 1216 may in some aspects be based on a prediction error reflecting or being a measure of a difference between a predicted energy usage for a current (e.g., second or fourth) time period based on a machine-trained model and an actual energy usage for a current (e.g., second or fourth) time period as indicated in the current (e.g., second or fourth) data being either greater than a first (control limit) value or less than a second (control limit) value. In some aspects, the identified change is further based on the difference being either greater than the first value or less than the second value at least a threshold number of times.

[0060] If the system determines at 1216 that a change has been detected (for the second time period), the system may proceed to display at 1218 an indication of the identified change. This indication may in some aspects be included within a graphical display of the forecast error over time for the current time period as one or more of an overlaid graphical element or a modification to the presentation of the forecast error (e.g., a change in color or line style). In some aspects, the indication may include a text-based alert. Displaying the indication of the identified change at 1218 may in some aspects include displaying an indication of a probable cause of the identified change (if it is a negative change such as a decrease in efficiency) and / or an indication of possible (or recommended) actions to mitigate the identified change. For example, with reference to FIGS. 4, 6, and 10, upon detecting or identifying a change, e.g., based on the set of change detection operations 630 (e.g., at 635), the change detection component or subsystem 430 may output (at 637) an indication of the identified change and / or recommendations for cause / mitigation (e.g., via the display area 1030, indication 1035 and / or an indication 1037 of recommendations for addressing the detected change and / or its predicted causes).

[0061] The system may then go back and collect data for an additional period (e.g., a third period) at 1202 to generate a new model based on the changed relationship. The system may, in some aspects, refrain from using the machine-trained model generated based on the first data until updating (or retraining) the machine-trained model based on the third data is completed. For example, based on an assumption that the machine-trained model generated based on the first data will not be accurate after the detected change, the system may not use the machine-trained model generated based on the first data and may instead wait for a new model to be generated based on the third data. The third period may, in some aspects, include at least a threshold time for collecting data to update (or generate) the machine-trained model after the identified change.

[0062] If the system determines at 1216 that no change was detected (e.g., there is no change beyond a threshold amount of change) in the current period (e.g., the fourth period), the system may proceed to determine whether to update the model based on recently collected data at 1220. In some aspects, if the system determines at 1220 not to update the model, the system may proceed to 1208 to identify data from a previous change detection period and / or window as the validation period and / or window for the next (now current) period (or step). If the system determines at 1220 to update the model, the system may proceed to updating the training data at 1222. In some aspects, updating the training data may include including the current validation data in the training dataset (with or without removing the oldest training window from the training dataset associated with the current or previous step). For example, after detecting no change in the relationship during the fourth period, the system may add data from a fifth period to the training dataset to update the model and then use it to perform change detection for the second period. After updating the training dataset at 1222, the system may return to 1206 to generate (or update) the machine-trained model. The method may be executed (e.g., looped) for a fixed number of loops, for a set amount of time, or until an input is received indicating a stop of the process. These operations may be executed in parallel, in some aspects, for different aggregation levels or different areas and / or components of the DC.

[0063] As discussed above, the disclosed methods and / or systems may provide improvements in training a model of the relationship between energy usage at a DC (or a related metric of interest, e.g., PUE or DCiE, or other efficiency or energy usage metric) and characteristics of the DC. Improved methods may include improvements related to inputs considered (e.g., types of data considered), updating the model as more data is collected and processed, and using the model to identify changes in the underlying (or actual) relationship between energy usage at a DC and characteristics of the DC.

[0064] For example, as described above, the method may in some aspects use a limited (or reduced) amount of data for initialization and then perform change-point detection using an incremental approach (e.g., reflected in incrementally adjusted control limits) that includes updating normal conditions after change-point detection. Because sudden changes in power normal conditions are quite common in DC and updates should be enabled as soon as possible, in some aspects, using limited (or reduced) data may significantly improve modeling and / or change-point detection. Furthermore, change-point detection based on regression modeling on PUE based on relevant characteristics such as OAT may be superior to simple change-point detection performed directly based on PUE alone. The disclosed machine-trained model-based system and / or method may reduce the likelihood of the system and / or method falsely identifying a change as a change-point based on influencing factors (which may not be suitable for change and / or mitigation and should be ignored).

[0065] 13 illustrates an exemplary computing environment including an exemplary computing device suitable for use in some exemplary implementations. The computing device 1305 in the computing environment 1300 can include one or more processing units, cores, or processors 1310, memory 1315 (e.g., RAM, ROM, and / or the like), internal storage 1320 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or IO interface 1325, any of which can be coupled over a communication mechanism or bus 1330 for communicating information or embedded in the computing device 1305. The IO interface 1325 is also configured to receive images from a camera or provide images to a projector or display, depending on the desired implementation.

[0066] The computing device 1305 can be communicatively coupled to an input / user interface 1335 and an output device / interface 1340. Either or both of the input / user interface 1335 and the output device / interface 1340 can be wired or wireless interfaces and can be removable. The input / user interface 1335 can include any device, component, sensor, or interface, physical or virtual, that can be used to provide input (e.g., buttons, touch screen interfaces, keyboards, pointing / cursor controls, microphones, cameras, Braille, motion sensors, accelerometers, optical readers, and / or the like). The output device / interface 1340 can include displays, televisions, monitors, printers, speakers, Braille, and the like. In some example implementations, the input / user interface 1335 and the output device / interface 1340 can be embedded in or physically coupled to the computing device 1305. In other example implementations, other computing devices may function as or provide the functionality of the input / user interface 1335 and the output device / interface 1340 for the computing device 1305 .

[0067] Examples of computing devices 1305 may include, but are not limited to, highly mobile devices (e.g., smart phones, devices in vehicles and other machines, devices carried by humans and animals, etc.), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), as well as devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions, radios with one or more processors embedded and / or coupled thereto, etc.).

[0068] Computing device 1305 can be communicatively coupled (e.g., via IO interface 1325) to external storage 1345 and a network 1350 to communicate with any number of networked components, devices, and systems, including one or more computing devices of the same or different configurations. Computing device 1305 or any connected computing device can function as, provide services to, or be referred to as a server, client, thin server, general purpose machine, special purpose machine, or other label.

[0069] The IO interface 1325 can include, but is not limited to, wired and / or wireless interfaces using any communication or IO protocol or standard (e.g., Ethernet, 1302.11x, Universal System Bus, WiMax, modem, cellular network protocols, etc.) to communicate information to and from at least all connected components, devices, and networks in the computing environment 1300. The network 1350 can be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephonic network, a cellular network, a satellite network, etc.).

[0070] The computing device 1305 may use and / or communicate using computer usable or computer readable media, including transitory and non-transitory media. Transitory media include transmission media (e.g., metallic cables, optical fibers), signals, carrier waves, etc. Non-transitory media include magnetic media (e.g., disks and tapes), optical media (e.g., CD ROM, digital video disks, Blu-ray disks), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage), and other non-volatile storage or memory.

[0071] The computing device 1305 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some exemplary computing environments. The computer-executable instructions can be retrieved from a transitory medium, stored in a non-transitory medium, and retrieved from a non-transitory medium. The executable instructions can originate from one or more of any programming language, scripting language, and machine language (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0072] The processor 1310 can run under any operating system (OS) (not shown) in a native or virtual environment. One or more applications can be deployed, including a logic unit 1360, an application programming interface (API) unit 1365, an input unit 1370, an output unit 1375, and an inter-unit communication mechanism 1395 for different units to communicate with each other, with the OS, and with other applications (not shown). The described units and elements can vary in design, function, configuration, or implementation and are not limited to the provided description. The processor 1310 can be in the form of a hardware processor, such as a central processing unit (CPU), or a combination of hardware and software units.

[0073] In some example implementations, when information or instructions for execution are received by the API unit 1365, they may be communicated to one or more other units (e.g., logic unit 1360, input unit 1370, output unit 1375). In some examples, logic unit 1360 may be configured to control information flow between units and, in some example implementations described above, direct the services provided by API unit 1365, input unit 1370, output unit 1375. For example, the flow of one or more processes or implementations may be controlled by logic unit 1360, alone or in cooperation with API unit 1365. Input unit 1370 may be configured to obtain inputs for calculations described in the example implementations, and output unit 1375 may be configured to provide outputs based on the calculations described in the example implementations.

[0074] The processor 1310 can be configured to collect first data for a first time period and second data for a second time period relating to energy usage and related characteristics of the data center. The processor 1310 can be configured to generate a first machine-trained model that models a relationship between the energy usage and the related characteristics based on the first data for the first time period. If a change in the relationship between the energy usage and the related characteristics is identified based on a first prediction error associated with the second time period, which is a measure of a difference between a first predicted energy usage for the second time period based on the first machine-trained model and a first actual energy usage for the second time period as indicated in the second data, being either greater than a first value or less than a second value, the processor 1310 can be configured to display an indication of the identified change and, based on the identified change, collect third data for a third time period and generate a second machine-trained model based on the third data. The processor 1310 can be configured to receive a selection of the first temporal granularity level and the second spatial granularity level. The processor 1310 may be configured to collect a fourth data for a fourth time period following the first time period and preceding the second time period. The processor 1310 may be configured to determine a second average prediction error for the fourth time period based on a second predicted energy usage for the fourth time period predicted by the first machine trained model and a second actual energy usage for the fourth time period indicated in the fourth data. The processor 1310 may be configured to determine a standard deviation of the second prediction error, the first value and the second value being based on the average of the second prediction error and the standard deviation of the second prediction error. The processor 1310 may be configured to collect a fifth data for a fifth time period following the first time period and preceding the fourth time period. The processor 1310 may be configured to determine at least a further mean or a further standard deviation for the third prediction error based on the third predicted energy usage for the fifth time period predicted by the machine-trained model and the third actual energy usage for the fifth time period represented in the fifth data.The processor 1310 can be configured to use the machine-trained model to predict the first projected energy usage. The processor 1310 can be configured to update the machine-trained model based on the fifth data and at least a subset of the first data. The processor 1310 can be configured to refrain from using the first machine-trained model until a second machine-trained model is generated based on the third data.

[0075] Some of the detailed descriptions are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the essence of their innovations to others skilled in the art. An algorithm is a sequence of defined steps leading to a desired end state or result. In exemplary implementations, the steps performed require physical manipulations of tangible quantities to achieve a tangible result.

[0076] Unless otherwise indicated, and as will be apparent from this discussion, it will be understood that throughout this description, discussions using terms such as "processing," "computing," "calculating," "determining," "displaying," and the like can include operations and processes of a computer system or other information processing device that manipulate and convert data represented as physical (electronic) quantities in the registers and memory of the computer system into other data that is similarly represented as physical quantities in the memory or registers of the computer system, or other information storage, transmission, or display device.

[0077] Exemplary implementations may also relate to an apparatus for performing the operations of the present specification. The apparatus may be specially constructed for the required purposes or may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored on a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. Computer-readable storage media may include tangible media, such as, but not limited to, optical disks, magnetic disks, read-only memory, random access memory, solid-state devices, and drives, or any other type of tangible or non-transitory medium suitable for storing electronic information. Computer-readable signal media may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. A computer program may include a pure software implementation that includes instructions to perform the operations of a desired implementation.

[0078] Various general-purpose systems may be used with the programs and modules according to the examples herein, or it may prove convenient to construct more specialized apparatus to perform the desired method steps. Also, the example implementations are not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​may be used to implement the teachings of the example implementations described herein. Instructions of the programming language may be executed by one or more processing devices, such as, for example, a central processing unit (CPU), a processor, or a controller.

[0079] As known in the art, the operations described above can be performed by hardware, software, or some combination of software and hardware. Various aspects of the exemplary implementations can be implemented using circuits and logic devices (hardware), while other aspects can be implemented using instructions stored on a machine-readable medium (software) that, when executed by a processor, cause the processor to perform methods for implementing the present application. Furthermore, some exemplary implementations of the present application can be performed exclusively in hardware, while other exemplary implementations can be performed exclusively in software. Furthermore, the various functions described can be performed in a single unit or distributed across several components in any number of ways. When performed by software, the methods can be executed by a processor, such as a general-purpose computer, based on instructions stored on a computer-readable medium. If desired, the instructions can be stored on the medium in compressed and / or encrypted form.

[0080] Moreover, other implementations of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the teachings of the present application. Various aspects and / or components of the described exemplary implementations may be used alone or in any combination. It is intended that the specification and exemplary implementations be considered as examples only, with a true scope and spirit of the present application being indicated by the following claims.

Claims

1. collecting first data for a first time period and second data for a second time period relating to energy usage and related characteristics of the data center; generating a first machine-trained model that models a relationship between the energy usage and the associated characteristic based on the first data for the first time period; a change in the relationship between the energy usage and the associated characteristic is identified based on a first prediction error associated with the second time period that is a measure of a difference between a first predicted energy usage for the second time period based on the first machine-trained model and a first actual energy usage for the second time period as indicated in the second data being one of greater than a first value or less than a second value; displaying an indication of the identified changes; and collecting third data for a third time period based on the identified change; and generating a second machine-trained model based on the third data; and A computer-implemented method comprising:

2. The method of claim 1 , wherein the generating the second machine-trained model is further based on the second data.

3. 2. The method of claim 1, further comprising refraining from using the first machine-trained model until the second machine-trained model is generated based on the third data, the third period of time including at least a threshold time for collecting data for generating the second machine-trained model after the identified change.

4. The method of claim 1 , wherein the identified change is further based on the difference being one of greater than the first value or less than the second value at least a threshold number of times.

5. 2. The method of claim 1, wherein the associated characteristic includes at least an outside temperature, the energy usage includes at least first energy usage data associated with a first power consumed by equipment providing information technology (IT) functions at the data center and second energy usage data associated with a second power consumed by the data center, and the relationship between the energy usage and the associated characteristic includes a particular relationship between the second power, the first power, and the associated characteristic.

6. 6. The method of claim 5, wherein the particular relationship between the second power, the first power, and the associated characteristic includes a function for calculating a power usage effectiveness (PUE) based on the first power and the associated characteristic, the PUE being calculated by dividing the second power by the first power.

7. 6. The method of claim 5, wherein the first energy usage data and the second energy usage data comprise one or more of energy usage data at a first set of two or more spatial granularity levels or energy usage data at a second set of two or more temporal granularity levels, the first set of two or more spatial granularity levels comprising one or more of an IT device level granularity, a rack level granularity, a rack group level granularity, a room level granularity, a room group level granularity, a floor level granularity, a building level granularity, or a data center level granularity, and the second set of two or more temporal granularity levels comprises one or more of seconds, minutes, hours, days, weeks, months, quarters, or years.

8. 8. The method of claim 7, further comprising receiving a selection of a first temporal level of granularity and a second spatial level of granularity, and generating the first machine-trained model is further based on the first temporal level of granularity and the second spatial level of granularity.

9. collecting fourth data for a fourth time period following the first time period and preceding the second time period; determining a second average prediction error for the fourth time period based on a second predicted energy usage for the fourth time period predicted by the first machine-trained model and a second actual energy usage for the fourth time period represented in the fourth data; determining a standard deviation of the second prediction error; 2. The method of claim 1 , further comprising: wherein the first value and the second value are based on the mean of the second prediction errors and the standard deviation of the second prediction errors.

10. 10. The method of claim 9, wherein the average of the second forecast errors is an exponentially weighted moving average (EWMA), the standard deviation of the second forecast errors is an exponentially weighted moving standard deviation (EWM standard deviation), the first value is the EWMA plus the EWM standard deviation, and the second value is the EWMA minus the EWM standard deviation.

11. collecting a fifth data for a fifth time period following the first time period and preceding the fourth time period; determining at least a further mean or a further standard deviation for a third prediction error based on a third predicted energy usage for the fifth time period predicted by the first machine-trained model and a third actual energy usage for the fifth time period represented in the fifth data; if a change in the relationship between the energy usage and the associated characteristic that exceeds a threshold is identified based on the second prediction error being within a range between a third value and a fourth value, and the third value and the fourth value are based on at least one of the further mean for the third prediction error or the further standard deviation for the third prediction error; using the first machine trained model to predict the first predicted energy usage; The method of claim 9 further comprising:

12. 12. The method of claim 11, further comprising updating the first machine-trained model based on the fifth data and at least the subset of the first data, and the first predicted energy usage for the second time period is based on the first machine-trained model after the updating of the first machine-trained model based on the fifth data and at least the subset of the first data.

13. Memory, at least one processor coupled to the memory; and based at least in part on the information stored in the memory, the at least one processor: collecting first data for a first time period and second data for a second time period relating to energy usage and related characteristics of the data center; generating a first machine-trained model that models a relationship between the energy usage and the associated characteristic based on the first data for the first time period; a change in the relationship between the energy usage and the associated characteristic is identified based on a first prediction error associated with the second time period that is a measure of a difference between a first predicted energy usage for the second time period based on the first machine-trained model and a first actual energy usage for the second time period as indicated in the second data being one of greater than a first value or less than a second value; displaying an indication of the identified changes; and collecting third data for a third time period based on the identified change; and generating a second machine-trained model based on the third data; and An apparatus configured to perform the steps of:

14. 14. The apparatus of claim 13, wherein the at least one processor configured to generate the second machine-trained model is configured to generate the second machine-trained model based on the second data.

15. The at least one processor 14. The apparatus of claim 13, further configured to refrain from using the first machine-trained model until the second machine-trained model is generated based on the third data, the third period of time including at least a threshold time for collecting data for generating the second machine-trained model after the identified change.

16. The apparatus of claim 13 , wherein the identified change is further based on the difference being one of greater than the first value or less than the second value at least a threshold number of times.

17. 14. The apparatus of claim 13, wherein the associated characteristic includes at least an outside temperature, the energy usage includes at least first energy usage data associated with a first power consumed by equipment providing information technology (IT) functions at the data center and second energy usage data associated with a second power consumed by the data center, and the relationship between the energy usage and the associated characteristic includes a particular relationship between the second power, the first power, and the associated characteristic.

18. 20. The apparatus of claim 17, wherein the first energy usage data and the second energy usage data comprise one or more of energy usage data at a first set of two or more spatial granularity levels or energy usage data at a second set of two or more temporal granularity levels, the first set of two or more spatial granularity levels comprising one or more of an IT device level granularity, a rack level granularity, a rack group level granularity, a room level granularity, a room group level granularity, a floor level granularity, a building level granularity, or a data center level granularity, and the second set of two or more temporal granularity levels comprises one or more of seconds, minutes, hours, days, weeks, months, quarters, or years.

19. The at least one processor 20. The apparatus of claim 18, further configured to perform receiving a selection of a first temporal granularity level and a second spatial granularity level, and wherein the at least one processor configured to generate the first machine-trained model is configured to generate the first machine-trained model based on the first temporal granularity level and the second spatial granularity level.

20. The at least one processor collecting fourth data for a fourth time period following the first time period and preceding the second time period; determining a second average prediction error for the fourth time period based on a second predicted energy usage for the fourth time period predicted by the first machine-trained model and a second actual energy usage for the fourth time period represented in the fourth data; determining a standard deviation of the second prediction error; 14. The apparatus of claim 13, further configured to perform: wherein the first value and the second value are based on the average of the second forecast errors and the standard deviation of the second forecast errors, wherein the average of the second forecast errors is an exponentially weighted moving average (EWMA), and the standard deviation of the second forecast errors is an exponentially weighted moving standard deviation (EWM standard deviation), and the first value is the EWMA plus the EWM standard deviation and the second value is the EWMA minus the EWM standard deviation.

Images