Abnormal behavior detection in AI-enabled systems with multiple correlated sensors

Abstract

Embodiments of the present disclosure relate to abnormal behavior detection of an artificial intelligence-enabled system having multiple related sensors. A multi-metric artificial intelligence (AI) / machine learning (ML) model for detecting abnormal behavior of a machine / system is disclosed. The multi-metric AI / ML model is configured to detect abnormal behavior of a system having multiple sensors (such as a coolant distribution unit (CDU)) that measure related sensor metrics. The multi-metric AI / ML model performs abnormal system behavior detection in a manner that can reduce both the number of sensor instruments required to monitor system operating behavior and the complexity of the firmware that controls the sensor instruments. In this way, the AI-enabled system and corresponding method for abnormal behavior detection disclosed herein provide a technical solution to the technical problem of increased failure rate of existing multi-sensor systems, which is caused by the presence of redundant sensor instruments, which requires complex firmware to control the sensor instruments.

Description

Background Art

[0001] The system may include multiple sensors configured to monitor the operating state of the system. The metrics measured by the sensors may be correlated with each other. For example, a first sensor metric measured by a first sensor may be directly related to a second sensor metric measured by a second sensor, such that if the first sensor detects an increase or decrease in the first sensor metric, the second sensor may also register a corresponding increase or decrease in the second sensor metric, and vice versa.

[0002] A coolant distribution unit (CDU) is one such system that includes a number of associated sensors. CDUs are widely deployed at customer sites along with liquid-cooled high performance computing (HPC) clusters. Typically, an HPC system is supported by multiple CDUs so that if a particular CDU fails, the portion of the HPC system supported by that CDU can be shut down, but the rest of the HPC system can continue to operate. However, for some HPC systems, this may include, for example, isolated loops where a single CDU supports multiple racks, and in the absence of redundant CDUs (which customers are typically reluctant to deploy due to increased costs), if a CDU fails, the racks that rely on the CDU for cooling functionality will shut down.

[0003] To increase CDU resiliency, a large number of sensors may be provided, and in many cases redundant sensors may be provided, to allow for extensive monitoring and data collection from the CDU. However, counter-intuitively, the increased complexity of the CDU and its firmware resulting from this increased sensor instrumentation may result in an increased failure rate for deployed CDUs due to failures in sensors, firmware, control systems, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] According to one or more different embodiments, the present disclosure is described in detail with reference to the following drawings. These drawings are provided for illustration purposes only and merely depict typical or example embodiments.

[0005] Figure 1 Depicted is an example configuration of a coolant distribution unit (CDU) including a plurality of related sensors configured to monitor operational behavior of the CDU according to an example embodiment of the present invention.

[0006] Figure 2 An example computational engine of a multi-metric, time-aware machine learning model configured to identify abnormal behavior of a CDU according to an example embodiment of the present invention is depicted.

[0007] Figure 3Depicted is a set of executable instructions stored in a machine-readable storage medium according to an example embodiment of the present invention that, when executed, causes an example method to be performed to identify abnormal behavior of a CDU using a multi-metric, time-aware machine learning model.

[0008] Figure 4A Illustrated is a graph of example actual sensor metric values ​​measured over time for a first CDU and corresponding reconstructed metric values ​​and reconstruction errors according to an example embodiment of the invention.

[0009] Figure 4B Illustrated is a graph of example actual sensor metric values ​​measured over time for a second CDU in the same cluster as a first CDU and corresponding reconstructed metric values ​​and reconstruction errors according to an example embodiment of the present invention.

[0010] Figure 5 are example computing components that can be used to implement various features of the example embodiments of the present invention.

[0011] The drawings are not exhaustive and do not limit the disclosure to the precise forms disclosed. DETAILED DESCRIPTION

[0012] Example embodiments of the present invention relate, among other things, to systems, methods, computer-readable media, techniques, and methodologies for employing multi-metric artificial intelligence (AI) / machine learning (ML) models to detect abnormal behavior of machines / systems. More specifically, example embodiments of the present invention relate to using multi-metric AI / ML models to detect abnormal behavior of a system having multiple sensors (e.g., a coolant distribution unit (CDU)) measuring relevant sensor metrics, wherein the abnormal behavior detection is performed in a manner that can reduce the number of sensor instruments required to monitor the system's operating behavior and enhance system resiliency via fast and scalable anomaly detection.

[0013] In particular, the AI-enabled system and corresponding method for abnormal behavior detection disclosed herein provide a technical solution for the technical problem of increased failure rates of multi-sensor systems (such as CDUs) caused by increased system and firmware complexity, where the increased system and firmware complexity is caused by the increase of sensor instruments. The technical solution is achieved by using AI / ML multi-metric models to perform highly scalable, automatic and high-speed anomaly detection for a single machine / system deployment and / or across multiple deployments within a cluster so that the complexity of the sensor instrument can be reduced, which in turn produces a result of more simplified firmware for controlling the instrument. For example, by reducing the number of sensors (e.g., reducing the redundancy level of one or more types of sensors) and / or by using lower cost sensors, the complexity reduction of the sensor instrument can be achieved. Fewer sensors also lead to simplified, more reliable and lower cost control hardware for controlling the sensors. The reduction in the complexity of the sensor instrument and the corresponding reduction in the complexity of the software, firmware and hardware required to control the operation of such an instrument produce the technical benefit of reduced system failure rates.

[0014] Referring now to illustrative embodiments of the present invention, Figure 1 An example configuration of a coolant distribution unit (CDU) 100 is depicted that includes multiple groups of related sensors configured to monitor the operational behavior of the CDU 100. For example, the CDU 100 can provide cooling functions to liquid-cooled servers within a deployment environment. The CDU 100 includes various types of components, including, but not limited to, mechanical components, electrical components, sensing and control components, fluid flow conditioning components, and heat exchange components. The primary functions of the CDU 100 are 1) to isolate coolant provided to information technology (e.g., server) racks from the (typically dirty) water at the facility where the servers are deployed, and 2) to transfer heat from the IT racks to the facility's cooling equipment.

[0015] For example, the fluid flow regulating components of the CDU 100 include pumps (e.g., pump 122), variable frequency drives (not shown), valves (e.g., valve 108, valve 110), etc. These fluid flow regulating components operate to circulate both coolant and facility water in respective isolated loops through the heat exchanger 102 of the CDU 100. The heat exchanger 102 can transfer heat from the coolant loop of the IT to the facility water so that the coolant can provide the necessary cooling function for the IT racks. For example, the sensing components include temperature sensors, flow meters, pressure (and pressure drop) sensors, etc.

[0016] Components of the CDU 100 depicted to the left of the heat exchanger 102 correspond to the primary (facility) side, while components of the CDU 100 depicted to the right of the heat exchanger 102 correspond to the secondary (server or information technology (IT)) side. The primary side includes a primary fluid flow 104 of "dirty" water for the deployment facility and various components for sensing fluid parameters (e.g., temperature, pressure, etc.); regulating fluid flow (e.g., valves); filtering contaminants from the primary fluid flow 104 prior to introduction into the heat exchanger 102; and the like. For example, a control valve 110 regulates fluid flow through the heat exchanger 102. Additionally, in the example embodiment, a normally closed valve 108 allows the CDU 100 to bypass flow around the heat exchanger 102. Similarly, the secondary side includes a secondary fluid flow 106 through the IT racks and fluid parameter sensing components, fluid flow regulating components (e.g., pump 122); and the like.

[0017] From a thermal perspective, the primary sensor metrics measured regarding the operation of the CDU 100 include coolant temperature, flow rate, pressure, and pressure drop. In addition, electrical metrics such as current and voltage, and control variables such as signals to and from the variable frequency drive are measured. Traditionally, instrumentation has played a critical role in ensuring smooth and reliable operation of the CDU. As such, a significant investment has been made in the instrumentation and associated firmware for the CDU. Furthermore, redundant instrumentation is typically employed. For example, multiple temperature sensors (e.g., thermocouples) may be used to measure the temperature of the pump 122 prior to operation ( Figure 1 ) and after operation and before and after operation of the heat exchanger 102 ( Figure 1 124B and T2C 124C, each measuring the temperature of the coolant after the pump 122 (i.e., the temperature of the coolant being delivered to the IT racks). The individual temperature readings from the redundant temperature sensors 124A, 124B, 124C may be aggregated (e.g., averaged), or some other statistic (e.g., median) may be derived from the multiple readings to obtain a composite / derived temperature reading. Other types of sensors may also be provided redundantly. For example, multiple pressure sensors PS1A 120A and PS1B 120B may be provided on the secondary side.

[0018] Reference now Figure 1As shown in the additional instrumentation depicted in FIG, the primary side may also include various temperature and pressure sensors, such as a thermocouple 112 and a pressure sensor 114 located before the filter 116 and a pressure sensor 118 located after the filter 116. The temperature reading from the thermocouple 112 can be used to monitor the operating status of the pump, cooling tower, etc. on the primary side. If the temperature T1 measured by the thermocouple 112 is too high, it will be difficult to maintain the target temperature of the coolant on the secondary side, which will eventually cause the IT rack to shut down or fail if the temperature on the secondary side is too high. A combination of pressure sensor 118 and pressure sensor 114 is used to measure the pressure drop across the filter 116. If the pressure drop exceeds a specified threshold, maintenance can be scheduled for the filter 116.

[0019] Further, in addition to the redundant pressure sensor 120 and the redundant thermocouple 124, the secondary side may also include: a pressure sensor 126 to measure the coolant pressure after the pump 122; a y-strainer 128 configured to mechanically remove unwanted solids from the coolant by means of a perforated or wire mesh filter element; a thermocouple 130 to measure the coolant temperature leaving the IT rack; and the like. It should be understood that Figure 1 The various components depicted as part of the CDU 100 (whether on the primary or secondary side) are not exhaustive, and the CDU 100 may include additional mechanical components, electrical components, sensing and control components, fluid flow conditioning components, and / or heat exchange components beyond those described.

[0020] Referring now to the operation of the CDU 100 in more detail, in general, the CDU 100 may attempt to maintain a target temperature on the secondary side. For example, the CDU 100 may attempt to maintain an average temperature reading of the individual temperature readings of the thermocouples T2A, T2B, and T2C within a certain desired range. In an example embodiment, the valve 110 may regulate the flow of water through the heat exchanger 102 in order to maintain a desired target temperature of the coolant supplied to the IT rack, the desired target temperature being determined, for example, based on the readings from the thermocouple 124. For example, if the temperature measured by the thermocouple 130 increases (which indicates, for example, an increase in the load of the IT racks served by the CDU 100), this may ultimately cause the coolant temperature measured by the thermocouples 124A, 124B, 124C after the pump 122 to rise outside of the desired range. To address this issue, valve 110 can be controlled to allow more fluid to flow into heat exchanger 102 to provide increased heat transfer from the coolant, thereby reducing the coolant temperature and ensuring that the temperature measured by thermocouples 124A, 124B, 124C is within a desired range. Conversely, if the coolant temperature measured by thermocouple 130 decreases (which indicates, for example, a decrease in load on the IT rack), valve 110 can be controlled to reduce fluid flow to heat exchanger 102 to increase the coolant temperature.

[0021] As is the case with any data center subsystem, the CDU 100 includes groups of closely related sensors. In other words, a sensed parameter (sensor metric) measured by a given sensor can be correlated with sensor metrics (multiple) measured by one or more other types of sensors, so that events detected by a given sensor can be picked up by other sensors in a group of related sensors at different levels. For example, the pump speed of pump 122 is directly related to the pump flow rate, power consumption, pressure rise across pump 122, and the absolute pressure at the outlet of pump 122. If the pump speed reported by pump 122 increases, then the other related sensors should also record an increase. If a particular sensor in a group of related sensors does not record an increase, it may indicate that the particular sensor is faulty because the likelihood of a single sensor functioning properly while every other sensor is not functioning properly is low.

[0022] For example, if the pump speed reported by pump 122 increases and the upstream pressure reported by pressure sensor 120 also increases, but no corresponding increase in the pressure reported by pressure sensor 126 is observed, it is possible that the pump speed sensor or pressure sensor 126 for pump 122 is not functioning properly. This determination can be made because the pump speed of pump 122 is known to be related to the pressure both upstream and downstream of pump 122. In this example scenario, additional related sensor metrics can be evaluated to find out which sensor is behaving abnormally. For example, the power consumption of pump 122 (which is known to be related to pump speed) can be evaluated. If no increase in the power consumption of pump 122 is observed, the confidence that the pump speed reported by pump 122 is inaccurate increases. On the other hand, if the power consumption of pump 122 does show an increase, this is more indicative of an anomaly associated with the pressure reported by pressure sensor 126.

[0023] CDUs such as CDU 100 are deployed to serve significantly more expensive IT racks. However, for cost-saving reasons, customers are generally reluctant to deploy CDUs redundantly. This reluctance brings with it the risk that a relatively cheap device (CDU) fails, causing a much more expensive device (IT rack) to fail. To compensate for this risk, CDU vendors have increased the amount of instruments in the CDU by increasing the redundancy of the instruments (e.g., redundant thermocouples, redundant pressure sensors, etc.) and by utilizing higher quality or more expensive instruments. Although high quality and expensive instruments can improve the reliability of the CDU, this also increases the computational requirements of the controller that receives and processes signals from the sensors and the storage / retrieval requirements of the storage medium that stores the CDU firmware (e.g., non-volatile memory such as read-only memory (ROM), electrically programmable read-only memory (EPROM), flash memory, etc.). In particular, the higher the instrument level, the more complex the firmware needs to be. In addition, the firmware of the deployed CDU needs to be regularly verified and upgraded, thereby incurring costs.

[0024] Exemplary embodiments of the present invention provide a technical solution for addressing the technical problem of increased failure rates associated with conventional CDU deployments due to increased instrumentation and the resulting increased firmware complexity. The technical solution is achieved by using AI / ML multi-metric models to perform highly scalable, automatic, and high-speed anomaly detection on a single machine / system deployment and / or across multiple deployments within a cluster so that the complexity of sensor instruments can be reduced, which in turn leads to simplified firmware and hardware, including lower cost and more reliable solutions. The reduced complexity of sensor instruments and the corresponding reduction in complexity of software / firmware used to control the operation of such instruments produce a technical benefit of reduced system failure rates.

[0025] The use of AI / ML multi-metric models to achieve the above technical solutions will now be described in more detail. Although the example embodiments disclosed herein are mainly described with respect to a CDU for providing liquid cooling functions to IT racks, it should be understood that embodiments of the present invention can be applied to any machine / system / subsystem with a group of related sensors, thereby performing anomaly detection in a manner that can reduce the complexity of sensor instruments and correspondingly reduce the complexity of software, firmware, processing hardware, etc. required to control the sensor instruments.

[0026] Figure 2 Depicted is an example computational engine of a multi-metric, time-aware machine learning model 200 configured to identify abnormal behavior of a CDU according to an example embodiment of the present invention. Figure 3 A computing component 300 is depicted that includes one or more hardware processors 302 and a machine-readable storage medium 304 storing a set of machine-readable / machine-executable instructions that, when executed, cause the hardware processor 302 to perform a method for identifying abnormal behavior of a CDU (e.g., CDU 100) using a multi-metric, time-aware machine learning model according to an example embodiment of the present invention. Figure 2 and Figure 3 .

[0027] In an example embodiment, Figure 3 The machine-readable storage medium 304 depicted in FIG. 5 may include a main memory 506, a read-only memory (ROM) 508, a storage device 510, or any other suitable machine-readable storage medium described herein. Figure 3 The instructions described as being stored on the machine-readable storage medium 304 may be modularized into one or more computing engines, such as Figure 2 In particular, each such computation engine may include Figure 3, such that the instructions, when executed by the hardware processor 302, cause the hardware processor 302 to perform corresponding tasks / processing. In an example embodiment, the set of tasks performed in response to the execution of a set of instructions forming part of a particular computing engine may be a set of dedicated / customized tasks for implementing a particular type / scope of processing.

[0028] In an example embodiment, Figure 3 The computing component 300 depicted in FIG. 3 may be, for example, a Figure 5 , or another computing device described herein. In some example embodiments, computing component 300 may be a desktop computer; a laptop computer; a tablet computer / device; a smart phone; a personal digital assistant (PDA); a wearable computing device; or the like. In other example embodiments, computing component 300 may be a server, a server cluster, or the like. In still other example embodiments, computing component 300 may be a customized computing device 212 ( Figure 2 ), including but not limited to field programmable gate array (FPGA), application specific integrated circuit (ASIC), programmable logic controller (PLC), programmable logic array (PLA), etc. In some other example embodiments, the computing component 300 may be the CDU 100 itself.

[0029] The hardware processor 302 may include, for example, Figure 5 304. In particular, the hardware processor 302 may include any suitable type of processing unit, including, but not limited to, a central processing unit (CPU), a microprocessor, a reduced instruction set computer (RISC) microprocessor, a complex instruction set computer (CISC) microprocessor, a microcontroller, a system on a chip (SoC), a digital signal processor (DSP), an ASIC, an FPGA, a PLC, a PLA, etc. In some example embodiments, a single integrated device may constitute the computing component 300, the hardware processor 302, and the machine-readable storage medium 304. For example, an ASIC, an FPGA, a SOC, etc. may be a computing component 300 that also includes a hardware processor 302 for executing logic hardwired into the device and / or instructions stored in the storage medium 304.

[0030] In an example embodiment, hardware processor 302 (or any other processing unit described herein) is configured to perform Figure 2, which in turn are configured to provide corresponding functionality related to AI / ML-enabled abnormal behavior detection for a system (such as CDU 100). In particular, hardware processor 302 may be configured to execute encoding engine 202, transformation engine 204, decoding engine 206, and reconstruction error determination engine 208. In an example embodiment, these engines may form part of a multi-metric, time-aware machine learning model 200 (hereinafter referred to as ML model 200). ML model 200 may be / utilize any suitable type of AI / ML model / algorithm, including but not limited to regression-based models (e.g., decision trees, random forests, artificial neural networks, etc.); classification-based models (e.g., support vector machines, Bayesian reasoning, etc.); unsupervised learning models, such as clustering-based models (e.g., k-means clustering); and the like.

[0031] Although the ML model 200 is depicted as being implemented on a custom computing device 200, it should be understood that the ML model 200 and its constituent engines may be implemented in any combination of hardware, software, and / or firmware. In some example embodiments, one or more of these engines may be implemented at least in part as software and / or firmware modules that include computer-executable / machine-executable instructions stored in a machine-readable storage medium 304 and executable by a hardware processor 302. It should be understood that any description herein of an engine performing a function inherently encompasses the function being performed in response to the engine's computer-executable / machine-executable instructions being executed by a processor.

[0032] Although the ML model 200 and its constituent computing engines are Figure 2 100, but in some example embodiments, the ML model 200 may actually be embedded in the firmware 214. More specifically, in some example embodiments, the ML model 200 may be integrated with the existing firmware 214 of the CDU 100. Regardless of whether the ML model 200 is independent of the CDU firmware 214 or integrated therewith, the AI ​​capabilities provided by the ML model 200 can reduce the complexity of the sensor instrumentation used for the CDU 100, and as a result, can reduce the complexity of the firmware 214. As previously described, this in turn achieves the technical benefit of a lower failure rate of the CDU 100.

[0033] Now combine Figure 2 For reference Figure 3 At block 306, the machine executable instructions of the ML model 200 may be executed by the hardware processor 302 such that an input vector X is received. The input vector X may take the following form: That is, the input vector X may include values ​​captured for N sensor measurements at various points in time during a time window of duration T. As previously described, the N sensor measurements may include one or more groups of related sensor measurements that may exhibit similar behavior in response to changes in sensed characteristics.

[0034] In an example embodiment, the ML model 200 is capable of identifying single or multiple metric points, contexts, and / or collective anomalies associated with the operation of the CDU 100. Some traditional anomaly detection methods assume a specific probability distribution of metric values, such as a multivariate Gaussian distribution. For such a distribution, if the metric values ​​are below a certain threshold given the distribution parameters, the values ​​are determined to be abnormal. However, in the case of a CDU, such an assumption may be unreliable. In particular, due to the nature of the physical processes within the CDU 100 represented by the sensor metrics, the assumption that the metrics follow a Gaussian distribution may not be accurate. For example, pump speed is a categorical metric that takes values ​​from a set of discrete values ​​and therefore may not be appropriately characterized by a Gaussian distribution. As such, in an example embodiment, the ML model 200 can be any suitable machine learning model / algorithm (e.g., a neural network-based approach, such as an autoencoder) configured to learn the expected behavior of a joint metric.

[0035] Still combined Figure 2 For reference Figure 3 At block 308, the machine executable instructions of the encoding engine 202 may be executed by the hardware processor 302 to cause the input vector X to be encoded to obtain a lower dimensional representation C of the input vector X. In particular, the dimension / magnitude of the lower dimensional representation C (given by |C|) may be smaller than the dimension / magnitude of the input vector X (given by |X|), i.e., |C|<|X|. Encoding the input vector X to obtain the encoded lower dimensional representation C may include losing at least some of the information in the input vector X (e.g., sensor metric values). In an example embodiment, also at block 308, the machine executable instructions of the transform engine 204 may be executed by the hardware processor 302 to cause a series of one or more non-linear transforms to be performed on the input vector X as part of the encoding process.

[0036] Additionally, in some example embodiments, values ​​corresponding to one or more sensor metrics may be discarded from the input vector X before encoding is performed. This discarding of values ​​may be possible because the ML model 200 supports abnormal behavior detection and has a high level of resilience despite the deployment of a small number of sensors. Therefore, sensor metric values ​​corresponding to one or more sensors may be discarded from the input vector X while still maintaining the ability to perform the desired abnormal behavior detection. In some example embodiments, discarding sensor metric values ​​may be due to the corresponding (multiple) sensors no longer being deployed (e.g., redundancy elimination for one or more sensors). In other example embodiments, sensor metric values ​​for particular sensors that are no longer deployed may simply not be present in the input vector X. It should be understood that discarding sensor metric values ​​(or the absence of such values) is different from the loss of information that may occur as a result of encoding the input vector X.

[0037] At block 310, the machine executable instructions of the decoding engine 206 may be executed by the hardware processor 302 to cause the input vector X to be reconstructed to obtain a reconstructed input vector X R Reconstructing the input vector X may include decoding the encoded lower dimensional representation C. In particular, the reconstructed input vector X R It can be given by the following formula: R =decode(encode(X)). In an example embodiment, the ML model 200 may be trained on a data set containing few or no anomalous data points, and therefore, the ML model 200 may learn to reconstruct metric values ​​associated with normal operating behavior of the CDU 100, but may not be able to properly reconstruct values ​​for data sets containing a large amount of anomalous data. As such, due to the nature of the training performed on the ML model 200, and the corresponding information loss when generating the encoded representation C relative to the lower dimensionality of the input vector X, the reconstructed input vector X may be R The degree of deviation from the original input vector X may be directly related to the amount of abnormal data in the input vector X.

[0038] At block 312, the machine executable instructions of the reconstruction error determination engine 208 may be executed by the hardware processor 302 to compare the input vector X with the reconstructed input vector X. R The reconstruction error (E) between is determined. In some example embodiments, the reconstruction error E may be given by E=X R -X gives the error vector. More generally, the reconstruction error E can be the input vector X R Any amount of deviation from the original input vector X. As mentioned above, the more abnormal data contained in the input vector X, the more abnormal data the reconstructed input vector X R The deviation from the original input vector X will be larger.

[0039] At block 314, the machine executable instructions of the transform engine 204 may be executed by the hardware processor 302 to cause one or more transforms to be performed on the reconstruction error E to obtain a transformed reconstruction error E. T In an example embodiment, the transform engine 204 may subject the reconstruction error vector E to one or more whitening transforms to obtain a transformed reconstruction error vector E T In some example embodiments, the whitening transform(s) may cause the elements of the reconstructed error vector E corresponding to the two or more sensor measurements to be in the transformed reconstructed error vector E T In addition, the whitening transform (multiple) may result in the transformed reconstruction error vector E T The elements of have the same variance.

[0040] At block 316, the machine executable instructions of the anomaly detection engine 210 may be executed by the hardware processor 302 to transform the anomaly score from the transformed reconstructed error vector E T In an example embodiment, the anomaly score of the initial input vector X may be determined as the transformed reconstruction error vector E T In some other example embodiments, the anomaly score may be the transformed reconstruction error vector E T Any linear combination of the elements of .

[0041] At block 318, machine executable instructions of the anomaly detection engine 210 may be executed by the hardware processor 302 to cause the anomaly score generated at block 316 to be compared to a threshold anomaly score to determine whether an anomaly exists. In an example embodiment, the anomaly may correspond to a plurality of sensor metrics having values ​​indicating abnormal behavior. However, due to correlations between the plurality of sensor metrics, even though abnormal behavior may be indicated by the values ​​of the plurality of sensor metrics, it may actually be that a smaller number of sensors / components of the CDU 100 are operating abnormally than the number of sensors generating the abnormal values. In an example embodiment, the threshold anomaly score may be determined from a distribution of multi-metric anomaly scores generated from a training data set used to train the ML model 200. In some example embodiments, the distribution of the multi-metric scores used to determine the threshold anomaly score may be a chi-square distribution. For example, the threshold anomaly score may be a value selected so that an expected percentage of values ​​in the distribution of the multi-metric anomaly scores is below the selected value.

[0042] In example embodiments, the relative number of related sensors recording abnormal values ​​may indicate whether it is more likely that the CDU (e.g., a flow conditioning component of the CDU) is experiencing abnormal behavior or that the sensors are exhibiting abnormal behavior. In certain example embodiments, if at least a minimum number of a group of related sensors record values ​​that exceed an abnormal threshold, it is more likely that the system rather than the sensors is exhibiting abnormal behavior (e.g., one or more components of the system are malfunctioning or otherwise not functioning properly). Even in some example scenarios where a minimum number of related sensors are recording abnormal values, one or more sensors in the group of related sensors may be behaving abnormally (e.g., malfunctioning) at the same time. For example, if among a group of five related sensors, four of the sensors record values ​​that exceed corresponding thresholds and one sensor does not, this may indicate 1) that the four sensors with abnormal values ​​are recording system-level fault events and 2) that the fifth sensor is also experiencing a fault event because it is exhibiting an abnormality by not recording a system-level event.

[0043] In some example embodiments, a sensor in a group of related sensors may be determined to be exhibiting abnormal behavior if 1) a sensor in the group of related sensors records a value of a sensor metric that exceeds a threshold (or otherwise exceeds an expected range of sensed values), and 2) at least a minimum number of other sensors in the group of related sensors do not exceed the corresponding threshold. In particular, the fact that other sensors in the group of related sensors did not record an abnormal event indicates that it is more likely that the sensor that did record the abnormal value is itself behaving abnormally. For example, if the first sensor in a group of five related sensors has an abnormal value, but the other three sensors in the group do not have abnormal values, it is more likely that the first sensor is exhibiting abnormal behavior.

[0044] In an example embodiment, an alarm may be generated to notify an operator of detected abnormal behavior. For example, based on at least a minimum number of relevant sensors recording corresponding values ​​that exceed corresponding thresholds, the alarm may indicate that a system-level fault may exist. The alarm may also identify one or more potential system-level causes of the observed behavior (e.g., a flow regulation component fails). In some other example embodiments, such as those example embodiments in which a sensor is determined to be more likely to exhibit abnormal behavior, the alarm may include an indication of a potential failed sensor. Based on the alarm, the operator may check the sensor, and if it is determined that the sensor is not functioning properly, the value measured by the sensor may be discarded. The sensor may be put online again after being repaired or replaced.

[0045] In some example embodiments, the threshold value that is compared to the sensor metric value to determine whether the measured metric value exceeds the threshold and therefore records an anomaly can be a dynamic value that can change based on one or more other parameters. In particular, the threshold value can be established based on training data provided to the multivariate AI / ML model and can change based on changes in one or more operating parameters, as reflected in the training data. For example, as the external temperature fluctuates, the threshold value that is compared to the coolant temperature to identify anomalies may also change. More specifically, for example, if the external temperature decreases, the expected coolant temperature may also decrease, in which case the anomaly threshold value may also decrease. Conversely, if the external temperature increases, the expected range of coolant temperature values ​​may also increase, in which case the anomaly threshold value may increase. It should be understood that many other examples are considered.

[0046] Figure 4A Illustrated is a graph of example actual sensor metric values ​​measured over time for CDU A (eg, CDU 100 ) and corresponding reconstructed metric values ​​and reconstruction errors according to an example embodiment of the invention. Figure 4A Graphs of actual values ​​and corresponding reconstructed values ​​of sensor metrics for three examples are illustrated, specifically: Figure 1 ) measured secondary temperature of the secondary stream 106; primary coolant flow rate; and valve position (valve demand). These sensor metrics may be related metrics that exhibit similar behavior related to the recorded events.

[0047] For each sensor metric, the chart includes the actual value captured for that sensor metric as well as the reconstructed value. As illustratively labeled with respect to the primary coolant flow rate metric, the actual metric value 402 is shown in slightly darker shading than the reconstructed value 404. Additionally, a reconstruction error 406 determined with respect to the relevant sensor metric among all relevant sensor metrics is shown below the chart of the actual metric value and the reconstructed value for each sensor metric.

[0048] There are various time periods during which capture and Figure 4A All three sensors of the example metric depicted in the sensor data recorded an event, which was then accompanied by an anomaly score that exceeded the anomaly threshold. In the example embodiment, this is interpreted as abnormal behavior. Figure 4A4. For example, event 408 may correspond to a water regulating valve on the primary side of CDU A abnormally opening and closing outside of its specified range (referred to as a "valve demand"), which in turn causes the temperature and primary coolant flow rate recorded by thermocouple 130 to cycle simultaneously. The abnormal cycling behavior 410 of the water regulating valve, in turn, causes corresponding cycles in other related sensor metrics, resulting in an anomaly score that exceeds an anomaly threshold, thereby indicating the presence of an anomaly.

[0049] The anomaly score determined based on the reconstruction error 406 is Figure 4A 106; primary coolant flow rate; valve demand). A multi-metric model such as ML model 200 that generates a multi-metric anomaly score provides several technical benefits over its single-metric counterpart. In particular, multi-metric models are more robust than single-metric models, produce fewer false alarms, and significantly reduce the amount of information that a data center operator needs to process. In addition, multi-metric models are able to exploit correlations between sensors in a group of related sensors. For example, if Figure 4A If any of the metrics in are removed from the multi-metric analysis, the ML model 200 will still flag the abnormal behavior. This demonstrates that even if the sensor stops functioning properly, the abnormal behavior associated with event 408 will still be flagged. For example, even if the valve demand sensor stops functioning, the temperature of the secondary stream 106 and the primary coolant flow rate will still exhibit abnormal cycle behavior, and the reconstruction error will still deviate from the abnormal threshold, indicating the presence of an anomaly. In this way, using the ML model 200 for abnormal behavior detection enables the deployment of fewer sensors and cheaper instrumentation while still maintaining a high level of system resiliency.

[0050] It should be understood that while the anomaly score exceeded the anomaly threshold during a portion of the time period corresponding to event 408 (as determined from reconstruction error 406), and thus exhibited abnormal behavior 410, this does not necessarily indicate that multiple sensors or multiple other components of CDU A are not functioning properly. For example, in the case of event 408, while each of the example sensors generated abnormal data for its corresponding sensor metric, the only component of CDU 100 that failed was the water regulating valve on the primary side. However, because the valve demand metric for the water regulating valve is related to multiple other sensor metrics, even if the sensor that captures data related to those other metrics is functioning properly, abnormal data about other sensor metrics may be observed. As such, in some example embodiments, the detection of abnormal behavior in a group of related sensor metrics may require further investigation of the sensor / component that failed and caused the other sensor metrics to exhibit abnormal behavior. Furthermore, in example embodiments, if a sensor is not functioning properly, the data from that sensor may be replaced by a historical or predicted value before being input into the ML model 200.

[0051] Now about Figure 4A and Figure 4B Another example of the ability of the multivariate ML model 200 to enhance CDU resilience is described below. Figure 4A Another example event 412 may correspond to a server or IT rack cooled by CDU A being powered off due to a maintenance event. This shutdown event 412 may result in Figure 4A 4. Abnormal data for various example sensor metrics depicted in FIG. 4 and ultimately resulting in an anomaly score exceeding an anomaly threshold, thereby indicating abnormal behavior 414. In an example embodiment, event 412 may be a facility-level event.

[0052] Figure 4B Graphs showing actual sensor measurement values ​​and corresponding reconstructed values ​​and reconstruction errors for Figure 4A 412 , but for a different CDU (CDU B). CDU B may, for example, be in the same cluster as CDU A. CDU A and CDU B may experience the same facility-level event 412, and may execute the same or different multivariate anomaly detection models. Figure 4B As shown in , the reconstruction error during a time period 416 that overlaps with the time period of event 412 exhibits an abnormal behavior 418 that is similar to the abnormal behavior 414 . Figure 4A and Figure 4BThe diagram illustrates that a multivariate model deployed on two or more CDUs is able to detect facility-level anomalies. This creates a technical benefit of achieving another level of sensor health validation. For example, if facility-level abnormal behavior is observed from 4 CDUs in a group of 5 CDUs, and the same facility-level behavior is not observed in CDU 5, then we can potentially find out that a sensor failure has occurred in CDU 5 because it is the only CDU that failed to detect the facility-level abnormal behavior. In addition, another technical benefit of employing the ML model 200 for abnormal behavior detection is that the same ML model 200 can be used not only for a given CDU (e.g., CDU A), but can also be deployed across other CDUs deployed within the same cluster (e.g., CDU B). Therefore, the ML model 200 can be employed to improve the resilience of individual and cluster-wide CDU deployments while simplifying CDU instrumentation and associated firmware.

[0053] Figure 5 A block diagram of an example computer system 500 is depicted in which various embodiments described herein may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, one or more hardware processors 504 coupled to bus 502 for processing information. For example, hardware processor(s) 504 may be one or more general purpose microprocessors.

[0054] Computer system 500 also includes a main memory 506, such as a random access memory (RAM), cache, and / or other dynamic storage device, coupled to bus 502 for storing information and instructions to be executed by processor 504. Memory 506 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 504. Such instructions, when stored in a storage medium accessible to processor 504, present computer system 500 as a special-purpose machine customized to perform specified operations in the instructions.

[0055] Computer system 500 also includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, such as a magnetic disk, optical disk, or USB thumb drive (flash drive), is provided and coupled to bus 502 for storing information and instructions.

[0056] The computer system 500 may be coupled to a display 512, such as a liquid crystal display (LCD) (or touch screen), via the bus 502 for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to the bus 502 for communicating information and command selections to the processor 504. Another type of user input device is a cursor control 516, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to the processor 504 and for controlling cursor movement on the display 512. In some embodiments, the same direction information and command selections as cursor control may be implemented without a cursor via receiving touches on a touch screen.

[0057] The computing system 500 may include a user interface module for implementing a GUI, which may be stored in a mass storage device as executable software code executed by the computing device(s). By way of example, this module and other modules may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, program code segments, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

[0058] In general, as used herein, the words "component", "engine", "system", "database", "data storage", etc. may refer to logic contained in hardware or firmware, or may refer to a collection of software instructions that may have entry points and exit points and are written in a programming language (such as, for example, Java, C, or C++). Software components may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language such as, for example, BASIC, Perl, or Python. It should be understood that software components may be callable from other components or from themselves, and / or may be called in response to detected events or interrupts. Software components configured to be executed on a computing device may be provided on a computer-readable medium such as an optical disc, digital video disc, flash drive, disk, or any other tangible medium, or provided as a digital download (and may initially be stored in a compressed or installable format that requires installation, decompression, or decryption before execution). Such software code may be stored in part or in whole on a memory device of an executing computing device for execution by the computing device. Software instructions may be embedded in firmware such as an EPROM. It should also be understood that hardware components may include connected logic units such as gates and flip-flops, and / or may include programmable units such as programmable gate arrays or processors.

[0059] The computer system 500 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with the computer system, make or program the computer system 500 into a special purpose machine. According to one embodiment, the techniques herein are performed by the computer system 500 in response to the processor(s) 504 executing one or more sequences of one or more instructions contained in the main memory 506. Such instructions may be read into the main memory 506 from another storage medium such as the storage device 510. Execution of the sequences of instructions contained in the main memory 506 causes the processor(s) 504 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0060] As used herein, the term "non-transient medium" and similar terms such as machine-readable storage media refer to any medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such non-transient media may include non-volatile media and / or volatile media. For example, non-volatile media include optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory such as main memory 506. For example, common forms of non-transient media include floppy disks, floppy disks, hard disks, solid-state drives, tapes or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with hole patterns, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips or cassettes, and networked versions thereof.

[0061] Non-transient media are distinct from transmission media but can be used in conjunction with transmission media. Transmission media participate in the transfer of information between non-transient media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including the cables that make up bus 502. Transmission media can also take the form of sound or light waves, such as those generated during radio wave and infrared data communications or broadcasts.

[0062] Computer system 500 also includes a communication interface 518 coupled to bus 502. Communication interface 518 provides two-way data communication coupled to one or more network links, and the one or more network links are connected to one or more local networks. For example, communication interface 518 can be an integrated service digital network (ISDN) card, a cable modem, a satellite modem, or a modem that provides a data communication connection to a corresponding type of telephone line. As another example, communication interface 518 can be a local area network (LAN) card (or a WAN component that communicates with a WAN) that provides a data communication connection to a compatible LAN. Wireless links can also be implemented. In any such implementation, communication interface 518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0063] A network link typically provides data communication to other data devices through one or more networks. For example, a network link may provide a connection to a host computer or data equipment operated by an Internet Service Provider (ISP) through a local network. The ISP in turn provides data communication services through a global packet data communication network now commonly referred to as the "Internet". Both the local network and the Internet use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link and through the communication interface 518 are example forms of transmission media that carry digital data to and from the computer system 500.

[0064] Computer system 500 can send messages and receive data, including program code, through network(s), network links and communications interface 518. In the Internet example, a server can transmit the requested code for an application through the Internet, an ISP, a local network and communications interface 518.

[0065] The received code may be executed by processor 504 as it is received, and / or may be stored in storage device 510 or other non-volatile storage for later execution.

[0066] Each of the processes, methods, and algorithms described in the foregoing sections may be embodied in a code component executed by one or more computer systems or a computer processor including computer hardware, and fully or partially automated by the code component. The one or more computer systems or computer processors may also operate to support the execution of related operations in a "cloud computing" environment or as "software as a service" (SaaS). The processes and algorithms may be implemented in part or in whole in a dedicated circuit. The various features and processes described above may be used independently of each other, or may be combined in various ways. Different combinations and sub-combinations are intended to fall within the scope of the present disclosure, and certain method or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular order, and the blocks or states associated therewith may be executed in other appropriate orders, or may be executed in parallel, or may be executed in some other manner. The blocks or states may be added to or removed from the disclosed example embodiments. The execution of certain operations or processes in the operations or processes may be distributed among computer systems or computer processors, not only residing in a single machine, but also deployed across multiple machines.

[0067] As used herein, any form of hardware, software or combination thereof can be utilized to implement circuits. For example, one or more processors, controllers, ASIC, PLA, PAL, CPLD, FPGA, logic components, software routines or other mechanisms can be implemented to form circuits. In implementation, the various circuits described herein can be implemented as discrete circuits, or the described functions and features can be partially or completely shared among one or more circuits. Although various features or functional elements can be described or claimed separately as separate circuits, these features and functionality can be shared among one or more common circuits, and such description should not require or imply that separate circuits are needed to implement these features or functionality. In the case where the circuit is implemented in whole or in part using software, such software can be implemented to operate together with a computing or processing system (such as computer system 500) that can perform the functionality described therein.

[0068] As used herein, the term "or" may be interpreted as inclusive or exclusive. In addition, descriptions of resources, operations, or structures in the singular should not be understood to exclude the plural. Unless otherwise specifically stated, or otherwise understood in the context of use, conditional language such as "can," "could," "might," or "may" is generally intended to convey that some embodiments include, while other embodiments do not include, certain features, elements, and / or steps.

[0069] Unless expressly stated otherwise, the terms and phrases used in this document and variations thereof should be interpreted as open ended and not restrictive. Adjectives such as "conventional," "traditional," "normal," "standard," "known," and terms of similar meaning should not be interpreted as limiting the items described to a given time period or limiting items available at a given time, but should be understood to cover conventional, traditional, normal, or standard technologies available or known at any time now or in the future. In some cases, the presence of broadening words and phrases such as "one or more," "at least," "but not limited to," or other similar phrases should not be understood to mean that a narrower case is intended or required where such broadening phrases may not be present.

Claims

1. A computer-implemented method, the method comprising: receiving sensor data captured by a plurality of sensors within a time window as an input to a machine learning model, the plurality of sensors being configured to monitor operational behavior of a machine, the sensor data comprising a respective plurality of values ​​for each of a plurality of sensor metrics; identifying a correlated set of two or more sensor metrics among the plurality of sensor metrics; encoding the sensor data using the machine learning model to obtain an encoded representation of the sensor data; reconstructing the sensor data from the encoded representation using the machine learning model to obtain reconstructed sensor data, and Using the machine learning model, determine whether the machine or at least one sensor is more likely to be exhibiting abnormal behavior based at least in part on the number of sensor metrics among the two or more sensor metrics having corresponding values ​​exceeding corresponding thresholds, and the difference between the reconstructed sensor data and the sensor data.

2. The computer-implemented method of claim 1 , further comprising: In response to the abnormal behavior, one or more remedial measures are initiated to alter the operational behavior of the machine.

3. The computer-implemented method of claim 1 , wherein determining whether the machine or the at least one sensor is more likely to be exhibiting the abnormal behavior comprises: determining that at least one relevant sensor metric exceeds a corresponding threshold; determining that at least a minimum number of the associated sensor metrics do not exceed a corresponding threshold; as well as It is determined that the at least one sensor is more likely than the machine to exhibit the abnormal behavior.

4. The computer-implemented method of claim 1 , wherein determining whether the machine or the at least one sensor is more likely to be exhibiting the abnormal behavior comprises: determining that at least a minimum number of the two or more related sensor metrics exceed respective thresholds; as well as It is determined that the machine is more likely than the particular sensor to be exhibiting the abnormal behavior.

5. The computer-implemented method of claim 1 , further comprising: using the machine learning model, determining a reconstruction error based on the difference between the reconstructed sensor data and the sensor data; as well as An anomaly score is determined from the reconstruction error using the machine learning model. 6 . The computer-implemented method of claim 5 , wherein the reconstruction error is an error vector, and wherein determining the anomaly score comprises determining a magnitude of the error vector.

7. The computer-implemented method of claim 5, further comprising: determining that the anomaly score exceeds a threshold score; as well as Based at least in part on the anomaly score exceeding the threshold score, it is determined that one of the machine or the at least one sensor is exhibiting the abnormal behavior.

8. The computer-implemented method of claim 5, wherein the two or more related sensor metrics include a first sensor metric and a second sensor metric, and wherein the reconstruction error includes a first set of values ​​corresponding to the first sensor metric and a second set of values ​​corresponding to the second sensor metric, the method further comprising: A transformation is performed on the reconstruction error, the transformation causing the first set of values ​​to become uncorrelated with the second set of values.

9. The computer-implemented method of claim 1 , further comprising: determining that the at least one sensor is exhibiting the abnormal behavior; as well as Prior to providing the sensor data as input to the machine learning model, the corresponding plurality of values ​​generated by the at least one sensor and corresponding to at least one sensor metric of the plurality of sensor metrics are discarded from the sensor data.

10. The computer-implemented method of claim 9, wherein the discarding occurs in response to the at least one sensor no longer being configured to monitor the operational behavior of the machine.

11. A system for supporting machine learning, comprising: a plurality of sensors configured to monitor operational behavior of the system; Memory, which stores machine-executable instructions; as well as a processor configured to access the memory and execute the machine-executable instructions to: receiving sensor data captured by the plurality of sensors within a time window, the sensor data relating to a plurality of related sensor metrics; encoding the sensor data to obtain encoded sensor data; decoding the encoded sensor data to obtain reconstructed sensor data; as well as Using a machine learning model, determine whether the system or at least one sensor is more likely to be exhibiting abnormal behavior based at least in part on the number of sensor metrics in the multiple related sensor metrics having corresponding values ​​exceeding corresponding thresholds during the time window, and the difference between the reconstructed sensor data and the sensor data.

12. The system of claim 11, wherein upon determining whether the machine or the at least one sensor is exhibiting the abnormal behavior, the processor is configured to execute the machine executable instructions to: determining that at least one relevant sensor metric exceeds a corresponding threshold; determining that at least a minimum number of the associated sensor metrics do not exceed a corresponding threshold; and It is determined that the at least one sensor is more likely than the machine to exhibit the abnormal behavior.

13. The system of claim 11, wherein upon determining whether the machine or the at least one sensor is exhibiting the abnormal behavior, the processor is configured to execute the machine executable instructions to: determining that at least a minimum number of the two or more correlated sensor metrics exceed respective thresholds; and It is determined that the machine is more likely than the at least one sensor to be exhibiting the abnormal behavior.

14. The system of claim 11, wherein when encoding the sensor data to obtain the encoded sensor data, the processor is configured to execute the machine executable instructions to: One or more non-linear transformations are applied to the input sensor data to obtain the encoded sensor data, wherein the encoded sensor data is a lower dimensional representation of the sensor data.

15. The system of claim 11, wherein the processor is further configured to execute the machine executable instructions to: determining a reconstruction error based on a difference between the reconstructed sensor data and the sensor data; and An anomaly score is determined from the reconstruction error.

16. The system of claim 15, wherein the processor is further configured to execute the machine executable instructions to: determining that the anomaly score exceeds a threshold score; and Based at least in part on the anomaly score exceeding the threshold score, it is determined that one of the system or the at least one sensor is exhibiting the abnormal behavior.

17. A computer program product comprising a non-transitory computer readable medium storing program instructions, the program instructions, when executed by a processor, causing operations to be performed, the operations comprising: receiving, by a machine learning model, an input vector comprising sensor measurements captured by a plurality of sensors monitoring an operating state of a machine, the sensor measurements being related to a plurality of related sensor metrics; encoding the sensor measurements using the machine learning model to obtain an encoded representation of the sensor measurements; reconstructing the sensor measurements from the encoded representation using the machine learning model to obtain reconstructed sensor measurements, and Using the machine learning model, determine whether the machine or at least one sensor is more likely to be exhibiting abnormal behavior based at least in part on a number of the sensor measurements related to the multiple related sensor metrics that exceed corresponding thresholds for the multiple related sensor metrics and a difference between the reconstructed sensor measurements and the sensor measurements.

18. The computer program product of claim 17, wherein determining whether the machine or the at least one sensor is more likely to be exhibiting the abnormal behavior comprises: determining that at least a threshold number of the plurality of correlated sensor measurements deviate from respective expected values ​​by more than respective threshold amounts; as well as The machine is determined to be more likely than the at least one sensor to exhibit the abnormal behavior.

Images