Detection of abnormal behavior by an artificial intelligence-enabled system with multiple correlated sensors

Abstract

A computer-implemented procedure that includes the following: Receiving sensor data as input for a machine learning model, acquired over a time window from a multitude of sensors configured to monitor the operational behavior of a machine, the sensor data comprising a corresponding multitude of values ​​for each from a multitude of sensor metrics; Identifying a set of two or more sensor metrics from the multitude of correlated sensor metrics; Encoding the sensor data using the machine learning model to obtain a coded representation of the sensor data; and Reconstructing the sensor data from the coded representation using the machine learning model to obtain reconstructed sensor data, and Determine, using the machine learning model, whether the machine or at least one sensor is more likely to exhibit abnormal behavior, at least partially based on a number of the two or more correlated sensor metrics that have corresponding values ​​exceeding corresponding thresholds, and a difference between the reconstructed sensor data and the sensor data.

Description

Declaration of national interest

[0001] This invention was made with the support of the United States government under grant number DE-AC36-08GO28308, issued by the National Renewable Energy Laboratory (NREL). The United States government holds certain rights to this invention. background

[0002] A system can contain multiple sensors configured to monitor the system's operational status. The metrics measured by the sensors can be correlated. For example, a first sensor measurement can be directly related to a second sensor measurement, such that if the first sensor detects an increase or decrease in the first sensor measurement, the second sensor can also register a corresponding increase or decrease in the second sensor measurement, or vice versa.

[0003] Coolant distribution units (CDUs) are a type of system that includes multiple correlated sensors. CDUs are commonly deployed at customer sites along with liquid-cooled high-performance computing (HPC) clusters. Typically, HPC systems are supported by multiple CDUs, so if one CDU fails, a portion of the HPC system supported by that CDU can be shut down while the remaining parts of the HPC system continue to operate. However, some HPC systems may be isolated loops, for example, where a single CDU supports multiple racks. If redundant CDUs are not present (which customers are often reluctant to use due to the added cost), a CDU failure will shut down the racks whose cooling functionality depends on that CDU.

[0004] To increase the resilience of a CDU, numerous sensors, and in many cases redundant sensors, can be incorporated to enable comprehensive monitoring and data acquisition. However, the increased complexity of the CDU and its firmware resulting from this enhanced sensor instrumentation can lead to a higher failure rate for deployed CDUs due to failures of sensors, firmware, control systems, and the like.

[0005] From US patent 2019 / 0 057 317 A1, a system for monitoring the health status of a system for identifying fault signatures is disclosed. When a low health index is detected, the individual sensors are subjected to plausibility checks to determine which sensors' results are primarily contributing to the low health index.

[0006] DE 11 2016 006 063 T5 concerns techniques for managing sensor anomalies in a computer system, whereby, based on the sensor data from another sensor and a correlation rule, it is determined whether the sensor data received from a first sensor is anomalous.

[0007] DE 11 2019 001 605 T5 describes a method in which sensor data from a vehicle is used to train deep neural networks, which are then tested in a simulated environment.

[0008] It is an object of the present invention to reliably detect a malfunction in a complex system with multiple correlated sensors. This object is achieved by a computer-implemented method according to claim 1, by a machine-learning-capable system according to claim 11, and by a computer program product according to claim 17. Brief description of the drawings

[0009] The present disclosure is described in detail in accordance with one or more different embodiments with reference to the following figures. The figures serve only for illustration and represent only typical or exemplary embodiments. Fig. Figure 1 shows an example configuration of a coolant distribution unit (CDU) with several correlated sensors configured to monitor the operating behavior of the CDU according to embodiments of the invention. Fig. Figure 2 shows an example of a computing machine of a multimetric, time-aware machine learning model configured to identify anomalous behavior of a CDU according to embodiments of the invention. Fig. Figure 3 shows a series of executable instructions stored in machine-readable storage media which, when executed, initiate an illustrative method for using a multimetric, time-aware machine learning model to identify anomalous behavior of a CDU according to embodiments of the invention. Fig. Figure 4A illustrates diagrams of examples of actual metric sensor values ​​measured over time for a first CDU, as well as of corresponding reconstructed metric values ​​and reconstruction errors according to embodiments of the invention. Fig. Figure 4B illustrates diagrams of examples of actual metric sensor values ​​measured over time for a second CDU in the same cluster as the first CDU, as well as corresponding reconstructed metric values ​​and reconstruction errors according to embodiments of the invention. Fig. Figure 5 is an example of a computer component that can be used to implement various features of embodiments of the invention.

[0010] The figures are not exhaustive and do not limit the present revelation to the exact form that is disclosed. Detailed description

[0011] Exemplary embodiments of the invention relate, among other things, to systems, processes, computer-readable media, techniques, and methods that employ multimetric artificial intelligence (AI) / machine learning (ML) models for detecting abnormal behavior of a machine / system. More specifically, exemplary embodiments of the invention relate to the use of multimetric AI / ML models for detecting abnormal behavior of systems with multiple sensors that measure correlated sensor metrics (e.g., coolant distribution units (CDUs)), wherein the detection of abnormal behavior is performed in a manner that allows for a reduction in the number of sensor instruments required to monitor the system's operational behavior and improves the system's resilience through fast and scalable anomaly detection.

[0012] In particular, the AI-enabled systems and corresponding methods for detecting anomalous behavior disclosed here offer a technical solution to the technical problem of increased failure rates in multi-sensor systems such as CDUs, caused by the increased system and firmware complexity resulting from the growing number of sensor instrumentations. This technical solution is achieved through the use of AI / ML multimetric models for highly scalable, automatic, and rapid anomaly detection for a single machine / system deployment and / or for multiple deployments within a cluster, thus reducing the complexity of the sensor instrumentation, which in turn leads to simplified firmware for controlling the instrumentation. Reduced sensor instrumentation complexity can be achieved, for example, by decreasing the number of sensors (e.g.,This can be achieved by reducing redundancy for one or more sensor types and / or by using more cost-effective sensors. Fewer sensors also lead to simplified, more reliable, and less expensive control hardware for managing the sensors. The reduced complexity of the sensor instruments and the corresponding reduction in the complexity of the software, firmware, and hardware required to control their operation result in the technical advantage of a lower system failure rate.

[0013] Fig. Figure 1 shows an example configuration of a Coolant Distribution Unit (CDU) 100 with several groups of correlated sensors configured to monitor the operational behavior of the CDU 100. The CDU 100 can, for example, provide cooling for liquid-cooled servers in a service environment. The CDU 100 comprises various types of components, including, but not limited to, mechanical components, electrical components, measurement and control components, components for regulating liquid flow, and heat exchange components. The main functions of the CDU 100 are: 1) isolating the coolant supplied to the IT racks (e.g., servers) from the water in the facility where the servers are located (which is typically contaminated), and 2) transferring heat from the IT racks to the facility's cooling system.

[0014] The components for regulating the fluid flow of the CDU 100 include, for example, pumps (e.g., pump 122), variable frequency drives (not shown), valves (e.g., valve 108, valve 110), etc. These fluid flow regulating components circulate both a coolant and process water through a heat exchanger 102 of the CDU 100 in corresponding insulated circuits. The heat exchanger 102 can transfer heat from the IT coolant circuit to the process water so that the coolant can provide the necessary cooling function for the IT racks. Sensor components include, for example, temperature sensors, flow meters, pressure sensors (and pressure drop sensors), and so on.

[0015] The components of the CDU 100 shown on the left side of the heat exchanger 102 correspond to a primary side (of the system), while the components shown on the right side of the heat exchanger 102 correspond to a secondary side (server or information technology (IT)). The primary side comprises the primary fluid flow 104 of the "dirty" water from the operating equipment, as well as various components for sensing fluid parameters (e.g., temperature, pressure, etc.), for controlling the fluid flow (e.g., valves), for filtering impurities from the primary fluid flow 104 before it is introduced into the heat exchanger 102, and so on. For example, the control valve 110 regulates the fluid flow through the heat exchanger 102. Additionally, the valve 108, which is normally closed, allows the CDU 100 to bypass the heat exchanger 102.Similarly, the secondary side includes the secondary fluid flow 106 through the IT racks, as well as components for recording fluid parameters, components for regulating the fluid flow (e.g., pump 122), and so on.

[0016] From a thermal standpoint, the primary sensor readings related to the operation of the CDU 100 include coolant temperature, flow rate, pressure, and pressure drop. Additionally, electrical measurements such as current and voltage, as well as control variables like signals to and from frequency converters, are measured. Instrumentation typically plays a crucial role in ensuring the smooth and reliable operation of a CDU. Therefore, significant investments are made in the instrumentation and associated firmware of a CDU. Furthermore, redundant instrumentation is generally employed. For example, multiple temperature sensors (e.g., thermocouples) can be used to measure the temperature of the cooling water before (in) the water. Fig. 1 not shown) and after the operation of the pump 122 as well as before and after the heat exchanger 102 (in Fig. (1 not shown) to measure. For example, multiple redundant thermocouples T2A 124A, T2B 124B, and T2C 124C can be provided, each measuring the temperature of the coolant after pump 122 (i.e., the temperature of the coolant supplied to the IT racks). The individual temperature readings from the redundant temperature sensors 124A, 124B, and 124C can be aggregated (e.g., averaged), or another statistical measure (e.g., the median) can be derived from the multiple readings to obtain a composite / derived temperature reading. Other types of sensors can also be redundantly present. For example, multiple pressure sensors PS1A 120A and PS1B 120B can be provided on the secondary side.

[0017] With reference to the in Fig. The additional instrumentation shown in Figure 1 can also include various temperature and pressure sensors on the primary side, such as a thermocouple 112 and a pressure sensor 114 located upstream of the filter 116, as well as a pressure sensor 118 located downstream of the filter 116. The temperature read from the thermocouple 112 can be used to monitor the operating status of pumps, cooling towers, or similar equipment on the primary side. If the temperature T1 measured by the thermocouple 112 is too high, it becomes difficult to maintain a target temperature for the coolant on the secondary side, which can ultimately lead to shutdown or failure of the IT racks if the temperature on the secondary side becomes too high. The combination of the pressure sensor 118 and the pressure sensor 114 is used to measure the pressure drop across the filter 116. If the pressure drop exceeds a certain threshold, maintenance of the filter 116 can be scheduled.

[0018] In addition to the redundant pressure sensor 120 and the redundant thermocouple 124, the secondary side can also include a pressure sensor 126 for measuring the coolant pressure downstream of the pump 122, a Y-filter 128 configured to mechanically remove unwanted solids from the coolant by means of a perforated or wire mesh filter element, a thermocouple 130 that measures the temperature of the coolant exiting the IT racks, etc. It should be noted that the various components in Fig. The components shown in Figure 1 are not complete as part of the CDU 100 (whether on the primary or secondary side) and the CDU 100 may contain additional mechanical components, electrical components, measuring and control components, components for regulating fluid flow and / or components for heat exchange beyond the components shown.

[0019] Now, if we examine the operation of the CDU 100 in more detail, the CDU 100 can generally attempt to maintain a target temperature on the secondary side. For example, the CDU 100 can try to maintain an average temperature reading of the individual temperature readings of thermocouples T2A, T2B, and T2C that lies within a certain desired range. In exemplary embodiments, the valve 110 can regulate the water flow through the heat exchanger 102 to maintain a desired target temperature for the coolant supplied to the IT racks, which is determined, for example, based on the reading of thermocouple 124. If, for example, the temperature measured by thermocouple 130 increases—which, for example,If a higher load on the IT racks supplied by CDU 100 indicates a higher load, this could ultimately lead to the coolant temperature measured by thermocouples 124A, 124B, and 124C downstream of pump 122 rising outside the desired range. To counteract this, valve 110 can be controlled to allow more fluid to flow into heat exchanger 102, thereby increasing heat transfer from the coolant and lowering the coolant temperature and ensuring that the temperature measured by thermocouples 124A, 124B, and 124C remains within the desired range. Conversely, if the coolant temperature measured by thermocouple 130 decreases—indicating, for example, a lower load on the IT racks—valve 110 can be controlled to reduce the fluid flow to heat exchanger 102, thus raising the coolant temperature.

[0020] Like every subsystem in a data center, the CDU 100 comprises groups of closely correlated sensors. In other words, a parameter measured by one sensor (sensor reading) can be correlated with the sensor reading(s) measured by one or more other sensor types, so that an event detected by one sensor can be reflected to varying degrees by other sensors in the correlated sensor group. For example, the pump speed of pump 122 is directly related to the pump flow rate, power consumption, pressure rise across pump 122, and the absolute pressure at the pump 122 outlet. If the pump speed reported by pump 122 increases, other correlated sensors should also register an increase.If a particular sensor in the group of correlated sensors does not register an increase, this could indicate a malfunction of that particular sensor, as the probability is low that the single sensor is functioning properly while all the other sensors are not.

[0021] For example, if the pump speed reported by pump 122 and the upstream pressure reported by pressure sensor 120 increase, without a corresponding increase in the pressure reported by pressure sensor 126, it could be that either the pump speed sensor for pump 122 or the pressure sensor 126 is malfunctioning. This conclusion can be drawn because the pump speed of pump 122 is known to correlate with the pressures both upstream and downstream of pump 122. In this example scenario, additional correlated sensor metrics can be evaluated to determine which sensor is behaving abnormally. For example, the current consumption of pump 122 (which is known to correlate with the pump speed) can be evaluated.If no increase in the power consumption of pump 122 is observed, the probability increases that the pump speed reported for pump 122 is incorrect. Conversely, if the power consumption of pump 122 shows an increase, this is more likely to indicate an anomaly related to the pressure reported by pressure sensor 126.

[0022] CDUs like the CDU 100 are used to power significantly more expensive IT racks. However, customers are generally hesitant to deploy CDUs redundantly for cost reasons. This reluctance carries the risk that the failure of relatively inexpensive equipment (a CDU) will lead to the failure of much more expensive equipment (IT racks). To mitigate this risk, CDU manufacturers have increased the number of instruments within the CDUs, both by increasing instrument redundancy (e.g., redundant thermocouples, redundant pressure sensors, etc.) and by using higher-quality or more expensive instruments. While high-quality and expensive instrumentation can improve the reliability of the CDU, it also increases the computational demands on the controller that receives and processes the signals from the sensors, as well as the storage / retrieval requirements of the storage media (e.g., hard drives).Non-volatile memory such as read-only memory (ROM), electrically programmable read-only memory (EPROM), flash memory, etc., stores the CDU's firmware. The greater the level of instrumentation, the more complex the firmware must be. Furthermore, the firmware for deployed CDUs must be regularly checked and updated, which incurs additional costs.

[0023] Exemplary embodiments of the invention offer a technical solution to the problem associated with conventional CDU deployments, namely the increased failure rates due to the growing instrumentation and the resulting higher firmware complexity. This technical solution is achieved by using AI / ML multimetric models for highly scalable, automatic, and rapid anomaly detection for a single machine / system and / or for multiple deployments within a cluster. This reduces the complexity of the sensor instrumentation, leading to simplified firmware and hardware, resulting in a more cost-effective and reliable solution. The reduced complexity of the sensor instruments and the correspondingly lower complexity of the software / firmware controlling their operation offer the technical advantage of lowering the system failure rate.

[0024] The use of multimetric AI / ML models to achieve the technical solution described above will now be described in more detail. While the embodiments disclosed here are primarily described in relation to a CDU for providing a liquid cooling function for IT racks, it should be understood that the embodiments of the invention can be applied to any machine / system / subsystem with correlated groups of sensors to perform anomaly detection in a manner that allows for a reduction in the sensor array and a corresponding reduction in the complexity of the software, firmware, processing hardware, or similar components required to control the sensor array.

[0025] Fig. Figure 2 shows an example of a computing machine of a multimetric, time-aware machine learning model 200, configured to identify anomalous behavior of a CDU according to embodiments of the invention. Fig. Figure 3 shows a computer component 300 containing one or more hardware processors 302 and machine-readable storage media 304 which store a set of machine-readable / machine-executable instructions which, when executed, cause the hardware processors 302 to perform a method using a multimetric, time-aware machine learning model to identify anomalous behavior of a CDU (e.g., CDU 100) according to embodiments of the invention. Fig. 2 and Fig. The three are described in the following section in conjunction with each other.

[0026] In exemplary implementations, this can be seen in Fig. 3. The machine-readable storage medium 304 shown here comprises the main memory 506, the read-only memory (ROM) 508, the memory 510, or any other suitable machine-readable storage medium described herein. The in Fig. The instructions shown in the 3, which are stored on the machine-readable storage media 304, can be modularized into one or more computing units, as shown in Fig. 2 are shown. In particular, each such computing machine can be a corresponding subset of the in Fig. The three illustrated machine-readable and machine-executable instructions are included, such that when executed by the hardware processors 302, the instructions cause the hardware processors 302 to perform corresponding tasks / processing operations. In exemplary embodiments, the set of tasks performed in response to the execution of the set of instructions that are part of a particular computing machine can be a set of specialized / adapted tasks to effect a particular type / scope of processing.

[0027] In exemplary embodiments, the in Fig. 3 computer component shown, 300 for example, the one in Fig. 5 computer system 500 shown or other computer device described herein. In some embodiments, the computer component 300 may be a desktop computer, a laptop computer, a tablet computer / device, a smartphone, a personal digital assistant (PDA), a portable computing device, or the like. In other embodiments, the computer component 300 may be a server, a server cluster, or the like. In further embodiments, the computer component 300 may be a custom computing device 212 ( Fig. 2) be, including but not limited to, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a programmable logic controller (PLC), a programmable logic array (PLA), or the like. In other embodiments, the computer component 300 may be the CDU 100 itself.

[0028] The 302 hardware processors, for example, can be used in Fig. The hardware processor(s) 304 shown in Figure 5 may comprise the processor(s) 504. In particular, the hardware processors 302 may comprise any suitable type of processing unit, including, but not limited to, a central processing unit (CPU), a microprocessor, a RISC (Reduced Instruction Set Computer) microprocessor, a CISC (Complex Instruction Set Computer) microprocessor, a microcontroller, a system-on-a-chip (SoC), a digital signal processor (DSP), an ASIC, an FPGA, a PLC, a PLA, and so on. In some embodiments, a single integrated device may comprise the computing component 300, the hardware processors 302, and the machine-readable storage media 304. For example, an ASIC, FPGA, SoC, or the like may be the computing component 300, which also includes the hardware processors 302 for executing logic hardwired into the device and / or instructions stored on the storage media 304.

[0029] In exemplary embodiments, the hardware processors 302 (or any other processing unit described herein) are configured to perform the various tasks described in Fig. The two depicted computing units execute, which in turn are configured to provide the corresponding functionality in conjunction with the AI / ML-supported detection of anomalous behavior for a system such as the CDU 100. In particular, the hardware processors 302 can be configured to execute an encoding engine 202, a transformation engine 204, a decoding engine 206, and a reconstruction error determination engine 208. In exemplary embodiments, these machines can form part of the multimetric, time-aware machine learning model 200 (hereafter referred to as ML model 200). ML model 200 can use any suitable type of AI / ML model / algorithm, including, but not limited to, regression-based models (e.g., decision trees, random forests, artificial neural networks, etc.); classification-based models (e.g., support vector machines, Bayesian inference, etc.).); unsupervised learning models such as cluster-based models (e.g., k-means clustering), etc.

[0030] Although the ML Model 200 is shown to be implemented on the customer-specific computer device 200, the ML Model 200 and its components can be implemented in any combination of hardware, software, and / or firmware. In some embodiments, one or more of these machines can be implemented, at least partially, as software and / or firmware modules containing the computer- and machine-executable instructions stored in the machine-readable storage media 304 and executable by the hardware processors 302. It should be appreciated that any description herein of a machine performing a function inherently includes the function that is performed in response to computer- and machine-executable instructions of the machine, which are executed by a processor.

[0031] While the ML model 200 and the associated computing units in Fig. Although the ML Model 200 is shown as separate from the CDU 100 firmware 214 in Figure 2, it can actually be embedded within firmware 214 in some embodiments. More precisely, in some embodiments, the ML Model 200 can be integrated into the existing CDU 100 firmware 214. Regardless of whether the ML Model 200 is independent of or integrated into the CDU firmware 214, the AI ​​capabilities provided by the ML Model 200 reduce the complexity of the sensor instrumentation for the CDU 100 and, consequently, the complexity of firmware 214. As mentioned earlier, this in turn leads to the technical advantage of a lower failure rate for the CDU 100.

[0032] Referring to Fig. 3 in conjunction with Fig. In block 306, machine-executable instructions of the ML model 200 can be executed by the hardware processors 302 to initiate the reception of an input vector X. The input vector X can take the following form: X=[x01,x11,…,xN1,…,x0T,x1T,…,xNT]. This means that the input vector X can contain values ​​acquired for N sensor metrics at different times during a time window of duration T. As mentioned earlier, the N sensor metrics can include one or more groups of correlated sensor metrics that exhibit similar behavior in response to changes in the acquired features.

[0033] In exemplary implementations, the ML model 200 is capable of identifying single or multiple metric point, contextual, and / or collective anomalies associated with the operation of the CDU 100. Some conventional approaches to anomaly detection assume a specific probability distribution of metric values, such as a multivariate Gaussian distribution. With such distributions, if the metric values ​​fall below a certain threshold derived from the distribution parameters, the values ​​are classified as anomalous. However, in the case of CDUs, such assumptions may not be justified. In particular, due to the nature of the physical processes within the CDU 100 that represent the sensor readings, the assumption that the readings follow a Gaussian distribution may not be accurate.For example, pump speed is a categorical metric that takes values ​​from a discrete set of values ​​and therefore may not be adequately characterized by a Gaussian distribution. In examples, the ML model 200 can be any suitable machine learning model / algorithm (e.g., a neural network-based approach like Autocoder) configured to learn the expected behavior of common metrics.

[0034] Further referring to Fig. 3 in conjunction with Fig. In block 308, machine-executable instructions of the coding machine 202 can be executed by the hardware processors 302 to cause the input vector X to be encoded in order to obtain a low-dimensional representation C of the input vector X. In particular, the dimensionality / magnitude of the low-dimensional representation C (given by |C|) can be less than the dimensionality / magnitude of the input vector X (given by |X|), that is, |C| < |X|. The encoding of the input vector X to obtain the encoded low-dimensional representation C may involve a loss of at least some of the information (e.g., metric sensor values) in the input vector X.In exemplary embodiments, machine-executable instructions of the transformation machine 204 can also be executed by the hardware processors 302 in block 308 to initiate a series of one or more nonlinear transformations performed on the input vector X as part of the coding process.

[0035] Furthermore, in some embodiments, values ​​corresponding to one or more sensor metrics can be discarded from the input vector X before encoding. This discarding of values ​​is possible because the ML model 200 enables anomalous behavior detection and a high level of robustness despite the use of a smaller number of sensors. Thus, metric sensor values ​​corresponding to one or more sensors can be discarded from the input vector X while maintaining the ability to perform the desired anomalous behavior detection. In some embodiments, the discarding of sensor measurements may result from the fact that the corresponding sensor(s) are no longer used (e.g., eliminating redundancy for one or more sensors).In other examples, metric sensor values ​​for a particular sensor that is no longer in use may simply be missing from the input vector X. It is important to note that discarding metric sensor values ​​(or the absence of such values) is distinct from the information loss that can occur as a result of encoding the input vector X.

[0036] In block 310, machine-executable instructions of the decoding machine 206 can be executed by the hardware processors 302 to reconstruct the input vector X, in order to produce a reconstructed input vector X R to obtain. The reconstruction of the input vector X can include the decoding of the encoded low-dimensional representation C. In particular, the reconstructed input vector X can R be given by: X R= decode(encode (X)). In exemplary implementations, the ML model 200 can be trained on a dataset containing few or no anomalous data points, and thus the ML model 200 can learn to reconstruct metric values ​​associated with the normal operating behavior of the CDU 100, but it may not be able to correctly reconstruct values ​​for a dataset with a large amount of anomalous data. Due to the nature of the training performed for the ML model 200, as well as the lower dimensionality of the coded representation C with respect to the input vector X and the corresponding information loss in generating the coded representation, the extent to which the reconstructed input vector X R The difference from the original input vector X is directly related to the set of anomalous data in the input vector X.

[0037] In block 312, machine-executable instructions of the reconstruction error determination machine 208 can be executed by the hardware processors 302 to determine a reconstruction error (E) between the input vector X and the reconstructed input vector X. R to determine. In some embodiments, the reconstruction error E can be an error vector defined by E = X R - X is given. More generally, the reconstruction error E can be any quantity representing the extent to which the reconstructed input vector X is inaccurate. R deviates from the original input vector X. As mentioned earlier, the deviation between the reconstructed input vector X R and the original input vector X is larger the more anomalous data is contained in the input vector X.

[0038] In block 314, machine-executable instructions of the transformation engine 204 can be executed by the hardware processors 302 to initiate one or more transformations performed on the reconstruction error E to produce a transformed reconstruction error E T to obtain. In exemplary embodiments, the transformation engine 204 can subject the reconstruction error vector E to one or more brightening transformations in order to obtain the transformed reconstruction error vector E. T to obtain. In some embodiments, the brightening transformation(s) can cause elements of the reconstruction error vector E corresponding to two or more sensor metrics to be included in the transformed reconstruction error vector E. T become uncorrelated. Furthermore, the brightening transformation(s) can lead to elements of the transformed reconstruction error vector E becoming uncorrelated. T exhibit the same variance.

[0039] In block 316, machine-executable instructions of the anomaly detection machine 210 can be executed by the hardware processors 302 to cause an anomaly evaluation to be derived from the transformed reconstruction error vector E. T is determined. In example implementations, the anomaly score for the initial input vector X can be defined as the magnitude of the transformed reconstruction error vector E. T can be determined. In other example implementations, the anomaly score can be any linear combination of the elements of the transformed reconstruction error vector Er.

[0040] In block 318, machine-executable instructions of the anomaly detection machine 210 can be executed by the hardware processors 302 to cause the anomaly rating generated in block 316 to be compared with an anomaly rating threshold to determine whether an anomaly is present. In exemplary embodiments, an anomaly may correspond to multiple sensor metrics exhibiting values ​​indicative of anomalous behavior. However, due to the correlation between multiple sensor metrics, even if the values ​​of several sensor metrics indicate anomalous behavior, a smaller number of sensors / components of the CDU 100 may actually be operating anomalous than the number of sensors that generated the anomalous values.In some embodiments, the anomaly threshold can be determined from a distribution of multi-metric anomaly scores generated from the training dataset used to train the ML model 200. In some embodiments, the distribution of multi-metric scores used to determine the threshold anomaly score can be a chi-square distribution. For example, the anomaly threshold can be a value chosen such that a desired percentage of the values ​​in the distribution of multi-metric anomaly scores falls below the selected value.

[0041] In exemplary embodiments, the relative number of correlated sensors registering anomalous values ​​can indicate whether it is more likely that a CDU (e.g., a flow-regulating component of the CDU) is exhibiting anomalous behavior or that a sensor is exhibiting the anomalous behavior. In certain exemplary embodiments, if at least a minimum number of a group of correlated sensors register values ​​exceeding the anomaly thresholds, it may be more likely that the system (e.g., one or more components of the system failing or otherwise malfunctioning) is exhibiting anomalous behavior than the sensors. Even in example scenarios where a minimum number of correlated sensors register anomalous values, one or more sensors in the group of correlated sensors may exhibit anomalous behavior (e.g., malfunction) at the same time.For example, if in a group of five correlated sensors four of the sensors register values ​​that exceed the respective thresholds, but one sensor does not, this may indicate 1) that a system-level error event is being registered by the four sensors with anomalous values ​​and 2) that the fifth sensor is also experiencing an error event because it is behaving anomalously by not registering the system-level event.

[0042] In some embodiments, a sensor in a group of correlated sensors may exhibit abnormal behavior if 1) it registers a sensor measurement value that exceeds a threshold (or is otherwise outside an expected range for the measured value) and 2) at least a minimum number of other sensors in the group of correlated sensors do not exceed their respective thresholds. In particular, the fact that other sensors in the group of correlated sensors do not register an abnormal event indicates a greater probability that the sensor registering an abnormal value is itself behaving abnormally. For example, if the first sensor in a group of five correlated sensors registers an abnormal value, but three other sensors in the group do not, then the probability that the first sensor is exhibiting abnormal behavior is greater.

[0043] In some examples, a warning can be generated to inform an operator of detected anomalous behavior. The warning message might indicate a system-level fault if at least a minimum number of correlated sensors register values ​​exceeding their respective thresholds. The warning message might also identify one or more potential system-level causes for the observed behavior (e.g., a malfunction in the flow control component). In other embodiments, such as those where a sensor is more likely to exhibit anomalous behavior, the warning message might include a reference to the potentially faulty sensor. Based on the warning message, an operator can verify the sensor, and the values ​​measured by the sensor might be discarded if it is found to be malfunctioning.The sensor can be put back into operation once it has been repaired or replaced.

[0044] In some embodiments, a threshold against which a metric sensor reading is compared to determine whether the measured metric value exceeds the threshold and thus registers an anomaly can be a dynamic value that can change based on one or more other parameters. Specifically, a threshold can be set based on training data provided to a multivariate AI / ML model and can change based on changes in one or more operational parameters reflected in the training data. For example, if the outside temperature fluctuates, the threshold against which the coolant temperature is compared to detect an anomaly can also change. Similarly, if the outside temperature decreases, the expected coolant temperature may also decrease, and in this case, the threshold for an anomaly may also decrease.Conversely, an increase in outside temperature can widen the expected range of coolant temperatures, thereby raising the threshold for the anomaly. Of course, numerous other examples are conceivable.

[0045] Fig. 4A illustrates diagrams of examples of actual sensor measurements taken over time for a CDU A (e.g. CDU 100), as well as corresponding reconstructed measurements and reconstruction errors according to embodiments of the invention. Fig. Figure 4A shows diagrams of the actual values ​​and the corresponding reconstructed values ​​for three example sensor metrics, namely the secondary temperature of the secondary current 106, measured by the thermocouple 130 ( Fig. 1) The primary coolant flow rate and the valve position (valve demand). These sensor measurements can be correlated and exhibit similar behavior with respect to registered events.

[0046] For each sensor metric, the graphs contain the actual values ​​recorded for that metric, as well as the reconstructed values. As illustrated with respect to the primary coolant flow metric, the actual metric values ​​(402) are shown in a slightly darker shade than the reconstructed values ​​(404). Furthermore, the reconstruction error (406), determined with respect to all correlated sensor metrics, is shown below the graphs of the actual and reconstructed values ​​for each sensor metric.

[0047] There are different time periods in which all three sensors, which provide sensor data in relation to the in Fig. The example metrics shown in 4A capture an event that is then accompanied by an anomaly score exceeding an anomaly threshold. In implementation examples, this is interpreted as anomalistic behavior. Such an event 408 is in Fig. Figure 4A illustrates this. Event 408, for example, could correspond to a water control valve on the primary side of CDU A that opens and closes anomalously outside its specified range (referred to as a "valve request"). This, in turn, causes the temperature recorded by thermocouple 130 and the primary coolant flow rate to cycle simultaneously. The anomalous cycling behavior 410 of the water control valve, which in turn causes corresponding cycles in other correlated sensor readings, results in an anomaly score that exceeds the anomaly threshold, thus indicating the presence of an anomaly.

[0048] The anomaly score determined on the basis of the reconstruction error 406 is a multimetric score for all in Fig. The sensor-related measured variables referred to as 4A (secondary flow temperature 106, primary coolant flow rate, valve demand) are multimetric models such as the ML Model 200, which generate multimetric anomaly assessments, offer several technical advantages over their single-metric counterparts. In particular, multimetric models are more robust than single-metric models, generate fewer false alarms, and significantly reduce the amount of information that data center operators need to process. Furthermore, multimetric models are able to exploit the correlation between sensors within a group of correlated sensors. For example, if one of the measured variables in Fig. If 4A had been removed from the multimetric analysis, the ML model 200 would still have indicated the anomalous behavior. This demonstrates that even if a sensor malfunctions, the abnormal behavior associated with event 408 would still be displayed. For example, even if the valve request sensor were to fail, the secondary flow temperature 106 and the primary coolant flow rate would still exhibit abnormal cyclic behavior, and the reconstruction error would still deviate from the anomaly threshold to indicate the presence of an anomaly. Therefore, using the ML model 200 for abnormal behavior detection allows for the use of fewer sensors and less expensive instruments while maintaining a high level of system stability.

[0049] It should be noted that while the anomaly values ​​(as determined from reconstruction error 406) exceed the anomaly threshold during part of the period corresponding to event 408, thus exhibiting anomalous behavior 410, this does not necessarily mean that multiple sensors or multiple other components of CDU A are malfunctioning. For example, while in the case of event 408 each of the example sensors produced anomalous data for the corresponding sensor metrics, the only component of CDU 100 that malfunctioned was the primary-side water control valve. However, because the valve request metric for the water control valve is correlated with several other sensor metrics, anomalistic data can also be observed in conjunction with other sensor metrics, even though the sensors collecting data with respect to those other metrics are functioning correctly.In some embodiments, the detection of anomalous behavior in a group of correlated sensor readings may require further investigation of the sensor / component that is malfunctioning and causing the other sensor readings to exhibit anomalous behavior. If a sensor is malfunctioning, its data can be replaced with historical or predicted values ​​before being fed into the ML model 200.

[0050] Another example of the ability of the multivariate ML model 200 to improve the resilience of the CDU is now presented with reference to the Fig. 4A and Fig. 4B described. Another example of an event 412 in Fig. 4A is the shutdown of a server or IT rack cooled by CDU A for a maintenance event. This shutdown event 412 can lead to anomalous data for the various components in Fig. The example sensor metrics shown in 4A ultimately lead to an anomaly evaluation that exceeds the anomaly threshold and thus indicates anomalous behavior 414. In exemplary embodiments, the event 412 can be an event at the plant level.

[0051] Fig. Figure 4B shows diagrams of the actual sensor readings and the corresponding reconstructed values ​​and reconstruction errors for the same example set of sensor readings as in Figure 4B. Fig. 4A is shown, but for a different CDU (CDU B). CDU B may, for example, be in the same cluster as CDU A. CDU A and CDU B may experience the same event at facility level 412 and run the same or different multivariate anomaly detection models. As shown in Fig. As shown in Figure 4B, the reconstruction error during a period 416, which overlaps with the period of event 412, exhibits anomalous behavior 418, which is similar to anomalous behavior 414. Fig. 4A and Fig. Figure 4B demonstrates that a multivariate model deployed across two or more CDUs is capable of detecting plant-level anomalies. This provides the technical advantage of achieving an additional level of sensor health verification. For example, if plant-level anomalous behavior is observed by four CDUs within a group of five, and the same plant-level behavior is not observed by CDU 5, then we may be able to determine that a sensor failure has occurred in CDU 5, as it is the only CDU that failed to detect the plant-level anomalous behavior. A further technical advantage of using the ML model 200 for anomalous behavior detection is that the same ML model 200 can be used not only for a specific CDU (e.g., CDU A) but also for other CDUs within the same cluster (e.g., CDU B).Thus, the ML model 200 can be used to improve the reliability of both individual and cluster-wide CDU deployments, while also enabling simplified CDU instrumentation and associated firmware.

[0052] Fig. Figure 5 shows a block diagram of an exemplary computer system 500, in which various embodiments of the system described here can be implemented. The computer system 500 comprises a bus 502 or other communication mechanism for transmitting information, and one or more hardware processors 504 connected to the bus 502 for processing information. The hardware processor(s) 504 can, for example, be one or more general-purpose microprocessors.

[0053] The Computer System 500 also includes a main memory 506, such as random access memory (RAM), a cache, and / or other dynamic memory devices connected to the 502 bus to store information and instructions to be executed by the 504 processor. The main memory 506 can also be used to store temporary variables or other intermediate information during the execution of instructions to be carried out by the 504 processor. When such instructions are stored in memory media accessible to the 504 processor, the Computer System 500 becomes a specialized machine adapted to perform the operations specified in the instructions.

[0054] The Computer System 500 also includes a read-only memory (ROM) 508 or other static storage device connected to the bus 502 to store static information and instructions for the processor 504. A storage device 510, such as a magnetic disk, an optical disk, or a USB flash drive, etc., is provided and connected to the bus 502 to store information and instructions.

[0055] The computer system 500 can be connected via bus 502 to a display 512, such as a liquid crystal display (LCD) (or a touchscreen), to show information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to bus 502 to transmit information and command selections to the processor 504. Another type of user input device is the cursor control 516, such as a mouse, trackball, or cursor direction keys, for transmitting directional information and command selections to the processor 504 and for controlling cursor movement on the display 512. In some embodiments, the same directional information and command selections as with cursor control can be implemented by receiving touch inputs on a touchscreen without a cursor.

[0056] The Computer System 500 can include a user interface module for implementing a graphical user interface, which can be stored on a mass storage device as executable software code that is executed by the computer device(s). This and other modules can include components such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0057] In general, the terms "component," "engine," "system," "database," "data store," and the like, as used here, can refer to logic embodied in hardware or firmware, or to a collection of software instructions that may have entry and exit points and are written in a programming language such as Java, C, or C++. A software component may be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language such as BASIC, Perl, or Python. It is clear that software components can be called by other components or by themselves, and / or invoked in response to detected events or interruptions. Software components configured to run on computer devices may be stored on a computer-readable medium, such as...Software may be provided on a compact disc, digital video disc, flash drive, magnetic disk, or other tangible medium, or as a digital download (and may initially be stored in a compressed or installable format that requires installation, decompression, or decryption before execution). Such software code may be stored partially or entirely in memory on the executing computer device for execution by the computer device. Software instructions may be embedded in firmware, such as an EPROM. Furthermore, the hardware components may consist of interconnected logic units such as gates and flip-flops, and / or programmable units such as programmable gate arrays or processors.

[0058] The computer system 500 can implement the techniques described herein using custom hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic, which, in combination with the computer system, make the computer system 500 a specialized machine or program it. According to one embodiment, the techniques described herein are executed by the computer system 500 in response to the processor(s) 504 executing one or more sequences of instructions contained in the main memory 506. Such instructions may be read into the main memory 506 from another storage medium, such as a storage device 510. The execution of the instruction sequences contained in the main memory 506 causes the processor(s) 504 to perform the process steps described herein.In alternative embodiments, hard-wired circuits can be used instead of, or in combination with, software instructions.

[0059] The term "non-transitory media" and similar terms such as machine-readable storage media, as used here, refer to any media that store data and / or instructions that cause a machine to operate in a particular way. Such non-transmittable media can include non-volatile and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as Storage Device 510. Volatile media include dynamic storage, such as Main Memory 506. Common forms of non-volatile media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tapes or other magnetic data storage media, CD-ROMs, other optical data storage media, physical media with hole patterns, RAM, PROM and EPROM, FLASH-EPROM, NVRAM, other memory chips or cartridges, and their networked versions.

[0060] Non-transitory media differ from transmission media but can be used in conjunction with them. Transmission media are involved in the transfer of information between non-transitory media. Examples of transmission media include coaxial cable, copper and fiber optic cables, including the wires that make up the 502 bus. Transmission media can also take the form of sound or light waves, such as those generated in radio and infrared data communication.

[0061] The Computer System 500 also includes a Communications Interface 518, which is connected to the Bus 502. The Communications Interface 518 provides a bidirectional data communication connection to one or more network connections that are connected to one or more local area networks (LANs). For example, the Communications Interface 518 could be an ISDN (Integrated Services Digital Network) card, a cable modem, a satellite modem, or a modem to establish a data communication connection to a corresponding type of telephone line. Another example: The Communications Interface 518 could be a LAN (Local Area Network) card to establish a data communication connection to a compatible LAN (or a WAN component for communication with a WAN). Wireless connections can also be implemented.In each of these implementations, the 518 communication interface sends and receives electrical, electromagnetic, or optical signals that transmit digital data streams representing various types of information.

[0062] A network connection typically enables data communication over one or more networks to other data devices. For example, a network connection might establish a connection over a local area network to a host computer or to data devices operated by an Internet service provider (ISP). The ISP, in turn, provides data communication services over the worldwide packet data communication network, commonly known today as the "Internet." Both the local area network and the Internet use electrical, electromagnetic, or optical signals to transmit digital data streams. The signals across the various networks, the signals on the network connection, and the communication interface 518, which transmit digital data to and from the computer system 500, are examples of transmission media.

[0063] The Computer System 500 can send messages and receive data, including program code, via the network(s), network connection, and communication interface 518. In the Internet example, a server could transmit requested code for an application program via the Internet, the ISP, the local network, and communication interface 518.

[0064] The received code can be executed by the 504 processor as soon as it is received, and / or stored in the 510 memory device or other non-volatile memory for later execution.

[0065] Each of the processes, methods, and algorithms described in the preceding sections can be embodied in code components and fully or partially automated by them, which are executed by one or more computer systems or computer processors with computer hardware. The one or more computer systems or computer processors can also be operated in such a way as to support the execution of the corresponding operations in a cloud computing environment or as Software as a Service (SaaS). The processes and algorithms can be partially or fully implemented in application-specific circuits. The various features and procedures described above can be used independently or combined in various ways.Various combinations and subcombinations are said to fall within the scope of this disclosure, and certain procedural or process blocks may be omitted in some implementations. The methods and processes described herein are also not restricted to any particular order, and the associated blocks or states may be executed in other suitable orders, in parallel, or otherwise. Blocks or states may be added to or removed from the disclosed examples. The execution of certain operations or processes may be distributed across computer systems or computer processors, not just on a single machine, but distributed across a number of machines.

[0066] A circuit can be implemented in any form of hardware, software, or a combination thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logic components, software routines, or other mechanisms can be implemented to form a circuit. In implementation, the various circuits described herein can be implemented as discrete circuits, or the described functions and features can be partially or completely distributed across one or more circuits. Even if various features or functional elements are individually described or claimed as separate circuits, these features and functions can be shared by one or more common circuits, and such a description is not intended to require or imply that separate circuits are necessary to implement these features or functions.If a circuit is implemented wholly or partially in software, this software can be implemented in such a way that it works with a computer or processing system capable of performing the functionality described in relation to it, such as the Computer System 500.

Claims

[1] Computer-implemented method comprising: Receiving, as input to a machine learning model, sensor data collected over a time window from a plurality of sensors configured to monitor an operating behavior of a machine, the sensor data comprising a corresponding plurality of values ​​for each of a plurality of sensor metrics; identifying a set of two or more sensor metrics from the plurality of sensor metrics that are correlated; Encoding the sensor data using the machine learning model to obtain an encoded representation of the sensor data; and Reconstructing the sensor data using the machine learning model from the encoded representation to obtain reconstructed sensor data, and Determining, using the machine learning model, whether the machine or at least one sensor is more likely to exhibit abnormal behavior based at least in part on a number of the two or more correlated sensor metrics having corresponding values ​​that exceed corresponding thresholds and a difference between the reconstructed sensor data and the sensor data. [2] The computer-implemented method of claim 1, further comprising: Initiate one or more corrective actions to change the operating behavior of the machine in response to the abnormal behavior. [3] A computer-implemented method according to claim 1, wherein it is determined whether the machine or the at least one sensor is more likely to exhibit the anomalous behavior, comprising: Determining that at least one correlated sensor measurement exceeds a corresponding threshold; Determine that at least a minimum number of the correlated sensor metrics do not exceed the respective thresholds; and Determination that at least one sensor is more likely to exhibit the anomalous behavior than the machine. [4] The computer-implemented method of claim 1, wherein determining whether the machine or the at least one sensor is more likely to exhibit the anomalous behavior comprises: Determine that at least a minimum number of the two or more correlated sensor metrics exceeds the respective thresholds; and the finding that the machine is more likely to exhibit the abnormal behavior than the respective sensor. [5] The computer-implemented method of claim 1, further comprising: Determining, using the machine learning model, a reconstruction error based on the difference between the reconstructed sensor data and the sensor data; and Determining an anomaly score from the reconstruction error using the machine learning model. [6] The computer-implemented method of claim 5, wherein the reconstruction error is an error vector, and wherein determining the anomaly score comprises determining a magnitude of the error vector. [7] A computer-implemented method according to claim 5, further comprising: Determine that the anomaly value exceeds a threshold; and Determining that one of the machines or the at least one sensor exhibits the anomalous behavior based at least in part on the anomaly value exceeding the threshold. [8] The computer-implemented method of claim 5, wherein the two or more correlated sensor metrics comprise a first sensor metric and a second sensor metric, and wherein the reconstruction error comprises a first set of values ​​corresponding to the first sensor metric and a second set of values ​​corresponding to the second sensor metric, the method further comprising: Performing a transformation of the reconstruction error that causes the first set of values ​​to become uncorrelated with the second set of values. [9] The computer-implemented method of claim 1, further comprising: Determining that the at least one sensor exhibits the anomalous behavior; and Discarding the respective plurality of values ​​generated by the at least one sensor and corresponding to at least one of the plurality of sensor metrics from the sensor data before providing the sensor data as input to the machine learning model. [10] The computer-implemented method of claim 9, wherein the discarding occurs in response to the at least one sensor no longer being configured to monitor the operating behavior of the machine. [11] Machine learning system comprising: a plurality of sensors configured to monitor system operating behavior; a memory that stores machine-executable instructions; and a processor configured to access the memory and execute the machine-executable instructions to: receive, by a machine learning model, sensor data collected from the plurality of sensors over a time window, wherein the sensor data relates to a plurality of correlated sensor metrics; encode the sensor data using the machine learning model to obtain encoded sensor data; decode the encoded sensor data using the machine learning model to obtain reconstructed sensor data; and Determining, using the machine learning model, whether the system or at least one sensor is more likely to exhibit abnormal behavior based at least in part on a number of the plurality of correlated sensor metrics having corresponding values ​​that exceed corresponding thresholds during the time window and a difference between the reconstructed sensor data and the sensor data. [12] The system of claim 11, wherein the processor is configured to determine whether the machine or the at least one sensor exhibits the anomalous behavior by executing the computer-executable instructions to: determine that at least one correlated sensor measurement exceeds a corresponding threshold; determine that at least a minimum number of the correlated sensor metrics do not exceed the respective thresholds; and determine that at least one sensor is more likely to exhibit the anomalous behavior than the machine. [13] The system of claim 11, wherein the processor is configured to determine whether the machine or the at least one sensor exhibits the anomalous behavior by executing the computer-executable instructions to: determine that at least a minimum number of the two or more correlated sensor metrics exceed the respective thresholds; and determine that the machine is more likely to exhibit the anomalous behavior than at least one sensor. [14] The system of claim 11, wherein the processor is configured to encode the sensor data to obtain the encoded sensor data by executing the machine-executable instructions to: Applying one or more nonlinear transformations to the input sensor data to obtain the encoded sensor data, wherein the encoded sensor data is a low-dimensional representation of the sensor data. [15] The system of claim 11, wherein the processor is further configured to execute the machine-executable instructions to: to determine a reconstruction error based on a difference between the reconstructed sensor data and the sensor data; and to determine an anomaly score from the reconstruction error. [16] The system of claim 15, wherein the processor is further configured to execute the computer-executable instructions to: determine that the anomaly value exceeds a threshold; and determine that the system or the at least one sensor exhibits the anomalous behavior based at least in part on the anomaly score exceeding the threshold. [17] Computer program product comprising a non-transitory computer-readable medium storing program instructions which, when executed by a processor, cause operations to be performed which include: Receiving, by a machine learning model, an input vector comprising sensor measurements acquired by a plurality of sensors monitoring an operating condition of a machine, wherein the sensor measurements relate to a plurality of correlated sensor metrics; Encoding the sensor data using the machine learning model to obtain an encoded representation of the sensor data; and Reconstructing the sensor data using the machine learning model from the encoded representation to obtain reconstructed sensor data and Determining, using the machine learning model, whether the machine or at least one sensor is more likely to exhibit abnormal behavior based at least in part on a number of sensor measurements related to the plurality of correlated sensor metrics that exceed corresponding thresholds for the plurality of correlated sensor metrics and a difference between the reconstructed sensor data and the sensor data. [18] The computer program product of claim 17, wherein determining whether the machine or the at least one sensor is more likely to exhibit the anomalous behavior comprises: Determining that at least one threshold number of the plurality of correlated sensor metrics deviates from the respective expected values ​​by more than the respective threshold values; and Determination that the machine is more likely to exhibit the anomalous behavior than at least one sensor.

Images