Recommendation generation using machine learning data validation

Abstract

Techniques are disclosed for using validated sensor data by a machine learning model to generate recommendations for repairing issues in a monitored system. The machine learning model is trained to identify correlations between sensors for the monitored system. Upon receiving current sensor data, the machine learning model identifies a subset of the current sensor data that cannot be validated. The system generates estimated values ​​for the sensor data that cannot be validated based on the learned correlations between the sensor values. The system generates recommendations for repairing issues in the monitored system based on the validated sensor values ​​and the estimated sensor values.

Description

[Technical Field]

[0001] Incorporation by Reference, Disclaimer The subsequent application, Application No. 17 / 455,536, filed November 18, 2021, is hereby incorporated by reference. Applicant hereby cancels any disclaimer of claim scope in the parent application or its prosecution history, and notifies the USPTO that the claims in this application may be broader than any claims in the parent application.

[0002] Technical Field The present disclosure relates to applying machine learning models to generated sensor data to generate recommendations for remediating detected issues using filtered datasets of the sensor data. In particular, the present disclosure relates to using machine learning validated and estimated sensor values ​​to generate recommendations for remediating issues. [Background technology]

[0003] background When an anomaly occurs in a monitored system, the anomaly may be the result of a failure in the monitored component or a failure in the sensor performing the monitoring. If a fault repair system initiates a repair action based on a detected anomaly that is the result of a faulty sensor, the repair action may fail to address the actual fault in the system. In a worst-case scenario, the repair action may exacerbate the fault, leading to system failure.

[0004] Organizations may utilize human oversight of computer-based monitoring and repair systems to verify computer results and perform actions that computers cannot perform. For example, in one environment, repair actions may require a human to physically manipulate an object, such as a valve, lever, or switch, or to physically observe a component. In another environment, an organization may require human confirmation of computer-generated data. For example, a sensor may indicate that a component is at a certain temperature. The organization may require a human-generated temperature measurement to confirm the temperature reading. A human may also determine whether any physical defects exist that could result in a faulty temperature reading by the sensor. In other environments, such as autonomous vehicles or computer-monitored data centers, both monitoring and repair functions may be capable of being performed by the computer without human intervention. A human may be required in the rare event of a computer failure. However, if a human is not part of the normal observation and repair process, human attention may be distracted and the human may not effectively monitor computer execution. Thus, an organization may require that certain monitoring and remediation tasks be performed by a human, even if a computer can perform the tasks without human intervention.

[0005] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

[0006] Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings. It should be noted that references to "one" or "an" embodiment in this disclosure are not necessarily to the same embodiment, but that such references mean at least one. In the drawings: [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates a system according to one or more embodiments. [Figure 2A] FIG. 1 illustrates an example set of operations for generating a task list based on machine learning validated data, according to one or more embodiments. [Figure 2B] FIG. 1 illustrates an example set of operations for generating a task list based on machine learning validated data, according to one or more embodiments. [Figure 3] FIG. 1 illustrates an example set of tasks, according to one or more embodiments. [Figure 4A] FIG. 1 illustrates an example of training a machine learning model, according to one or more embodiments. [Figure 4B] FIG. 1 illustrates another example of training a machine learning model, according to one or more embodiments. [Figure 5A] FIG. 1 illustrates an example embodiment of generating a task list based on data validated by machine learning. [Figure 5B] FIG. 1 illustrates an example embodiment of generating a task list based on data validated by machine learning. [Figure 6] FIG. 1 shows a block diagram illustrating a computer system according to one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0008] Detailed Description In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding. One or more embodiments may be practiced without these specific details. Features described in one embodiment may be combined with features described in a different embodiment. In some instances, well-known structures and devices are described with reference to block diagram form in order to avoid unnecessarily obscuring the present invention.

[0009] 1. Overall Overview 2. System Architecture 3. Repairing system anomalies based on machine learning 4. Training a machine learning model 5. Example Embodiments 6. Computer Networks and Cloud Networks 7. Miscellaneous, Extensions 8. Hardware Overview 1. Overall Overview One or more embodiments filter out a subset of current sensor data that cannot be validated based on historical sensor data to generate filtered sensor data. Optionally, the system may generate and add estimated sensor data to the filtered sensor data in place of the filtered subset of sensor data. The system analyzes the filtered sensor data to identify anomalous events and / or generate a task list for remediating the anomalous events.

[0010] One or more embodiments train a machine learning model based on historical sensor data to validate a target set of sensor data. The system uses the set of historical sensor data to train the machine learning model to learn correlations between sensors. Upon receiving current sensor data, the machine learning model identifies a subset of sensor data from the sensor that cannot be validated. For example, a sensor may be in a faulty state and not transmitting data. Another sensor may be drifting, causing the sensor to transmit data, but the data is inaccurate. The system uses the learned correlations between sensors to estimate values ​​for the sensors that cannot be validated. The system uses a filtered dataset including the validated sensor data and estimated sensor values ​​for the sensor data that could not be validated to generate recommendations for repairing faults in the monitored system.

[0011] One or more embodiments train a machine learning model based on historical sensor data to generate estimated sensor data. The system uses a set of historical sensor data to train the machine learning model to learn correlations between sensors. Upon receiving current sensor data, the machine learning model predicts values ​​for the sensor values ​​based on correlations with the values ​​of the remaining sensor values. The machine learning model generates estimated sensor data based on the predicted values ​​for the sensors.

[0012] The sensor-based monitoring system includes one or both of machine learning-based and human-based sensor validation and fault remediation, where the machine learning model is trained to identify correlations between sensor data values ​​of the sensors monitoring the system.

[0013] A sensor-based monitoring system may generate a task list for repairing one or more detected faults. The system selects tasks for the task list based on a filtered dataset including validated sensor data and estimated sensor values. The task list may include computer-executed tasks and human-executed tasks. A task specifies the entity that should perform the task, the system component on which the task should be performed, the action to be performed or carried out, any dependencies of the task on any other tasks, and any parameters required to perform the task. The parameters include the validated sensor data. For example, the system may select one task to be performed if a sensor generates a first temperature measurement and another task to be performed if the sensor generates a second temperature measurement. The system specifies the parameters in the selected task as the generated temperature measurements. If the system cannot validate a parameter for a task, such as when a required sensor value is missing, the system inputs an estimated sensor value generated by a machine learning model into the parameter based on a correlation between the missing sensor value and one or more validated sensor values.

[0014] In one embodiment, the machine learning model is a deterministic machine learning model. For example, the machine learning model may be a multivariate state estimation technique (MSET) model. The MSET model is trained using a set of sensor values ​​that represent a range of non-anomalous values ​​for the sensor. The MSET model identifies correlations between the sensor values. In operation, the MSET model obtains current sensor values, compares the current sensor values ​​to predicted sensor values, and identifies anomalies between the sensor values. Additionally, the MSET model determines whether the abnormal sensor values ​​are the result of a faulty sensor or a fault in the monitored component based on the identified correlations between the sensor values. The MSET model uses a filtered dataset including the non-anomalous sensor values ​​and the estimated sensor values ​​to identify a root cause or condition that is responsible for the detected anomaly in the sensor data. The MSET model uses the filtered dataset including the non-anomalous sensor values ​​and the estimated sensor values ​​to further instantiate task parameters for one or more tasks in a task list to remediate the root cause or condition that is responsible for the detected anomaly.

[0015] In one or more embodiments, the system receives input to modify a task in a task list. For example, the system may receive input to (a) add or remove a task from the task list that is associated with a particular root cause or condition, (b) add or remove a parameter associated with a task in the task list, or (3) modify one or more attributes associated with a task, such as the entity on which the task should be performed.

[0016] In one embodiment, the system generates a task list by selecting two or more task templates from a set of stored task templates. The system queries the set of stored task templates based on identified root causes or conditions underlying one or more detected anomalies in the monitored system. The system inputs specific data associated with the identified root causes or conditions into a query template. For example, the query may return specific task templates associated with the identified conditions. The system further identifies specific entities that match entity descriptions in the task templates, specific components that match component descriptions in the task templates, and sensor values ​​that match parameter descriptions in the task templates.

[0017] The system arranges tasks in a task list according to a set of rules. The set of rules may specify the relative priority of the tasks and the dependency of one task on another. For example, a series of three tasks may need to be executed consecutively, in order, within a predetermined period of time. According to another example, a task may specify two actions that must be performed simultaneously by two separate entities. A task may specify whether one task may be started before another and whether one task may be completed before another. One task in a task list may be given the highest priority, indicating that it must be executed before any other task. For example, a task list guiding a human through a process to detect voltage in a component may begin with a task to disconnect the component from the power source. A second-highest priority task may, in turn, depend on the first task and instruct the human to drain stored power in the component before starting a sequence of tasks to take measurements on the component.

[0018] In one embodiment, the system may determine that one or more sensor values ​​may not be validated and may even not be estimated. Thus, the system may generate a human-readable explanation indicating that certain data is required to generate a task list, but that the data is unavailable. The system may receive input to initiate the associated task list. Alternatively, a user may manually provide the required data via a graphical user interface (GUI) to enable the system to generate a task list.

[0019] In one embodiment, the system tracks whether a task is completed. The system may generate a user interface including visual elements representing a series of tasks. When the system detects a completed task, the system may modify the visual elements of the GUI to indicate that the task is completed. The system may detect the completion of both computer-performed and human-performed tasks. Alternatively, the system may receive human-generated input to indicate when a human-performed task is completed. In one or more embodiments, the system may update or modify a task list based on data obtained during the execution of one or more tasks. For example, an initial task list may include a series of tasks to replace a system component. In an intermediate task, the system may detect a human-generated measurement indicating that a subcomponent has a value outside a safe operating range. The system may generate a notification to abort execution of the initial task list. The system may even generate a new task list to shut off power to the subcomponent. The new task list may replace the initial task list. Alternatively, the new task list may be added to the initial task list.

[0020] One or more embodiments described herein and / or recited in the claims may not be included in this General Overview section.

[0021] 2. System Architecture FIG. 1 illustrates a system 100 according to one or more embodiments. As illustrated in FIG. 1, the system 100 includes a monitored system 102 and a system monitor 110. The monitored system 102 may include any type of system having any number of components. Examples of monitored systems include vehicles, data centers, computer networks, manufacturing facilities, medical facilities, and power generation facilities. The monitored system 102 may include any number and types of components, including structural components, motion components, power generation components, computational components, and data storage components. The components of the monitored system 102 are monitored by sensors 103. The sensors include physical sensors and software that monitors computing elements. For example, physical motion elements may be monitored by temperature sensors and piezoelectric sensors. Data processing components may be monitored by temperature sensors and applications that track data traffic to and from the data processing components. Medical devices may be monitored by location sensors. Vehicles may be monitored by global positioning system (GPS) sensors, altimeters, and speedometers. In one or more embodiments, sensors 103 include tens, hundreds, or thousands of sensors. For example, the number and types of sensors may be sufficient to allow a machine learning model to identify correlations between different sensors monitoring the same system component.

[0022] The system monitor 110 includes a sensor analytics machine learning model engine 111. The sensor analytics machine learning model engine 111 applies one or more machine learning models to sensor data obtained from sensors 103 monitoring the monitored system 102. The machine learning models may be probabilistic-type machine learning models, such as neural networks. Alternatively, the machine learning models may be deterministic-type machine learning models, such as multivariate state estimation techniques (MSET) models. Probabilistic-type machine learning models apply probabilistic methods to generate predictions. In contrast, deterministic-type machine learning models implement mathematical structures that, for a given set of input values, will return the same set of output values.

[0023] The sensor analytics machine learning engine 111 trains a machine learning model using historical sensor data 117 stored in the data repository 116. The sensor analytics machine learning engine 111 trains the machine learning model to identify correlations between sensor values. In one embodiment, the trained machine learning model receives current sensor data and generates a set of validated sensor values. The validated sensor values ​​may correspond to sensor values ​​that meet predefined criteria. For example, the validated sensor values ​​may fall within a range of expected values ​​for the sensor. The trained machine learning model may further identify a set of unvalidated sensor values. The unvalidated sensor values ​​may include values ​​that are outside a predicted range of values ​​for a particular sensor. Alternatively, the unvalidated sensor values.

[0024] In one embodiment, in which the machine learning model is a deterministic model such as the MSET model, the machine learning model may generate, for each sensor value, a delta value representing the difference between the current sensor value and the predicted value for the sensor. The MSET model may indicate an anomaly in a sensor value if (a) the sensor value is outside a range of non-anomalous values ​​for the particular sensor, or (b) the sensor value varies from the predicted value by more than a predetermined threshold. For example, the MSET model may identify a correlation between the values ​​of three sensors. The MSET model may determine that a third sensor should have a particular value based on the values ​​of two of the sensors. The MSET model may detect an anomaly if the current value for the third sensor varies from the particular value by more than a predefined threshold. Additionally, the MSET model may determine that the third sensor has a predefined range of non-anomalous operating values. The MSET model may detect an anomaly if the current sensor value for the third sensor is outside the predefined range of non-anomalous operating values.

[0025] The event detection engine 112 detects whether a triggering event has occurred in the system. The triggering event may be an event detected based on current values ​​of the sensors 103. Alternatively, the triggering event may be a manually triggered event intended to initiate a sequence of tasks associated with the monitored system 102. The event detection engine 112 monitors the output from the machine learning model and one or both of the sensors 103 to determine whether an anomaly in the sensor values ​​corresponds to a sensor-based triggering event in the monitored system 102. For example, one combination of anomalous sensor values ​​may correspond to an overheating-type triggering event. Another combination may correspond to a power failure-type triggering event. Yet another combination may correspond to a data communication failure-type triggering event.

[0026] The causal analysis engine 113 analyzes output from one or both of the machine learning model and the sensors 103 to identify the root cause of the triggering event. For example, if the event detection engine 112 detects sensor data associated with an overheating component, the causal analysis engine 113 may analyze the sensor data and metadata describing the component to identify a failing power adapter in the component as the cause of the overheating event. The causal analysis engine 113 may analyze metadata describing the component to identify, for example, the component type and component specifications. According to another example, if the event detection engine 112 detects a data communication failure event, the causal analysis engine 113 may analyze sensor data from a set of components associated with the communication failure to determine that a router is causing a data latency rate that exceeds a non-abnormal data latency rate. If the causal analysis engine 113 determines that the triggering event was a manually triggered event, the causal analysis engine 113 may identify a reason associated with the manually triggered event. For example, a user may interact with the user interface 120 to initiate a power shutdown for a facility based on an emergency situation that may not be detected by the sensors 103. The causal analysis engine 113 determines that the root cause of the event is a manually initiated power shutdown.

[0027] The task list generation engine 114 generates a task list for repairing the identified root cause of the triggering event. The task list generation engine 114 queries the body of task templates 118 stored in the data repository 116 to identify a task template 118 associated with the identified root cause. FIG. 3 illustrates example task templates 301 and 307 according to one embodiment. The task list generation engine 114 may select a set of tasks associated with a particular root cause based on one or more of: (1) the identified root cause of the event; (2) the output from the sensor analytics machine learning model; and (3) the output from the sensor 103. For example, the task list generation engine 114 may generate a set of queries having the following conditions: (a) a power level above a threshold; (b) a particular power adapter; (c) providing power to a particular component; and (d) receiving power from a particular supply. The set of queries may return a corresponding set of tasks selected to be performed for the specified conditions. In one embodiment, the set of tasks is stored as a task list 119. For example, if the same root cause has occurred previously, the previously generated set of tasks may be stored as a task list 119. Additionally, if a user modifies the set of tasks associated with a particular root cause, the modifications may be stored in the task list 119.

[0028] The task list generation engine 114 populates the task template 118 based on specific information associated with the monitored system 102. Referring to FIG. 3 , a task 301 includes fields for a system component 302, an action to be performed 303, an entity to perform the action 304, the task's relationship to other tasks 305, and a parameter value 306 corresponding to one or both of a measured sensor value and an output from a sensor analytics machine learning model. The task list generation engine 114 populates the system component field 302 with the component in the monitored system 102 that should be acted upon by the entity. For example, a set of tasks associated with an overheating power adapter may include tasks that specify a power supply as the system component 302, flip a power switch as the action to be performed 303, and specify a particular operator as the entity to perform the action 304. The task list generation engine 114 may populate the "system component" field 302 with a description of the component to be acted upon. For example, the task list generation engine 114 may reference the rack containing the power supply and the specifications of the power adapter to enter the data “Power supply M103 located on the lower right side of the server rack” into the “System Components” field 302.

[0029] The task list generation engine 114 may enter the name or role of a particular individual responsible for the action in the "Entity to perform action" field 304. For example, the task list generation engine 114 may access an organizational roles table to identify "D. Scott" as the engineer responsible for servicing overheating power adapters. The task list generation engine 114 may enter the name "D. Scott" or the title "Engineer" in the "Entity to perform action" field 304. In one or more embodiments, some tasks in a task list may be performed by a human operator, and other tasks may be performed by a computer without human intervention. For example, the task template 301 may identify the computer "COMP-1" as performing a data analysis task subsequent to a user performing a previous task in the task list.

[0030] The task template 301 further identifies each task's relationships 305 to other tasks. For example, the task template 301 may indicate that the current task must be performed immediately after task A, without any intervening tasks. The task template 301 may further specify that the current task must be performed before task C, although one or more tasks may intervene between the current task and task C. The task template 301 may further identify the chain or family of tasks to which the particular template belongs.

[0031] 3 , task template 307 may specify a complex task that may necessarily include two or more actions 308 and 309 to be performed simultaneously. Task template 307 may further specify that the two or more actions 308 and 309 are performed by one or more entities. For example, complex task template 307 may specify pressing one button and, while the first button is pressed, pressing a second button. Complex task template 307 may specify pressing a button on one system component by one operator and, while the first button is pressed, a measurement by another operator on another system component.

[0032] The task template 301 specifies a set of parameters 306 required to perform a task. For example, the task template 301 may specify that if a temperature sensor value is within a first range, the entity should perform a first action. The task template 301 may further specify that if the temperature sensor value is within a second range, the entity should perform a second action. The action performed may depend on parameters 306 corresponding to one or both of an output from the sensor 103 and an output from a sensor analytics machine learning model. The parameters 306 may be direct measurements from the sensor 103 and values ​​derived from the measurements. For example, a data system monitor may measure the latency of a data flow associated with a component. The parameters 306 may be a measure of average packet delay based on the measured latency.

[0033] In one embodiment, the task list generation engine 114 determines whether validated sensor data exists for input into the parameter field 306 of the task template 301. For example, validated sensor data may include sensor data determined to be within a non-anomalous range of values ​​for a sensor. If the validated sensor data exists, the task list generation engine 114 inputs the validated sensor data into the parameter field 306. If the task list generation engine 114 determines that validated sensor data does not exist for a particular parameter, the task list generation engine 114 may determine whether estimated sensor data exists for the particular parameter. For example, a sensor analytics machine learning model may estimate an unvalidated sensor value based on learned correspondences between a sensor with an unvalidated value and other sensors. If an estimated sensor value exists for the parameter, the task list generation engine 114 inputs the estimated sensor value into the parameter field 306. If neither validated nor estimated sensor values ​​exist for parameters associated with a task template, the task list generation engine 114 may generate a human-readable explanation indicating that the task cannot be performed and the data necessary to complete the task. Additionally, or alternatively, the task list generation engine 114 may determine whether one or more alternative tasks may be substituted for the task that cannot be performed. For example, the task list generation engine 114 may apply a set of rules specifying that if a task cannot be performed, one or more alternative tasks may be executed to infer data about the task that cannot be performed. According to one example, a task template may require specific attention of a component by an operator when a particular parameter has a particular value. However, the value of a particular parameter may not be amenable to being validated or estimated.Thus, the task list generation engine 114 may substitute two alternative tasks for a task having parameters that cannot be validated or estimated. The two alternative tasks may include measurements of associated components based on parameters that may be validated or estimated. Values ​​for parameters that cannot be validated or estimated may be inferred from the two alternative tasks. Alternatively, the alternative tasks may achieve a desired result without inferring values ​​for parameters that cannot be validated or estimated.

[0034] In one embodiment, a human expert and / or operator generates the task templates 118. In addition, the task list generation engine 114 may implement a machine learning model to identify existing task templates to associate with the identified root cause. Additionally, or alternatively, the task list generation engine 114 may include a machine learning model to generate new task templates based on newly identified root causes. For example, the task generation machine learning model may be trained with a dataset including (a) the root cause of the triggering event and (b) a set of task templates associated with the root cause (including components, actions, entities, dependencies, and parameters associated with the task templates). Upon receiving a current root cause with specific characteristics, including specific components and event types, the task generation machine learning model may generate a new task template associated with the specific components and event types, which may specify the task to be performed, the type of entity on which the task should be performed, the task's relationships with other tasks, and any parameters required to complete the task.

[0035] The task list generation engine 114 may arrange the set of retrieved task templates 118 into a list by (a) the task's priority level and (b) the task's dependency on other tasks. In particular, a task that depends on another task to be completed is arranged lower on the task list than the parent task. In addition, if one task does not have a dependency on another, the system may arrange the tasks by a selected priority. For example, the system may apply a rule that, unless a dependency exists, computer-performed tasks should be given priority over human-performed tasks when ordering the task list. In one embodiment, the task list may include parallel lists of tasks. For example, a vehicle power-down sequence may require an ordered sequence of tasks performed by two different humans. The task list generation engine 114 may represent the separate sequences of tasks to be performed by the two humans as parallel lists of tasks.

[0036] The task list execution engine 115 executes, manages, or tracks tasks in the task list. For example, the task list execution engine 115 may include an application that causes the user interface 120 to generate a graphical user interface (GUI) that includes a representation of one or more tasks on the task list. The task list execution may directly monitor the system 102 to determine whether a task is executed. For example, the task list execution engine 115 may display an interface element representing a task for an operator to turn off power to a component. The task list execution engine 115 may detect whether the power has been turned off. The task list execution engine 115 may change the representation of the interface element to indicate that it has detected that the power has been turned off. For example, the task list execution engine 115 may gray out the representation of the task or display a check mark next to the representation of the task. Additionally or alternatively, the task list execution engine 115 may allow an operator to interact with the representation of the task in the user interface 120 to indicate completion of the task. When a task is completed, the task list execution engine 115 may highlight the next task in the task list to be executed. Additionally, the task list execution engine 115 may display tasks in the task list to be executed by the computer. The task list execution engine 115 may initiate execution of a computer-executed task based on detecting the completion of any prerequisite tasks without human intervention. For example, if a task list includes a human-executed task to obtain measurements and a subsequent computer-executed task to analyze system performance using the measurements, the task list execution engine 115 may (a) wait to execute the computer-executed task until an operator inputs the measurements from the prerequisite tasks, and (b) initiate the computer-executed task without human intervention upon detecting that the operator has input the measurements.

[0037] The task list execution engine 115 may further provide functionality via the GUI of the user interface 120 to allow an operator to modify one or more tasks. For example, the task list generation engine 114 may generate a task list for flushing data storage components in a data center. The task list execution engine 115 may display the task list via the GUI of the user interface 120. The operator may determine that the displayed task list includes a task that will cause a significant disruption to a customer. The operator may modify one or more tasks using the GUI to add a condition that the task should not be performed during a time period known to be a high customer usage period. Alternatively, the operator may modify the task list to include a new task requested by the customer. For example, the customer may request that the customer be notified of any potential disruption. The operator may add a task that requires the operator to send a notification and receive permission to perform the remaining tasks in the task list. By way of another example, the organization maintaining the monitored system 102 may have safety protocols required before a task can be performed. The task list execution engine 115 may allow a user to add new tasks, including tasks specified by safety protocols, using the GUI of the user interface 120.

[0038] In one embodiment, the task list execution engine 115 determines whether the task list is completed. If the operator indicates that the task list cannot be completed, the task list execution engine 115 may provide information associated with the task list and any tasks that cannot be completed to the causal analysis engine 113. The causal analysis engine 113 may determine, based on the completed and incomplete tasks, whether a detected triggering event in the monitored system 102 is associated with a different root cause than a previously identified root cause. If the causal analysis engine 113 identifies a different root cause associated with the triggering event, the task list generation engine 114 may generate a new task list to be executed, monitored, and tracked by the task list execution engine 115.

[0039] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

[0040] In one or more embodiments, system 100 may include more or fewer components than those illustrated in Figure 1. The components illustrated in Figure 1 may be local or remote from one another. The components illustrated in Figure 1 may be implemented in software and / or hardware. Each component may be distributed across multiple applications and / or machines. Multiple components may be combined into one application and / or machine. Operations described with respect to one component may instead be performed by another component.

[0041] Additional embodiments and / or examples relating to computer networks are described below in Section 6 entitled "Computer Networks and Cloud Networks."

[0042] In one or more embodiments, the data repository 116 is any type of storage unit and / or device for storing data (e.g., a file system, a database, a collection of tables, or any other storage mechanism). Furthermore, the data repository 116 may include multiple different storage units and / or devices. The multiple different storage units and / or devices may or may not be of the same type or located at the same physical site. Furthermore, the data repository 116 may be implemented or executed on the same computing system as the system monitor 110. Alternatively, or in addition, the data repository 116 may be implemented or executed on a computing system separate from the system monitor 110. The data repository 116 may be communicatively coupled to the system monitor 110 via a direct connection or via a network.

[0043] Information describing historical sensor data, task templates, and task lists may be embodied throughout any of the components in system 100. However, this information is illustrated in data repository 116 for purposes of clarity and exposition.

[0044] In one or more embodiments, system monitor 110 refers to hardware and / or software configured to perform the operations described herein for remediating a triggering event detected in a monitored system using a filtered dataset including validated and estimated sensor data. Example operations for remediating a triggering event detected in a monitored system using a filtered dataset including validated and estimated sensor data are described below with reference to Figures 2A and 2B.

[0045] In one embodiment, the system monitor 110 is implemented on one or more digital devices. The term "digital device" generally refers to any hardware device that includes a processor. A digital device may refer to a physical device that runs an application or a virtual machine. Examples of digital devices include computers, tablets, laptops, desktops, netbooks, servers, web servers, network policy servers, proxy servers, general-purpose machines, function-specific hardware devices, hardware routers, hardware switches, hardware firewalls, hardware network address translators (NATs), hardware load balancers, mainframes, televisions, content receivers, set-top boxes, printers, mobile handsets, smartphones, personal digital assistants ("PDAs"), wireless receivers and / or transmitters, base stations, communication management devices, routers, switches, controllers, access points, and / or client devices.

[0046] In one or more embodiments, user interface 120 refers to hardware and / or software configured to facilitate communication between a user and system monitoring device 110. Interface 120 renders user interface elements and receives input via user interface elements. Examples of interfaces include graphical user interfaces (GUIs), command line interfaces (CLIs), tactile interfaces, and voice command interfaces. Examples of user interface elements include check boxes, radio buttons, drop-down lists, list boxes, buttons, toggles, text fields, date and time selectors, command lines, sliders, pages, and forms.

[0047] In one embodiment, different components of interface 120 are specified in different languages. The behavior of user interface elements is specified in a dynamic programming language such as JavaScript. The content of user interface elements is specified in a markup language such as HyperText Markup Language (HTML) or XML User Interface Language (XUL). The layout of user interface elements is specified in a style sheet language such as Cascading Style Sheets (CSS). Alternatively, interface 120 is specified in one or more other languages, such as Java, C, or C++.

[0048] 3. Repairing system anomalies based on machine learning 2A and 2B illustrate an example set of operations for remediating a system anomaly based on a machine learning model by generating and implementing a task list, according to one or more embodiments. One or more of the operations illustrated in FIGS. 2A and 2B may be modified, rearranged, or removed altogether. Thus, the particular sequence of operations illustrated in FIGS. 2A and 2B should not be construed as limiting the scope of one or more embodiments.

[0049] The system obtains current sensor data (operation 202). The current sensor data includes measurements from multiple sensors monitoring one or more components of the monitored system. Examples of monitored systems include vehicles, data centers, computer networks, manufacturing facilities, medical facilities, and power generation facilities. The monitored system may include any number and types of components, including structural components, motion components, power generation components, computational components, and data storage components. Sensors include physical sensors such as thermal sensors, visual sensors, and optical sensors. Sensors include software-based sensors, such as applications, clients, or programs that monitor computing elements in a computer or a network of computing, network, and storage devices.

[0050] The system validates the sensor data (operation 204). Validating the sensor data includes determining that the sensor data meets predetermined criteria. For example, a machine learning model may be trained to identify correlations between sensors. Validating the sensor data may include determining that the current sensor data values ​​are consistent with the learned correlations of the machine learning model. Additionally, or alternatively, validating the sensor data may include determining that the values ​​for the sensors are within a previously determined range of non-anomalous values ​​for the sensors.

[0051] In one embodiment, the machine learning model is a deterministic machine learning model, such as the MSET model. The MSET model is trained on a set of sensor values ​​that includes the entire range of non-anomalous sensor values ​​for the set of sensors. Applying the MSET model to current sensor data, the system validates the sensor data by (a) determining that each sensor value is within a previously learned range of non-anomalous values ​​for the respective sensor, and (b) determining that the sensor data values ​​are consistent with learned correlations with other sensor data values.

[0052] The system identifies one or more sensor values ​​that cannot be validated (operation 206). Unvalidated sensor values ​​may include sensor values ​​that do not match learned correlations between the sensor and other sensors. For example, a machine learning model may learn that three sensors have output values ​​that follow the same path as each other. The system may detect that a current sensor value for one of the sensors cannot be validated based on detecting that the current sensor value for that sensor does not follow the same path as the output values ​​for the other two sensors.

[0053] In one embodiment, where the machine learning model is a deterministic model such as the MSET model, the machine learning model may generate, for each sensor value, a delta value representing the difference between the current sensor value and the predicted value for the sensor. The MSET model may determine that a sensor value cannot be validated based on determining that (a) the sensor value is outside a range of non-anomalous values ​​for the particular sensor, or (b) the sensor value varies from the predicted value by more than a predetermined threshold. According to one example, the MSET model applies a Sequential Probability Ratio Test (SPRT) algorithm to the sensor value and the predicted sensor value to determine whether the sensor value deviates from the predicted value by more than a predetermined threshold. The SPRT algorithm detects abnormal deviations from normal operation by calculating a cumulative sum of log-likelihood ratios for each successive residual between the measured signal and the estimated value and comparing the cumulative sum to a threshold to determine at which a fault is detected.

[0054] According to one example, the MSET model may identify correlations between values ​​of three sensors. The MSET model may determine that a third sensor should have a particular value based on the values ​​of two of the sensors. The MSET model may determine that the third sensor cannot be validated if the current value for the third sensor varies from the particular value by more than a predefined threshold. Additionally, the MSET model may determine that the third sensor has a predefined range of non-abnormal operating values. The MSET model may determine that the third sensor cannot be validated if the current sensor value for the third sensor is outside the predefined range of non-abnormal operating values.

[0055] The system determines whether the unvalidated sensor data can be estimated (operation 208). In one embodiment, the system trains a machine learning model with the dataset of sensor values ​​to identify correlations between sensors. The system may determine whether the unvalidated sensor values ​​can be estimated based on the learned correlations of the sensor values ​​by the machine learning model.

[0056] Based on determining that sensor values ​​that cannot be validated may be estimated, the system generates estimated values ​​for any sensor data that cannot be validated but can be estimated (operation 210). For example, in one embodiment where the sensor analysis model is an MSET model, the MSET model generates estimated values ​​for the sensor based on learned correlations between the sensor and other sensors.

[0057] In one embodiment, the system trains and applies one machine learning model to the sensor data to validate the sensor data, and trains and applies another machine learning model to the sensor data to estimate sensor values ​​that cannot be validated. For example, the system may train a machine learning model using a set of historical sensor data to learn correlations between sensor values. The system provides current sensor data to the trained machine learning model to determine whether correlations between current sensor values ​​match learned correlations between historical sensor values. The trained machine learning model validates the sensor values ​​based on determining that the values ​​match the learned correlations. The machine learning model determines that a subset of the values ​​cannot be validated based on determining that the subset of values ​​does not match learned correlations with other sensor values. The system trains another machine learning model to learn correlations between sensor values ​​in the set of historical sensor data and to predict sensor values ​​based on other sensor values ​​in the set of historical sensor data. The system provides current sensor data to the machine learning model. The machine learning model predicts a value for the sensor value based on the values ​​of other sensor values ​​in the current sensor data. The system substitutes predicted values ​​produced by the second machine learning model for measured values ​​identified by the first machine learning model as values ​​that cannot be validated.

[0058] In an alternative embodiment, the system uses the same machine learning model to both validate sensor data and estimate sensor data values ​​that cannot be validated. For example, the MSET machine learning model detects when sensor values ​​vary from a non-abnormal operating range and generates predictions for the sensor values ​​based on learned correlations between sensor values. The MSET model replaces sensor values ​​that cannot be validated with predicted sensor values.

[0059] Based on the validated sensor data and the estimated sensor data, if any, the system generates a filtered set of sensor data (operation 212). The system excludes sensor data, if any, that cannot be validated from the filtered set of sensor data.

[0060] Using the filtered set of sensor data, the system detects events associated with the monitored system (operation 214). In one embodiment, the trained sensor analytics machine learning model identifies anomalies in the sensor data. The system may detect a triggering event based on detecting one or more conditions associated with the current sensor data. For example, the system may detect a predefined number of sensor data values ​​above a threshold value. The system may detect specific sensor data values ​​above a threshold value. Additionally, or alternatively, the system may detect a combination of sensor values ​​that equates to a triggering event, even if one or more of the sensor values ​​are within a non-anomalous sensor value range for the specific sensor. In one embodiment, the system compares the estimated value for the sensor with the current value for the sensor to detect whether the sensor value is anomalous and whether the sensor value equates to a triggering event. In other words, the system may use a combination of the filtered set of sensor data and the current sensor data to detect whether a triggering event has occurred in the monitored system.

[0061] The system identifies a root cause of the detected triggering event (operation 216). The system may analyze outputs from one or both of the machine learning model and the system monitoring sensors to identify the root cause of the triggering event. In one embodiment, the system applies a trained causal analysis machine learning model to the filtered set of sensor data to identify the root cause of the triggering event. For example, a root cause of an overheating component may be a failure of a power regulator on the component. The machine learning model may be trained on a dataset including (a) historical sensor data and (b) historically identified causes for triggering events in the historical data to identify the root cause of the triggering event associated with the current sensor data.

[0062] The system identifies a set of tasks associated with the root cause (operation 218). According to one embodiment, the system generates a set of queries to a database of task templates. The system generates the set of queries based on one or more of: (a) the identified root cause, (b) the identified triggering event, and (c) one or both of the current sensor data and the filtered set of sensor data. The task template includes the action to be performed and any dependencies of the task on any other tasks. The task template further includes editable fields that the system fills in according to the specifications of the monitored system to generate the set of tasks. For example, the system may enter a specific model or description of a system component in the "Component" field. The system may enter the name of an operator, the operator's title, or the name of the computer selected to perform the action in the "Entity to perform action on" field.

[0063] The system instantiates task parameters for a set of tasks (operation 220). The set of tasks includes parameters that are populated by the system based on current sensor data. The system may populate the parameters based on validated sensor data. For example, a task may require a data latency measurement. Another task may require an environmental temperature measurement. The system populates the task parameters based on the measurements in the current sensor data.

[0064] The system determines whether the set of tasks includes any missing parameters (operation 222). For example, the system may determine that a task includes a parameter associated with a current sensor value that cannot be validated. The sensor value that cannot be validated results in a missing task parameter.

[0065] The system determines whether, for any missing parameters, the parameters can be estimated (operation 224). In one embodiment in which the system includes a sensor analytics machine learning model that learns correlations between different sensors, the system may determine whether the sensor analytics machine learning model generates output values ​​that are estimates of sensor values ​​that cannot be validated. Additionally or alternatively, the system may identify any other sensor values ​​that may be used to estimate the missing task parameters. For example, the system may determine that a first sensor value of a first component can be substituted into a formula to calculate an estimate for a second sensor having a second value that cannot be validated. In yet another embodiment, the system may determine whether any tasks may be substituted for a task with a missing parameter. For example, the system may determine that a data backup-type operation may be initiated either by a first task associated with the first component or by a second task associated with the second component. If a required task parameter of the first task is missing, the system determines that a second task may be substituted for the first task to achieve the desired result.

[0066] If the system determines that the missing parameters cannot be estimated, the system generates an explanation for the missing parameters (operation 226). The explanation may be a human-understandable explanation. The explanation may be selected from a set of explanation templates. The system may fill in fields of the explanation template to generate the human-understandable explanation. For example, the explanation may indicate that a particular task cannot be completed without a particular measurement and that sensor data associated with a particular measurement cannot be validated. In one embodiment, the human-understandable explanation is presented to the operator via a GUI. If a task is not essential to achieving the goal, the system may give the operator the option of generating a task list without the task. Additionally or alternatively, the system may provide interface elements to enable the operator to provide the necessary information to complete the task. For example, the user may supply the missing parameter information via the GUI.

[0067] If the system determines that the missing parameters can be estimated, the system instantiates the parameters based on the estimated parameter values ​​(operation 228). In embodiments where the system includes a sensor analytics machine learning model that learns correlations between different sensors, the system estimates the parameters based on estimated sensor values ​​for sensor values ​​that cannot be validated. In examples where a first sensor value of a first component is substituted into a formula to calculate an estimate for a second sensor having a second value that cannot be validated, the system may estimate the missing parameters based on the first sensor value instead of the second sensor value. According to an alternative embodiment, the system substitutes one task for another as an alternative to estimating the missing parameters. In examples where two tasks can achieve the desired result, the system may substitute the second task for the first task if the first task includes the missing parameter.

[0068] The system arranges the set of tasks in a task list (operation 230). The system arranges the set of tasks in the list based on one or both of the dependencies between the tasks and the priority levels of the tasks. The system arranges the tasks according to dependencies specified in the task template. For example, a task may specify that the task must always be preceded by a specific task. Furthermore, a task may specify that the task must be passed on to another specific task. A series of two or more tasks may specify that no intervening tasks may be performed between the two or more tasks. A task may specify a time period for performing the task. A task may specify any related tasks, such as tasks that must be performed simultaneously by another entity. The system may also arrange the tasks in the task list according to the priority level of the task. The priority level may be set by an operator. According to one example, an operator may identify a particular type of task as having a high priority. Alternatively, an operator may identify a task associated with a particular component as having a high priority. The system may arrange two tasks that do not have a dependency relationship according to the identified priority.

[0069] The system begins executing the task list (operation 232). The system may provide the task list to the operator via a user interface GUI, via printed media, or by any other means. In an embodiment in which the system provides the task list to the operator via a GUI, the system may display the task list and track its completion. The system may update the display of the task list based on detecting the completion of a task on the task list. During the execution of the task list, the system may execute any computer-implemented tasks without human intervention based on detecting the completion of a nearby preceding task on the task list.

[0070] The system determines whether the task list can be completed (operation 234). For example, a user may generate input via the GUI that one or more tasks are missing from the task list. A user may generate input indicating that two or more tasks on the task list are redundant. The system may identify one or more tasks that cannot be completed. For example, the environment that would require the task to be performed may not be accessible by a human or a computer. For example, physical components of a vehicle may not be accessible. Also, a task to validate a set of data may not be accessible due to security protocols.

[0071] Based on determining that the list of tasks cannot be completed, the system may modify one or both of the task parameters and the specific task associated with the particular root cause (operation 236). For example, if the user indicates that a task cannot be performed due to the user's inability to access a component for taking measurements, the system may attempt to identify an alternative measurement to substitute for the initial measurement or an alternative task to substitute for the initial task. If the user indicates that a task is missing, the system may generate new dependencies for an existing task template or an entirely new task template based on the user input. Based on updates to the set of tasks to be performed, the system determines whether the initially identified root cause is still determined to be the root cause of the triggering event. For example, the system may determine that the user's inability to perform a task indicates that another root cause different from the initially identified root cause may be the actual cause of the triggering event. The system repeats the process of identifying the root cause (operation 216) and identifying the task associated with the root cause (operation 218).

[0072] 4. Training a machine learning model 4A and 4B illustrate an example set of operations for training a machine learning model to validate sensor data, identify sensor data that cannot be validated, and estimate values ​​for sensor data that cannot be validated, according to one or more embodiments. One or more of the operations illustrated in FIGS. 4A and 4B may be modified, rearranged, or eliminated altogether. Thus, the particular sequence of operations illustrated in FIGS. 4A and 4B should not be construed as limiting the scope of one or more embodiments.

[0073] FIG. 4A illustrates an example set of operations for training a machine learning model, such as a neural network or a support vector machine (SVM), using stochastic optimization of weights, according to one or more embodiments.

[0074] The system obtains a set of historical data including historical sensor values ​​and identified root causes associated with particular combinations of sensor values ​​(operation 402). Examples of sensor values ​​include sensor signals output from physical sensors such as thermal sensors, light sensors, strain sensors, and positioning sensors, as well as sensors implemented as software executing on a processing device, such as programs for tracking data volume, bandwidth, data latency, and dropped data packet rates. Examples of root causes include physical malfunction of a component, circuitry failure, electrical component failure, power outage to a component, and misconfiguration of a software component resulting in a measurable data transmission or data storage failure.

[0075] The system generates a training dataset from the set of historical data (operation 404). The training dataset includes sensor values ​​for a set of sensors monitoring the system. The training dataset may further include labels for root causes in one or more components of the system being monitored. For example, one data point in the training data set includes a set of sensor data values ​​and labels that identify one or more causes of anomalous sensor data values ​​in the set of sensor data values. Another data point in the training data set includes another set of sensor data values ​​and another label that identifies one or more causes of the anomalous sensor data values.

[0076] The system applies a machine learning algorithm to the training dataset to train a machine learning model to learn relationships or correlations between the sensors and to predict root causes associated with different combinations of sensor values ​​(operation 406). One or more embodiments include multiple different machine learning models to perform different decisions or predictions. For example, the system may train one machine learning model to predict root causes associated with a particular set of current sensor values. The system may train another machine learning model to identify relationships or correlations between different sensors. The machine learning model may be trained to identify a subset of sensor values ​​in the current set of sensor values ​​that cannot be validated. The machine learning model identifies a subset of sensors that cannot be validated based on determining that a particular value for the subset of sensors does not match a learned relationship between the sensors. The system may train yet another machine learning model to generate predicted or estimated values ​​for the sensors. The system trains the machine learning model to learn relationships or correlations between sensors. The machine learning model predicts a value for a particular sensor based on values ​​of other sensors associated with the current set of sensor values.

[0077] In one embodiment, training the machine learning model includes receiving feedback on the root cause identification generated by the machine learning model (operation 408). For example, the system may display the root cause identification generated by the machine learning model on a GUI. The system may receive one or more inputs to modify the identified root cause. The system updates the machine learning model based on the feedback (operation 410).

[0078] FIG. 4B illustrates an example set of operations for training a deterministic-type machine learning (ML) model, such as an MSET model, according to one or more embodiments.

[0079] The system obtains a training data set (operation 412) including a range of sensor values ​​under non-anomalous conditions for a set of sensors monitoring the monitored system. The set of training data may include time series data. The time series data includes multiple signals generated by multiple sensors monitoring the monitored system. In one embodiment, the output levels of the multiple signals in the training data include a range of values ​​that are all non-anomalous values. The set of training data may be selected to include sensor data over time in which the sensor values ​​for each sensor vary within a range of values ​​defined as non-anomalous for the respective sensor.

[0080] The system applies a machine learning algorithm to the training dataset to train a machine learning model to identify (a) a range of non-anomalous values ​​for each sensor and (b) relationships between sensor values ​​for the set of sensors (operation 414). A deterministic-type ML model is trained using the non-anomalous training dataset to model non-anomalous operation of the monitored system. In one embodiment, the deterministic ML model includes an anomalous signal prediction model to identify time-series signal fluctuations from pre-defined operating parameters. The anomalous signal prediction model predicts values ​​for the set of sensors based on learned relationships with other sensors and identifies when the predicted value for a sensor differs from the current sensor value for the sensor.

[0081] One or more of the operations illustrated in Figure 4A may be modified, rearranged, or eliminated altogether, and thus the particular sequence of operations illustrated in Figure 4A should not be construed as limiting the scope of one or more embodiments.

[0082] 5. Example Embodiments Detailed examples are described below for clarity. The components and / or operations described below should be understood as specific examples that may not be applicable to certain embodiments. Therefore, the components and / or operations described below should not be construed to limit the scope of any of the claims.

[0083] 5A and 5B illustrate a system 500 according to an example embodiment. System 500 includes a data center 510 that is monitored by a sensor 513. Data center 510 includes computing devices 511a, 511b, and 511c. Cooling units 512a, 512b, and 512c cool computing devices 511a, 511b, and 511c, respectively.

[0084] Sensors 510 provide sensor data to data center monitor 520. Data center monitor includes MSET ML model 521 for analyzing the sensor data. MSET ML model 521 has been trained with a training dataset of sensor values ​​associated with sensors 513 to model non-anomalous sensor values ​​associated with non-anomalous operation of data center 510.

[0085] MSET ML model 521 generates predicted values ​​for sensor 513. Based on the predicted values, MSET ML model 521 identifies expected current sensor values ​​522a, such as within a predefined range from the predicted sensor value. Additionally, MSET ML model 521 identifies current sensor values ​​522b that are anomalous, i.e., have values ​​that are (a) outside a predefined range of predicted values ​​or (b) outside a learned range of non-anomalous values ​​for the sensor. For sensor values ​​identified as anomalous based on being outside a predefined range of predicted values, MSET ML model 521 generates estimated sensor data values ​​522c by replacing the current sensor value with the predicted current sensor value. For example, if a sensor is drifting such that the output value is losing correlation with the sensed characteristic over time, MSET ML model 521 replaces the inaccurate output sensor value resulting from the drift with a predicted sensor value based on learned correlations between the sensor and other sensors.

[0086] Causal analysis engine 523 analyzes sensor data output from MSET ML model 521 to identify a root cause associated with a detected triggering event. If MSET ML model 521 determines, based on learning correlations between sensor values, that a particular current sensor value (a) is consistent with learned relationships with other sensors and (b) is at an abnormal level outside of non-abnormal levels, causal analysis engine 523 uses the sensor value. Causal analysis engine 523 determines that the sensor is likely to accurately track an abnormal condition of a monitored component in data center 510. If MSET ML model 521 determines, based on learning correlations between sensor values, that a particular current sensor value (a) is inconsistent with learned relationships with other sensors, causal analysis engine 523 uses a predicted sensor value for the particular sensor instead of the current sensor value. Causal analysis engine 523 determines that the sensor is likely in a bad state and that the sensor's output is likely not reflective of the condition of the monitored component in data center 510.

[0087] In one or more embodiments, the causal analysis engine 523 exploits the deterministic nature of the MSET ML model to perform a traceback operation to identify the root cause of an event indicated by an anomalous signal value. In particular, the MSET ML model implements a mathematical structure that, for a given set of input values, will return the same set of output values. As a result, the MSET ML model is invertible. A given set of input values ​​may be determined based on a respective set of output values. Thus, the causal analysis engine 523 performs a traceback operation on the signals output by the MSET model to identify the specific subset of input signals that corresponds to the detected anomaly.

[0088] In the example embodiment of FIG. 5A , cause analysis engine 523 identifies cooling unit failure 524 as the root cause of the detected triggering event. Task list generation engine 525 identifies a set of task templates 531, stored in data repository 530, associated with cooling unit failure 524. In the embodiment of FIG. 5A , task list generation engine 525 references task-to-cause mapping 532 to identify the particular set of tasks associated with cooling unit failure 524. Task list generation engine 525 generates task list 526, which represents an ordered sequence of tasks 526a, 526b, 526c, and 526d to be performed in response to cooling unit failure 524.

[0089] The task list generation engine determines whether the set of tasks 526a-526d includes any missing parameters. For example, the task list generation engine may determine that task 526b, "Turn off cooling unit 1," requires a parameter indicating the current temperature of computing unit 511a and a parameter indicating the temperature of cooling unit 1 512a. If the system determines that the sensor data for the current temperature of computing device 511a cannot be validated, task list generation engine 525 replaces the current sensor data for the temperature of computing device 511a with the predicted sensor data generated by MSET model 521.

[0090] Task list generation engine 525 arranges a set of tasks 526a-526d based on the dependencies between the tasks and the priority levels of the tasks. For example, task 526a is a computer-executed task that has no dependencies on other tasks. Task list generation engine 525 applies rules that instruct task list generation engine 525 to assign higher priority to computer-executed tasks than tasks performed by humans. Task 526c, performed by a human, to measure the temperature in cooling unit 2 512b has a defined dependency relationship to task 526b. Thus, task list generation engine 525 arranges task 526c to be executed subsequent to task 526b.

[0091] 5B , the task execution engine begins executing task list 526. Task execution engine 527 causes user interface 528 to display a GUI depicting task list 526. As a task is executed, task execution engine 527 may cause user interface 528 to modify the depiction of the task in the GUI to indicate that the task is completed. For example, task execution engine 527 may cause user interface 528 to modify the depiction of the task in the GUI to minimize the completed task, change the color of the completed task, or display a check mark next to the completed task. Furthermore, task execution engine 527 may cause user interface 528 to modify the depiction of the task in the GUI to display details associated with the current task while minimizing other tasks. Task execution engine 527 may start task 526a without human input based on applying a rule that executes the computer-implemented rule upon the completion of a preceding task without waiting for human input.

[0092] Task execution engine 527 receives user input via user interface 528 indicating that a task has been completed. Task execution engine 527 may also detect whether a task has been completed independent of user input via user interface 528. For example, upon detecting via sensor 513 that power to cooling unit 1 512a has been turned off, task execution engine 527 may cause user interface 528 to modify the depiction of the task in the GUI to indicate that task 526b has been completed, without human input via user interface 528.

[0093] Additionally, if the user determines that the task list 526 requires modification, such as adding or removing a task or modifying a task element, such as the entity on which the task should be performed, the user may input the changes via a GUI displayed in the user interface 528. The task execution engine 527 may modify the task list 526 based on the user input.

[0094] 6. Computer Networks and Cloud Networks In one or more embodiments, a computer network provides connectivity between a set of nodes. The nodes can be local to each other and / or remote from each other. The nodes are connected by a set of links. Examples of links include coaxial cable, unshielded twisted cable, copper cable, optical fiber, and virtual links.

[0095] A subset of nodes implements computer networks. Examples of such nodes include switches, routers, firewalls, and network address translators (NATs). Another subset of nodes uses computer networks. Such nodes (further referred to as "hosts") may run client processes and / or server processes. A client process makes a request for a computing service (such as running a particular application and / or storing a particular amount of data). A server process responds by performing the requested service and / or returning corresponding data.

[0096] A computer network may be a physical network, including physical nodes connected by physical links. A physical node is any digital device. A physical node may be a function-specific hardware device, such as a hardware switch, a hardware router, a hardware firewall, and a hardware NAT. Additionally or alternatively, a physical node may be a general-purpose machine configured to run various virtual machines and / or applications that perform respective functions. A physical link is a physical medium connecting two or more physical nodes. Examples of links include coaxial cable, unshielded twisted cable, copper cable, and optical fiber.

[0097] A computer network may be an overlay network. An overlay network is a logical network implemented on top of another network (such as a physical network). Each node in the overlay network corresponds to a respective node in the underlying network. Thus, each node in the overlay network is associated with both an overlay address (for addressing the overlay node) and an underlay address (for addressing the underlay node that implements the overlay node). Overlay nodes may be digital devices and / or software processes (such as virtual machines, application instances, or threads). Links connecting overlay nodes are implemented as tunnels through the underlying network. Overlay nodes at either end of the tunnel treat the underlying multi-hop path between them as a single logical link. Tunneling is performed by encapsulation and decapsulation.

[0098] In one embodiment, a client may be local to and / or remote from a computer network. A client may access a computer network over a private network or another computer network, such as the Internet. A client may communicate a request to a computer network using a communication protocol, such as the Hypertext Transfer Protocol (HTTP). The request is communicated through an interface, such as a client interface (e.g., a web browser), a program interface, or an application programming interface (API).

[0099] In one embodiment, a computer network provides connectivity between clients and network resources. The network resources include hardware and / or software configured to run server processes. Examples of network resources include processors, data storage, virtual machines, containers, and / or software applications. The network resources are shared among multiple clients. The clients request computing services from the computer network independently of each other. The network resources are dynamically allocated to requests and / or clients on an on-demand basis. The network resources allocated to each request and / or client may be scaled up or down based on, for example, (a) the computing services requested by a particular client, (b) the aggregated computing services requested by a particular tenant, and / or (c) the aggregated computing services requested of the computer network. Such a computer network may be referred to as a "cloud network."

[0100] In one embodiment, a service provider offers a cloud network to one or more end users. Various service models may be enabled by the cloud network, including, but not limited to, Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). In SaaS, the service provider offers end users the ability to use the service provider's applications running on the network resources. In PaaS, the service provider offers end users the ability to deploy custom applications onto the network resources. The custom applications may be created using programming languages, libraries, services, and tools supported by the service provider. In IaaS, the service provider offers end users the ability to provision the processing, storage, network, and other basic computing resources provided by the network resources. Any optional applications, including operating systems, may be deployed on the network resources.

[0101] In one embodiment, a computer network may implement various deployment models, including, but not limited to, private clouds, public clouds, and hybrid clouds. In a private cloud, network resources are provisioned for exclusive use by a specific group of one or more entities (as used herein, the term "entity" refers to a corporation, organization, individual, or other entity). The network resources may be local to and / or remote from the premises of the specific group of entities. In a public cloud, cloud resources are provisioned for multiple entities (further referred to as "tenants" or "customers") that are independent of each other. The computer network and its network resources are accessed by clients corresponding to different tenants. Such a computer network may be referred to as a "multi-tenant computer network." Different tenants may use the same specific network resources at different times and / or at the same time. The network resources may be local to and / or remote from the tenant's premises. In a hybrid cloud, the computer network includes a private cloud and a public cloud. The interface between the private cloud and the public cloud allows for data and application portability. Data stored in the private cloud and data stored in the public cloud may be exchanged through the interface. Applications implemented in the private cloud and applications implemented in the public cloud may have dependencies on each other. Calls from applications in the private cloud to applications in the public cloud (and vice versa) may be made through the interface.

[0102] In one embodiment, tenants of a multi-tenant computer network are independent of one another. For example, the business or operations of one tenant may be separate from the business or operations of another tenant. Different tenants may desire different network requirements for the computer network. Examples of network requirements include processing speed, amount of data storage, security requirements, performance requirements, throughput requirements, latency requirements, resiliency requirements, quality of service (QoS) requirements, tenant isolation, and / or integrity. The same computer network may be required to achieve different network requirements desired by different tenants.

[0103] In one or more embodiments, in a multi-tenant computer network, tenant isolation is implemented to ensure that applications and / or data of different tenants are not shared with each other. Various tenant isolation techniques may be used.

[0104] In one embodiment, each tenant is associated with a tenant ID. Each network resource in a multi-tenant computer network is tagged with a tenant ID. A tenant is granted access to a particular network resource if the tenant and the particular network resource are associated with the same tenant ID.

[0105] In one embodiment, each tenant is associated with a tenant ID. Each application implemented by the computer network is tagged with a tenant ID. Additionally or alternatively, each data structure and / or dataset stored by the computer network is tagged with a tenant ID. A tenant is granted access to a particular application, data structure, and / or dataset if the tenant and the particular application, data structure, and / or dataset are associated with the same tenant ID.

[0106] As an example, each database implemented by a multi-tenant computer network may be tagged with a tenant ID. Only the tenant associated with the corresponding tenant ID may access the data in a particular database. As another example, each entry in a database implemented by a multi-tenant computer network may be tagged with a tenant ID. Only the tenant associated with the corresponding tenant ID may access the data in a particular entry. However, a database may be shared by multiple tenants.

[0107] In one embodiment, a subscription list indicates which tenants have permission to access which applications. For each application, a list of tenant IDs of tenants authorized to access the application is stored. A tenant is granted access to a particular application if the tenant's tenant ID is included in the subscription list corresponding to the particular application.

[0108] In one embodiment, network resources (such as digital devices, virtual machines, application instances, and threads) corresponding to different tenants are isolated to tenant-specific overlay networks maintained by a multi-tenant computer network. As an example, a packet from any source device in a tenant overlay network may be transmitted to other devices in the same tenant overlay network. An encapsulation tunnel is used to prohibit any transmission from a source device on a tenant overlay network to a device in another tenant overlay network. Specifically, a packet received from a source device is encapsulated in an outer packet. The outer packet is transmitted from a first encapsulation tunnel endpoint (communicating with a source device in the tenant overlay network) to a second encapsulation tunnel endpoint (communicating with a destination device in the tenant overlay network). The second encapsulation tunnel endpoint decapsulates the outer packet to obtain the original packet transmitted by the source device. The original packet is transmitted from the second encapsulation tunnel endpoint to a destination device in the same specific overlay network.

[0109] 7. Miscellaneous, Extensions Embodiments are directed to systems involving one or more devices that include a hardware processor and are configured to perform any of the operations described herein and / or recited in any of the claims below.

[0110] In one embodiment, a non-transitory computer-readable storage medium includes instructions that, when executed by one or more hardware processors, cause the performance of any of the operations described herein and / or recited in any of the claims.

[0111] Any combination of the features and functionality described herein may be used by one or more embodiments. In the foregoing specification, the embodiments have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings should therefore be regarded in an illustrative rather than restrictive sense. The sole and exclusive indication of the scope of the invention, and what is intended by the applicants to be the scope of the invention, is the literal and equivalent range of the set of claims issuing from this application, in the particular form in which the claims are issued, including any subsequent amendments.

[0112] 8. Hardware Overview According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hardwired to execute the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or network processing units (NPUs) that are permanently programmed to execute the techniques, or may include one or more general-purpose hardware processors that are programmed to execute the techniques according to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices may even combine custom hardwired logic, ASICs, FPGAs, or NPUs with custom programming to accomplish the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices, or any other devices that incorporate hardwired and / or program logic to implement the techniques.

[0113] 6 is a block diagram illustrating an exemplary computer system 600 on which an embodiment of the present invention may be implemented. Computer system 600 includes a bus 602 or other communication mechanism for communicating information, and a hardware processor 604 coupled with bus 602 for processing information. Hardware processor 604 may be, for example, a general-purpose microprocessor.

[0114] Computer system 600 also includes a main memory 606, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 602 for storing information and instructions to be executed by processor 604. Main memory 606 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 604. Such instructions, when stored in a non-transitory storage medium accessible to processor 604, transform computer system 600 into a special-purpose machine that is customized to perform the operations specified in the instructions.

[0115] Computer system 600 further includes a read-only memory (ROM) 608 or other static storage device coupled to bus 602 for storing static information and instructions for processor 604. A storage device 610, such as a magnetic disk or optical disk, is provided and coupled to bus 602 for storing information and instructions.

[0116] Computer system 600 may be coupled via bus 602 to a display 612, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 614, including alphanumeric and other keys, is coupled to bus 602 for communicating information and command selections to processor 604. Another type of user input device is a cursor control 616, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to processor 604 and for controlling cursor movement on display 612. The input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify a position in a plane.

[0117] Computer system 600 may implement the techniques described herein using customized hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with the computer system, causes or programs computer system 600 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 600 in response to processor 604 executing one or more sequences of one or more instructions contained in main memory 606. Such instructions may be read into main memory 606 from another storage medium, such as storage device 610. Execution of the sequences of instructions contained in main memory 606 causes processor 604 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0118] The term "storage medium" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 610. Volatile media include dynamic memory, such as main memory 606. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROMs, and EPROMs, flash EPROMs, NVRAMs, any other memory chips or cartridges, content addressable memories (CAMs), and ternary content addressable memories (TCAMs).

[0119] Storage media is distinct from but may be used in conjunction with transmission media. Transmission media involves transferring information between storage media. For example, transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise bus 602. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0120] Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 604 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 600 may receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector may receive the data carried in the infrared signal and appropriate circuitry may place the data on bus 602. Bus 602 carries the data to main memory 606, from which processor 604 retrieves and executes the instructions. The instructions received by main memory 606 may optionally be stored on storage device 610 either before or after execution by processor 604.

[0121] Computer system 600 also includes a communication interface 618 coupled to bus 602. The communication interface 618 provides a two-way data communication coupling to a network link 620 that is connected to a local network 622. For example, the communication interface 618 may be an Integrated Services Digital Network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, the communication interface 618 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. A wireless link may also be implemented. In any such implementation, the communication interface 618 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0122] Network link 620 typically provides data communication through one or more networks to other data devices. For example, network link 620 may provide a connection through local network 622 to a host computer 624 or to data equipment operated by an Internet Service Provider (ISP) 626. ISP 626 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the “Internet” 628. Local network 622 and Internet 628 both use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 620 and through communication interface 618, which carry the digital data to and from computer system 600, are example forms of transmission media.

[0123] Computer system 600 can send messages and receive data, including program code, through the network(s), network link 620 and communication interface 618. In the Internet example, a server 630 might transmit a requested code for an application program through Internet 628, ISP 626, local network 622 and communication interface 618.

[0124] The received code may be executed by processor 604 as it is received, and / or stored in storage device 610, or other non-volatile storage for later execution.

[0125] In the foregoing specification, embodiments of the present invention have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings should therefore be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indication of the scope of the invention, and what is intended by the applicants to be the scope of the invention, is the literal and equivalent range of the set of claims issuing from this application in the particular form in which they are issued, including any subsequent amendments.

Claims

1. 1. A computer-readable program comprising instructions that, when executed by one or more hardware processors, cause operations to be performed, the operations including: training a first machine learning model based on historical sensor data obtained from a plurality of data sources; applying the trained first machine learning model to current sensor data detected by the plurality of data sources to identify a particular subset of the current sensor data that may not be validated based on data relationships corresponding to the historical sensor data; applying the trained second machine learning model to generate estimated sensor data to substitute for the particular subset of the current sensor data that may not be validated; and analyzing the current sensor data with the estimated sensor data substituted for the particular subset of the current sensor data to generate a task list for remediating anomalous events; the task list for repairing the abnormal event includes a task for repairing a root cause of the abnormal event; The operation is In response to receiving input to modify two or more tasks associated with the root cause, adding a new task to the two or more tasks; and Removing at least one task from the two or more tasks. modifying the two or more tasks by performing one or both of: and regenerating the task list based on modifying the two or more tasks.

2. 2. The computer-readable program of claim 1, wherein the task list specifies, for a particular task, (a) an entity that is to perform the particular task, (b) an action to be performed, and (c) a component of a monitored system on which the action is to be performed.

3. The computer readable program of claim 1 , wherein the task list comprises an ordered sequence of two or more tasks.

4. at least one first task among the two or more tasks is a task to be performed by a first entity; The computer-readable program of claim 3 , wherein at least one second task among the two or more tasks is a task to be performed by a second entity different from the first entity.

5. the two or more tasks further specify a dependency of one task among the two or more tasks on another task among the two or more tasks; at least one task among the two or more tasks includes a dependency on another task among the two or more tasks; The computer-readable program of claim 3 , wherein the two or more tasks are arranged in a sequence according to the dependency relationship.

6. at least one first task among the two or more tasks is a task to be performed by a human; at least one second task among the two or more tasks is a task to be performed by a computer without human intervention; The instruction:

4. The computer readable program of claim 3, further comprising: in response to detecting completion of the at least one second task, generating a human readable notification associated with the completion of the at least one second task.

7. The operation is 2. The computer-readable program of claim 1, further comprising identifying the anomalous event based on the current sensor data with the estimated sensor data substituted for the particular subset of the current sensor data.

8. The computer-readable program of claim 1 , wherein the trained first machine learning model and the trained second machine learning model correspond to the same machine learning model.

9. The computer readable program of claim 8 , wherein the same machine learning model is a multivariate state estimation technique (MSET) model.

10. The operation is obtaining a training data set from the historical sensor data; training a second machine learning model to identify correlations between the plurality of data sources based on the training dataset; applying the trained second machine learning model to generate estimated sensor data to substitute for the particular subset of the current sensor data that may not be validated; the trained second machine learning model identifying the correlation between the particular subset of the current sensor data and another subset of the current sensor data to be validated; and generating the estimated sensor data based on the correlation.

11. A computer-readable program comprising instructions that, when executed by one or more hardware processors, cause operations to be performed, the operations comprising: training a first machine learning model based on historical sensor data obtained from a plurality of data sources; applying the trained first machine learning model to current sensor data detected by the plurality of data sources to identify a particular subset of the current sensor data that may not be validated based on data relationships corresponding to the historical sensor data; applying the trained second machine learning model to generate estimated sensor data to substitute for the particular subset of the current sensor data that may not be validated; and analyzing the current sensor data with the estimated sensor data substituted for the particular subset of the current sensor data to generate a task list for remediating anomalous events; the task list for repairing the abnormal event includes a task for repairing a root cause of the abnormal event; generating the task list, generating at least one query based on the root cause; and querying a set of stored task templates to identify two or more tasks that satisfy a query condition associated with the at least one query.

12. 1. A computer-readable program comprising instructions that, when executed by one or more hardware processors, cause operations to be performed, the operations including: training a machine learning model based on historical sensor data obtained from a plurality of sensors; applying the trained machine learning model to current sensor data detected by the plurality of sensors to identify a particular subset of the current sensor data that may not be validated based on data relationships corresponding to the historical sensor data; filtering out the particular subset of current sensor data that cannot be validated to obtain a filtered set of current sensor data that includes validated sensor data; performing an analysis on the filtered set of current sensor data that does not include the particular subset of current sensor data to identify issues to be remediated; generating recommendations for repairing the identified problems based on the filtered set of current sensor data; The instruction: performing further operations including estimating a second subset of the current sensor data for use in the analysis in place of the particular subset of the current sensor data, the estimating being based on the data relationships corresponding to the historical sensor data; Generating recommendations for repairing the issues includes: generating a task list including a plurality of tasks to be executed in sequence to repair the problem; generating the task list, identifying a set of parameters required to generate the task list, the set of parameters including at least one value from the filtered set of current sensor data and at least one value from the estimated second subset of the current sensor data.

13. The computer-readable program of claim 12, wherein the analysis is further performed on the estimated second subset of the current sensor data in addition to the filtered set of the current sensor data to identify the problem to be repaired.

14. A method comprising the operations of any of claims 1 to 13.

15. 1. A system comprising: one or more processors; a memory storing instructions that, when executed by said one or more processors, cause said system to perform the operations of any of claims 1 to 13.

16. A system comprising means for performing the operations of any of claims 1 to 13.

Images