Recommendation generation using machine learning data validation
A machine learning model validates and estimates sensor data to generate task lists for remediation, addressing faulty sensor issues and ensuring accurate system fault rectification.
Patent Information
- Application Number
- JP2024529591
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-18
- Filing Date
- 2022-08-19
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2042-08-19
AI Technical Summary
Existing systems face issues with remediation actions failing to address actual faults in monitored systems due to faulty sensors, potentially worsening defects and leading to system failure, necessitating human oversight to verify sensor readings.
A machine learning model is trained on historical sensor data to validate and estimate sensor values, generating a filtered dataset to identify anomalies and generate task lists for remediation, including both computer-performed and human-performed tasks, using deterministic models like MSET to detect and rectify faults.
The system effectively identifies and rectifies system abnormalities by using machine learning to validate and estimate sensor data, ensuring accurate remediation actions and reducing the risk of system failure.
Smart Images

Figure 2024542210000001_ABST
Abstract
Description
[Technical field]
[0001] Incorporation by Reference, Disclaimer The subsequent application, Application No. 17 / 455,536, filed November 18, 2021, is hereby incorporated by reference. Applicant hereby cancels any disclaimers of claim scope in the parent application or its prosecution history, and notifies the USPTO that the claims in this application may be broader than any claims in the parent application.
[0002] Technical Field The present disclosure relates to applying machine learning models to generated sensor data to generate recommendations for remediating detected issues using a filtered dataset of the sensor data. In particular, the present disclosure relates to using machine learning validated and estimated sensor values to generate recommendations for remediating issues. [Background technology]
[0003] background When an anomaly occurs in a monitored system, the anomaly may be the result of a fault in the monitored component or in the sensor performing the monitoring. If a fault remediation system initiates a remedial action based on a detected anomaly that is the result of a faulty sensor, the remedial action may fail to address the actual fault in the system. In a worst case scenario, the remedial action may exacerbate the fault and lead to a system failure.
[0004] Organizations may utilize human oversight of computer-based monitoring and repair systems to verify computer results and take actions that a computer cannot perform. For example, in one environment, a repair action may require a human to physically manipulate an object, such as a valve, lever, or switch, or to physically watch a component. In another environment, an organization may require human confirmation of computer-generated data. For example, a sensor may indicate that a component is at a certain temperature. An organization may require a human-generated temperature measurement to confirm the temperature reading. A human may even determine whether there is any physical defect present that may result in a bad temperature reading by the sensor. In other environments, such as autonomous vehicles or computer-monitored data centers, both the monitoring and repair functions may have the ability to be performed by a computer without human intervention. A human may be required in the rare event of a computer failure. However, if a human is not part of the normal watching and repair process, human attention may be distracted and the human may not effectively monitor the computer's execution. Thus, an organization may require that certain monitoring and remediation tasks be performed by a human, even if a computer is capable of performing the tasks without human intervention.
[0005] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
[0006] The embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings. It should be noted that references to "an" or "one" embodiment in this disclosure are not necessarily to the same embodiment, but that such references mean at least one. In the drawings: [Brief description of the drawings]
[0007] [Figure 1] FIG. 1 illustrates a system according to one or more embodiments. [Figure 2A] FIG. 1 illustrates an example set of operations for generating a task list based on data validated by machine learning, in accordance with one or more embodiments. [Figure 2B] FIG. 1 illustrates an example set of operations for generating a task list based on data validated by machine learning, in accordance with one or more embodiments. [Diagram 3] FIG. 1 illustrates an example set of tasks, according to one or more embodiments. [Figure 4A] FIG. 1 illustrates an example of training a machine learning model, according to one or more embodiments. [Figure 4B] FIG. 1 illustrates another example of training a machine learning model, according to one or more embodiments. [Figure 5A] FIG. 1 illustrates an example embodiment of generating a task list based on data validated by machine learning. [Figure 5B] FIG. 1 illustrates an example embodiment of generating a task list based on data validated by machine learning. [Figure 6] FIG. 1 shows a block diagram illustrating a computer system in accordance with one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] Detailed Description In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding. One or more embodiments may be practiced without these specific details. Features described in one embodiment may be combined with features described in a different embodiment. In some instances, well-known structures and devices are described with reference to block diagram form in order to avoid unnecessarily obscuring the present invention.
[0009] 1. Overall Overview 2. System Architecture 3. Fix system anomalies based on machine learning 4. Training the machine learning model 5. Example embodiment 6. Computer Networks and Cloud Networks 7. Miscellaneous; Extensions 8. Hardware Overview 1. Overall Overview One or more embodiments filter out a subset of current sensor data that cannot be validated based on the historical sensor data to generate filtered sensor data. Optionally, the system may generate and add estimated sensor data to the filtered sensor data in place of the filtered subset of sensor data. The system analyzes the filtered sensor data to identify anomalous events and / or generate a task list for remediating the anomalous events.
[0010] One or more embodiments train a machine learning model based on historical sensor data to validate a target set of sensor data. The system uses a set of historical sensor data to train the machine learning model to learn correlations between sensors. Upon receiving current sensor data, the machine learning model identifies a subset of sensor data from a sensor that cannot be validated. For example, a sensor may be in a faulty state and not transmitting data. Another sensor may be drifting, such that the sensor transmits data, but the data is inaccurate. The system uses the learned correlations between sensors to estimate values for sensors that cannot be validated. The system uses a filtered dataset that includes the validated sensor data and estimated sensor values for sensor data that could not be validated to generate recommendations for repairing faults in the monitored system.
[0011] One or more embodiments train a machine learning model based on the historical sensor data to generate estimated sensor data. The system trains the machine learning model using a set of historical sensor data to learn correlations between sensors. Upon receiving current sensor data, the machine learning model predicts values for the sensor values based on correlations with the values of the remaining sensor values. The machine learning model generates estimated sensor data based on the predicted values for the sensors.
[0012] The sensor-based monitoring system includes one or both of machine learning and human based sensor validation and fault remediation. Machine learning models are trained to identify correlations between sensor data values of sensors monitoring the system.
[0013] The sensor-based monitoring system may generate a task list for repairing one or more detected faults. The system selects tasks for the task list based on a filtered data set including the validated sensor data and the estimated sensor values. The task list may include computer-performed tasks and human-performed tasks. The tasks specify the entity to perform the task, the system component on which the task should be performed, the action to be performed or accomplished, any dependencies of the task on any other tasks, and any parameters required to perform the task. The parameters include the validated sensor data. For example, the system may select one task to be performed if a sensor generates a first temperature measurement and another task to be performed if the sensor generates a second temperature measurement. The system specifies the parameters in the selected tasks as the generated temperature measurements. If the system cannot validate the parameters for the tasks, such as when a required sensor value is missing, the system populates the parameters with estimated sensor values generated by a machine learning model based on a correlation between the missing sensor value and one or more validated sensor values.
[0014] In one embodiment, the machine learning model is a deterministic machine learning model. For example, the machine learning model can be a multivariate state estimation technique (MSET) model. The MSET model is trained using a set of sensor values that represent a range of non-anomalous values for the sensor. The MSET model identifies correlations between the sensor values. In operation, the MSET model obtains current sensor values, compares the current sensor values to predicted sensor values, and identifies anomalies between the sensor values. Additionally, the MSET model determines whether the abnormal sensor values are the result of a faulty sensor or a fault in the monitored component based on the identified correlations between the sensor values. The MSET model uses the filtered dataset including the non-anomalous sensor values and the estimated sensor values to identify a root cause or condition that is responsible for the detected anomaly in the sensor data. The MSET model uses the filtered dataset including the non-anomalous sensor values and the estimated sensor values to further instantiate task parameters for one or more tasks in a task list to remediate the root cause or condition that is responsible for the detected anomaly.
[0015] In one or more embodiments, the system receives input to modify a task in a task list. For example, the system may receive input to (a) add or remove a task from the task list that is associated with a particular root cause or situation, (b) add or remove a parameter associated with a task in the task list, or (3) modify one or more attributes associated with a task, such as the entity on which the task should be performed.
[0016] In one embodiment, the system generates a task list by selecting two or more task templates from among a set of stored task templates. The system queries the set of stored task templates based on an identified root cause or condition underlying one or more detected anomalies in the monitored system. The system inputs specific data associated with the identified root cause or condition into a query template. For example, the query may return a specific task template associated with the identified condition. The system further identifies specific entities that match entity descriptions in the task template, specific components that match component descriptions in the task template, and sensor values that match parameter descriptions in the task template.
[0017] The system arranges tasks in a task list according to a set of rules. The set of rules may specify the relative priority of the tasks and the dependency of one task on another. For example, a series of three tasks may need to be executed consecutively in sequence within a predetermined period of time. According to another example, a task may specify two actions that must be executed simultaneously by two separate entities. A task may specify whether one task may be started before another and whether one task may be completed before another. One task in a task list may be given the highest priority, which indicates that the task must be executed before any other task. For example, a task list that guides a human through a process to detect a voltage in a component may begin with a task to disconnect the component from a power source. A second highest priority task may in turn depend on the first task and may instruct the human to drain the power stored in the component before starting a sequence of tasks to take measurements on the component.
[0018] In one embodiment, the system may determine that one or more sensor values may not be validated and may even not be estimated. Thus, the system may generate a human-readable explanation indicating that certain data is required to generate a task list, but that the data is not available. The system may receive an input to initiate an associated task list. Alternatively, a user may manually provide the required data via a graphical user interface (GUI) to enable the system to generate a task list.
[0019] In one embodiment, the system tracks whether a task is completed. The system may generate a user interface that includes visual elements that represent the set of tasks. When the system detects a completed task, the system may modify the visual elements of the GUI to indicate that the task is completed. The system may detect the completion of both computer-performed and human-performed tasks. Alternatively, the system may receive human-generated input to indicate when a human-performed task is completed. In one or more embodiments, the system may update or modify a task list based on data obtained during the execution of one or more tasks. For example, an initial task list may include a set of tasks to replace a system component. The system may detect, in an intermediate task, a human-generated measurement that indicates that a sub-component has a value outside of a safe operating range. The system may generate a notification to stop execution of the initial task list. The system may even generate a new task list to shut off power to the sub-component. The new task list may replace the initial task list. Alternatively, the new task list may be added to the initial task list.
[0020] One or more embodiments described herein and / or recited in the claims may not be included in this General Overview section.
[0021] 2. System Architecture FIG. 1 illustrates a system 100 according to one or more embodiments. As illustrated in FIG. 1, the system 100 includes a monitored system 102 and a system monitor 110. The monitored system 102 may include any type of system having any number of components. Examples of monitored systems include vehicles, data centers, computer networks, manufacturing facilities, medical facilities, and power generation facilities. The monitored system 102 may include any number and type of components, including structural components, motion components, power generation components, computational components, and data storage components. The components of the monitored system 102 are monitored by sensors 103. The sensors include physical sensors and software that monitor the computing elements. For example, physical motion elements may be monitored by temperature sensors and piezoelectric sensors. Data processing components may be monitored by temperature sensors and applications that track data traffic to and from the data processing components. Medical devices may be monitored by location sensors. Vehicles may be monitored by global positioning system (GPS) sensors, altimeters, and speedometers. In one or more embodiments, the sensors 103 include tens, hundreds, or thousands of sensors. For example, the number and types of sensors may be sufficient to enable a machine learning model to identify correlations between different sensors monitoring the same system component.
[0022] The system monitor 110 includes a sensor analytics machine learning model engine 111. The sensor analytics machine learning model engine 111 applies one or more machine learning models to sensor data acquired from sensors 103 monitoring the monitored system 102. The machine learning models may be probabilistic type machine learning models, such as neural networks. Alternatively, the machine learning models may be deterministic type machine learning models, such as multivariate state estimation technique (MSET) models. Probabilistic type machine learning models apply probabilistic methods for generating predictions. In contrast, deterministic type machine learning models implement mathematical structures that will return the same set of output values for a given set of input values.
[0023] The sensor analytics machine learning engine 111 trains a machine learning model using historical sensor data 117 stored in the data repository 116. The sensor analytics machine learning engine 111 trains the machine learning model to identify correlations between sensor values. In one embodiment, the trained machine learning model receives current sensor data and generates a set of validated sensor values. The validated sensor values may correspond to sensor values that meet predefined criteria. For example, the validated sensor values may fall within a range of expected values for the sensor. The trained machine learning model may further identify a set of sensor values that may not be validated. The unvalidated sensor values may include values that are outside of a predicted range of values for a particular sensor. Alternatively, the unvalidated sensor values.
[0024] In one embodiment where the machine learning model is a deterministic model such as the MSET model, the machine learning model may generate, for each sensor value, a delta value that represents the difference between the current sensor value and the predicted value for the sensor. The MSET model may indicate an anomaly in the sensor value if (a) the sensor value is outside of a range of non-anomalous values for the particular sensor, or (b) the sensor value varies from the predicted value by more than a pre-determined threshold. For example, the MSET model may identify a correlation between the values of three sensors. The MSET model may determine that a third sensor should have a particular value based on the values of two of the sensors. The MSET model may detect an anomaly if the current value for the third sensor varies from the particular value by more than a pre-defined threshold. In addition, the MSET model may determine that the third sensor has a pre-defined range of non-anomalous operating values. The MSET model may detect an anomaly if the current sensor value for the third sensor is outside of a pre-defined range of non-anomalous operating values.
[0025] The event detection engine 112 detects whether a triggering event has occurred in the system. The triggering event may be an event detected based on the current values of the sensors 103. Alternatively, the triggering event may be a manually triggered event intended to initiate a sequence of tasks associated with the monitored system 102. The event detection engine 112 monitors the output from one or both of the machine learning models and the sensors 103 to determine whether an anomaly in the sensor values corresponds to a sensor-based triggering event in the monitored system 102. For example, one combination of anomalous sensor values may correspond to an overheating type triggering event. Another combination may correspond to a power failure type triggering event. Yet another combination may correspond to a data communication failure type triggering event.
[0026] The causal analysis engine 113 analyzes the output from the machine learning model and one or both of the sensors 103 to identify a root cause of the triggering event. For example, if the event detection engine 112 detects sensor data associated with an overheating component, the causal analysis engine 113 may analyze the sensor data and metadata describing the component to identify a failing power adapter in the component as the cause of the overheating event. The causal analysis engine 113 may analyze metadata describing the component to identify, for example, the type of component and the component's specifications. According to another example, if the event detection engine 112 detects a data communication failure event, the causal analysis engine 113 may analyze sensor data from a set of components associated with the communication failure to determine that a router is causing a data latency rate that exceeds a non-abnormal data latency rate. If the causal analysis engine 113 determines that the triggering event was a manually triggered event, the causal analysis engine 113 may identify a reason associated with the manually triggered event. For example, a user may interact with the user interface 120 to initiate a power shutdown for a facility based on an emergency situation that may not be detected by the sensors 103. The causal analysis engine 113 determines that the root cause of the event is a manually initiated power shutdown.
[0027] The task list generation engine 114 generates a task list for repairing the identified root cause of the triggering event. The task list generation engine 114 queries a body of task templates 118 stored in the data repository 116 to identify a task template 118 associated with the identified root cause. FIG. 3 illustrates examples of task templates 301 and 307 according to one embodiment. The task list generation engine 114 may select a set of tasks associated with a particular root cause based on one or more of: (1) the identified root cause of the event, (2) the output from the sensor analytics machine learning model, and (3) the output from the sensor 103. For example, the task list generation engine 114 may generate a set of queries having the following conditions: (a) power level above a threshold, (b) in a particular power adapter, (c) providing power to a particular component, and (d) receiving power from a particular supply. The set of queries may return a corresponding set of tasks selected to be performed for the specified condition. In one embodiment, the set of tasks is stored as a task list 119. For example, if the same root cause has occurred previously, the previously generated set of tasks may be stored as a task list 119. Additionally, if a user modifies the set of tasks associated with a particular root cause, the modifications may be stored in the task list 119.
[0028] The task list generation engine 114 populates the task template 118 based on certain information associated with the monitored system 102. With reference to FIG. 3, a task 301 includes fields for a system component 302, an action to be performed 303, an entity to perform the action 304, the task's relationship to other tasks 305, and a parameter value 306 that corresponds to one or both of a measured sensor value and an output from a sensor analytics machine learning model. The task list generation engine 114 populates the system component field 302 with the component in the monitored system 102 that is to be acted upon by the entity. For example, a set of tasks associated with an overheating power adapter may include a task that specifies a power supply as the system component 302, specifies flipping a power switch as the action to be performed 303, and specifies a particular operator as the entity to perform the action 304. The task list generation engine 114 may populate the "system component" field 302 with a description of the component to be acted upon. For example, the task list generation engine 114 may reference the rack containing the power supply and the specifications of the power adapter to enter the data “Power supply M103 located on the lower right of the server rack” into the “System Components” field 302.
[0029] The task list generation engine 114 may populate the "entity to perform action" field 304 with the name or role of a particular individual responsible for the action. For example, the task list generation engine 114 may access an organization roles table to identify "D. Scott" as an engineer responsible for servicing overheating power adapters. The task list generation engine 114 may populate the "entity to perform action" field 304 with the name "D. Scott" or with the title "engineer." In one or more embodiments, some tasks in a task list may be performed by a human operator and other tasks may be performed by a computer without human intervention. For example, the task template 301 may identify a computer "COMP-1" as performing a data analysis task subsequent to a user performing a preceding task in the task list.
[0030] The task template 301 further identifies each task's relationships 305 to other tasks. For example, the task template 301 may indicate that the current task must be performed immediately after task A, without any intervening tasks. The task template 301 may further specify that the current task must be performed before task C, although one or more tasks may intervene between the current task and task C. The task template 301 may further identify the chain or family of tasks to which a particular template belongs.
[0031] 3, a task template 307 may specify a compound task that may necessarily include two or more actions 308 and 309 to be performed simultaneously. The task template 307 may further specify that the two or more actions 308 and 309 are performed by one or more entities. For example, the compound task template 307 may specify pressing one button and pressing a second button while the first button is pressed. The compound task template 307 may specify pressing a button on one system component by one operator and a measurement by another operator on another system component while the first button is pressed.
[0032] The task template 301 specifies a set of parameters 306 required to perform a task. For example, the task template 301 may specify that if a temperature sensor value is within a first range, the entity should perform a first action. The task template 301 may further specify that if a temperature sensor value is within a second range, the entity should perform a second action. The action performed may depend on parameters 306 corresponding to one or both of an output from the sensor 103 and an output from a sensor analytics machine learning model. The parameters 306 may be direct measurements from the sensor 103 and values derived from the measurements. For example, a data system monitor may measure the latency of a data flow associated with a component. The parameters 306 may be a measure of average packet delay based on the measured latency.
[0033] In one embodiment, the task list generation engine 114 determines whether validated sensor data exists for input into the parameter field 306 of the task template 301. For example, the validated sensor data may include sensor data determined to be within a non-anomalous range of values for a sensor. If the validated sensor data exists, the task list generation engine 114 inputs the validated sensor data into the parameter field 306. If the task list generation engine 114 determines that validated sensor data does not exist for a particular parameter, the task list generation engine 114 may determine whether estimated sensor data exists for the particular parameter. For example, a sensor analytics machine learning model may estimate an unvalidated sensor value based on a learned correspondence between a sensor having an unvalidated value and other sensors. If an estimated sensor value exists for the parameter, the task list generation engine 114 inputs the estimated sensor value into the parameter field 306. If neither a validated nor an estimated sensor value exists for a parameter associated with the task template, the task list generation engine 114 may generate a human-readable explanation indicating that the task cannot be performed and the data required to complete the task. Additionally or alternatively, the task list generation engine 114 may determine whether one or more alternative tasks may be substituted for the task that cannot be performed. For example, the task list generation engine 114 may apply a set of rules that specify that if a task cannot be performed, one or more alternative tasks may be performed to infer data about the task that cannot be performed. According to one example, a task template may require a specific gaze of a component by an operator when a particular parameter has a particular value. However, the value of a particular parameter may not be validated or estimated.Thus, the task list generation engine 114 may substitute two alternative tasks for a task having parameters that may not be validated or estimated. The two alternative tasks may include measurements of associated components based on parameters that may be validated or estimated. Values for parameters that may not be validated or estimated may be inferred from the two alternative tasks. Alternatively, the alternative tasks may achieve a desired result without inferring values for parameters that may not be validated or estimated.
[0034] In one embodiment, a human expert and / or operator generates the task template 118. Additionally, the task list generation engine 114 may implement a machine learning model to identify an existing task template for association with the identified root cause. Additionally or alternatively, the task list generation engine 114 may include a machine learning model for generating a new task template based on the newly identified root cause. For example, the task generation machine learning model may be trained with a dataset including (a) the root cause of the triggering event and (b) a set of task templates associated with the root cause (including components, actions, entities, dependencies, and parameters associated with the task template). Upon receiving a current root cause with specific characteristics, including specific components and event types, the task generation machine learning model may generate a new task template associated with the specific components and event types, which may specify the task to be performed, the type of entity on which the task should be performed, the task's relationship to other tasks, and any parameters required to complete the task.
[0035] The task list generation engine 114 may arrange the set of retrieved task templates 118 into a list by (a) the priority level of the task and (b) the dependency of the task on other tasks. In particular, a task that depends on another task to be completed is arranged on the task list lower than the parent task. In addition, if one task does not have a dependency on another, the system may arrange the tasks by a selected priority. For example, the system may apply a rule that computer-performed tasks should be given priority over human-performed tasks when ordering the task list, unless a dependency exists. In one embodiment, the task list may include a parallel list of tasks. For example, a vehicle power-down sequence may require an ordered sequence of tasks to be performed by two different humans. The task list generation engine 114 may represent the separate sequences of tasks to be performed by the two humans as parallel lists of tasks.
[0036] The task list execution engine 115 executes, manages, or tracks tasks in the task list. For example, the task list execution engine 115 may include an application that causes the user interface 120 to generate a graphical user interface (GUI) that includes a representation of one or more tasks on the task list. The task list execution may directly monitor the system 102 to determine whether a task is executed. For example, the task list execution engine 115 may display an interface element that represents a task for an operator to turn off power to a component. The task list execution engine 115 may detect whether the power has been turned off. The task list execution engine 115 may change the representation of the interface element to indicate that it has detected that the power is turned off. For example, the task list execution engine 115 may gray out the representation of the task or display a check mark next to the representation of the task. Additionally or alternatively, the task list execution engine 115 may allow an operator to interact with the representation of the task in the user interface 120 to indicate completion of the task. Upon completion of a task, the task list execution engine 115 may highlight the next task in the task list to be executed. Additionally, the task list execution engine 115 may display the tasks in the task list to be executed by the computer. The task list execution engine 115 may initiate execution of a computer-executed task based on detecting completion of any prerequisite tasks without human intervention. For example, if a task list includes a human-executed task to obtain measurements and then a computer-executed task to analyze system performance using the measurements, the task list execution engine 115 may (a) wait to execute the computer-executed task until an operator inputs measurements from the prerequisite tasks, and (b) initiate the computer-executed task without human intervention upon detecting that the operator has input the measurements.
[0037] The task list execution engine 115 may further provide functionality via the GUI of the user interface 120 to allow an operator to modify one or more tasks. For example, the task list generation engine 114 may generate a task list for flushing data storage components in a data center. The task list execution engine 115 may display the task list via the GUI of the user interface 120. The operator may determine that the displayed task list includes a task that will cause significant disruption to a customer. The operator may modify one or more tasks using the GUI to add a condition that the task should not be performed during a time period known to be a high customer usage time. Alternatively, the operator may modify the task list to include a new task requested by the customer. For example, the customer may request that the customer be notified of any potential disruption. The operator may add a task that requires the operator to send a notification and receive permission to perform the remaining tasks in the task list. According to another example, the organization that maintains the monitored system 102 may have safety protocols required prior to performing a task. The task list execution engine 115 may allow a user to add new tasks, including tasks specified by safety protocols, using the GUI of the user interface 120.
[0038] In one embodiment, the task list execution engine 115 determines whether the task list is completed. If the operator indicates that the task list cannot be completed, the task list execution engine 115 may provide information associated with the task list and any tasks that cannot be completed to the causal analysis engine 113. The causal analysis engine 113 may determine whether a detected triggering event in the monitored system 102 is associated with a different root cause than a previously identified root cause based on the completed and incomplete tasks. If the causal analysis engine 113 identifies a different root cause associated with the triggering event, the task list generation engine 114 may generate a new task list to be executed, monitored, and tracked by the task list execution engine 115.
[0039] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
[0040] In one or more embodiments, system 100 may include more or fewer components than those illustrated in FIG. 1. The components illustrated in FIG. 1 may be local or remote from one another. The components illustrated in FIG. 1 may be implemented in software and / or hardware. Each component may be distributed across multiple applications and / or machines. Multiple components may be combined into one application and / or machine. Operations described with respect to one component may instead be performed by another component.
[0041] Additional embodiments and / or examples relating to computer networks are described below in Section 6, entitled "Computer Networks and Cloud Networks."
[0042] In one or more embodiments, the data repository 116 is any type of storage unit and / or device for storing data (e.g., a file system, a database, a collection of tables, or any other storage mechanism). Additionally, the data repository 116 may include multiple different storage units and / or devices. The multiple different storage units and / or devices may or may not be of the same type or located at the same physical site. Additionally, the data repository 116 may be implemented or executed on the same computing system as the system monitor 110. Alternatively, or in addition, the data repository 116 may be implemented or executed on a computing system separate from the system monitor 110. The data repository 116 may be communicatively coupled to the system monitor 110 via a direct connection or via a network.
[0043] Information describing historical sensor data, task templates, and task lists may be embodied throughout any of the components in system 100. However, this information is illustrated in data repository 116 for purposes of clarity and illustration.
[0044] In one or more embodiments, the system monitor 110 refers to hardware and / or software configured to perform operations described herein for remediating a triggering event detected in a monitored system using a filtered data set including validated and estimated sensor data. An example of an operation for remediating a triggering event detected in a monitored system using a filtered data set including validated and estimated sensor data is described below with reference to Figures 2A and 2B.
[0045] In one embodiment, the system monitor 110 is implemented on one or more digital devices. The term "digital device" generally refers to any hardware device that includes a processor. A digital device may refer to a physical device that runs an application or a virtual machine. Examples of digital devices include computers, tablets, laptops, desktops, netbooks, servers, web servers, network policy servers, proxy servers, general purpose machines, function-specific hardware devices, hardware routers, hardware switches, hardware firewalls, hardware firewalls, hardware network address translators (NATs), hardware load balancers, mainframes, televisions, content receivers, set-top boxes, printers, mobile handsets, smartphones, personal digital assistants ("PDAs"), wireless receivers and / or transmitters, base stations, communication management devices, routers, switches, controllers, access points, and / or client devices.
[0046] In one or more embodiments, user interface 120 refers to hardware and / or software configured to facilitate communication between a user and system monitoring device 110. Interface 120 renders user interface elements and receives input via user interface elements. Examples of interfaces include graphical user interfaces (GUIs), command line interfaces (CLIs), tactile interfaces, and voice command interfaces. Examples of user interface elements include check boxes, radio buttons, drop-down lists, list boxes, buttons, toggles, text fields, date and time selectors, command lines, sliders, pages, and forms.
[0047] In one embodiment, different components of interface 120 are specified in different languages: The behavior of user interface elements is specified in a dynamic programming language, such as JavaScript; The content of user interface elements is specified in a markup language, such as HyperText Markup Language (HTML) or XML User Interface Language (XUL); The layout of user interface elements is specified in a style sheet language, such as Cascading Style Sheets (CSS); Alternatively, interface 120 is specified in one or more other languages, such as Java, C, or C++.
[0048] 3. Fix system anomalies based on machine learning 2A and 2B illustrate an example set of operations for repairing a system anomaly based on a machine learning model by generating and implementing a task list, according to one or more embodiments. One or more of the operations illustrated in FIG. 2A and 2B may be modified, rearranged, or removed altogether. Thus, the particular sequence of operations illustrated in FIG. 2A and 2B should not be construed as limiting the scope of one or more embodiments.
[0049] The system obtains current sensor data (operation 202). The current sensor data includes measurements from a plurality of sensors monitoring one or more components of a monitored system. Examples of monitored systems include vehicles, data centers, computer networks, manufacturing facilities, medical facilities, and power generation facilities. Monitored systems may include any number and types of components, including structural components, motion components, power generation components, computational components, and data storage components. Sensors include physical sensors, such as thermal sensors, visual sensors, and light sensors. Sensors include software-based sensors, such as applications, clients, or programs that monitor computing elements in a computer or a network of computing devices, network devices, and storage devices.
[0050] The system validates the sensor data (operation 204). Validating the sensor data includes determining that the sensor data meets predetermined criteria. For example, a machine learning model may be trained to identify correlations between sensors. Validating the sensor data may include determining that current sensor data values match the learned correlations of the machine learning model. Additionally, or alternatively, validating the sensor data may include determining that values for the sensors are within a previously determined range of non-anomalous values for the sensors.
[0051] In one embodiment, the machine learning model is a deterministic machine learning model, such as the MSET model. The MSET model is trained on a set of sensor values that includes the full range of non-anomalous sensor values for the set of sensors. Applying the MSET model to the current sensor data, the system validates the sensor data by (a) determining that each sensor value is within a previously learned range of non-anomalous values for the respective sensor, and (b) determining that the sensor data values are consistent with learned correlations with other sensor data values.
[0052] The system identifies one or more sensor values that may not be validated (operation 206). An unvalidated sensor value may include a sensor value that does not match a learned correlation between the sensor and other sensors. For example, a machine learning model may learn that three sensors have output values that follow the same course as each other. The system may detect that a current sensor value for one of the sensors may not be validated based on detecting that the current sensor value for that sensor does not follow the same course as the output values for the other two sensors.
[0053] In one embodiment where the machine learning model is a deterministic model such as the MSET model, the machine learning model may generate, for each sensor value, a delta value that represents the difference between the current sensor value and the predicted value for the sensor. The MSET model may determine that the sensor value may not be validated based on determining that (a) the sensor value is outside a range of non-anomalous values for the particular sensor, and (b) the sensor value varies from the predicted value by more than a predetermined threshold. According to one example, the MSET model applies a Sequential Probability Ratio Test (SPRT) algorithm to the sensor value and the predicted sensor value to determine whether the sensor value deviates from the predicted value by more than a predetermined threshold. The SPRT algorithm detects abnormal deviations from normal operation by calculating a cumulative sum of log-likelihood ratios for each successive residual between the measured signal and the estimated value, and comparing the cumulative sum to a threshold to determine at which a fault is detected.
[0054] According to one example, the MSET model may identify correlations between the values of three sensors. The MSET model may determine that a third sensor should have a particular value based on the values of two of the sensors. The MSET model may determine that the third sensor cannot be validated if the current value for the third sensor varies from the particular value by more than a predefined threshold. In addition, the MSET model may determine that the third sensor has a predefined range of non-abnormal operating values. The MSET model may determine that the third sensor cannot be validated if the current sensor value for the third sensor is outside the predefined range of non-abnormal operating values.
[0055] The system determines whether the unvalidated sensor data can be estimated (operation 208). In one embodiment, the system trains a machine learning model with the dataset of sensor values to identify correlations between sensors. The system can determine whether the unvalidated sensor values can be estimated based on the learned correlations of the sensor values by the machine learning model.
[0056] Based on determining that sensor values that cannot be validated may be estimated, the system generates estimated values for any sensor data that cannot be validated but may be estimated (operation 210). For example, in one embodiment where the sensor analysis model is an MSET model, the MSET model generates estimated values for the sensor based on learned correlations between the sensor and other sensors.
[0057] In one embodiment, the system trains and applies one machine learning model to the sensor data to validate the sensor data, and trains and applies another machine learning model to the sensor data to estimate sensor values that may not be validated. For example, the system may train a machine learning model using a set of historical sensor data to learn correlations between sensor values. The system provides current sensor data to the trained machine learning model to determine whether correlations between current sensor values match the learned correlations between historical sensor values. The trained machine learning model validates the sensor values based on determining that the values match the learned correlations. The machine learning model determines that a subset of the values may not be validated based on determining that a subset of the values does not match the learned correlations with other sensor values. The system trains another machine learning model to learn correlations between sensor values in the set of historical sensor data and to predict sensor values based on other sensor values in the set of historical sensor data. The system provides current sensor data to the machine learning model. The machine learning model predicts a value for the sensor value based on the values of other sensor values in the current sensor data. The system substitutes predicted values produced by the second machine learning model for measured values identified by the first machine learning model as values that cannot be validated.
[0058] In an alternative embodiment, the system uses the same machine learning model to both validate sensor data and estimate sensor data values that cannot be validated. For example, the MSET machine learning model detects when sensor values vary from non-abnormal operating ranges and generates predictions for sensor values based on learned correlations between sensor values. The MSET model replaces sensor values that cannot be validated with predicted sensor values.
[0059] Based on the validated sensor data and the estimated sensor data, if any, the system generates a filtered set of sensor data (operation 212). The system excludes sensor data, if any, that cannot be validated from the filtered set of sensor data.
[0060] Using the filtered set of sensor data, the system detects events associated with the monitored system (operation 214). In one embodiment, the trained sensor analytics machine learning model identifies anomalies in the sensor data. The system may detect a triggering event based on detecting one or more conditions associated with the current sensor data. For example, the system may detect a predefined number of sensor data values that exceed a threshold value. The system may detect a particular sensor data value that exceeds a threshold value. Additionally or alternatively, the system may detect a combination of sensor values that amount to a triggering event even if one or more of the sensor values are within a non-anomalous sensor value range for a particular sensor. In one embodiment, the system compares the estimated value for the sensor with the current value for the sensor to detect whether the sensor value is anomalous and whether the sensor value amount to a triggering event. In other words, the system may use a combination of the filtered set of sensor data and the current sensor data to detect whether a triggering event has occurred in the monitored system.
[0061] The system identifies a root cause of the detected triggering event (operation 216). The system may analyze outputs from one or both of the machine learning model and the system monitoring sensors to identify the root cause of the triggering event. In one embodiment, the system applies a trained causal analysis machine learning model to the filtered set of sensor data to identify the root cause of the triggering event. For example, a root cause of an overheating component may be a failure of a power regulator on the component. The machine learning model may be trained on a dataset that includes (a) historical sensor data and (b) historically identified causes for the triggering event in the historical data to identify a root cause of the triggering event associated with the current sensor data.
[0062] The system identifies a set of tasks associated with the root cause (operation 218). According to one embodiment, the system generates a set of queries to a database of task templates. The system generates the set of queries based on one or more of: (a) the identified root cause, (b) the identified triggering event, and (c) one or both of the current sensor data and the filtered set of sensor data. The task template includes the action to be performed and any dependencies of the task on any other tasks. The task template further includes editable fields that the system populates according to the specifications of the monitored system to generate a set of tasks. For example, the system may populate a specific model or description of a system component in the "Component" field. The system may populate the "Entity to perform action on" field with the name of an operator, the operator's title, or the name of a computer that is selected to perform the action.
[0063] The system instantiates task parameters for a set of tasks (operation 220). The set of tasks includes parameters that are populated by the system based on current sensor data. The system may populate the parameters based on validated sensor data. For example, a task may require a data latency measurement. Another task may require an environmental temperature measurement. The system populates the task parameters based on measurements in the current sensor data.
[0064] The system determines whether the set of tasks includes any missing parameters (operation 222). For example, the system may determine that a task includes a parameter associated with a current sensor value that cannot be validated. The sensor value that cannot be validated results in a missing task parameter.
[0065] The system determines whether, for any missing parameters, the parameters can be estimated (operation 224). In an embodiment in which the system includes a sensor analytics machine learning model that learns correlations between different sensors, the system may determine whether the sensor analytics machine learning model generates output values that are estimates of sensor values that cannot be validated. Additionally or alternatively, the system may identify any other sensor values that may be used to estimate the missing task parameters. For example, the system may determine that a first sensor value of a first component may be substituted into a formula to calculate an estimate for a second sensor having a second value that cannot be validated. In yet another embodiment, the system may determine whether any tasks may be substituted for a task having a missing parameter. For example, the system may determine that a data backup type operation may be initiated either by a first task associated with the first component or by a second task associated with the second component. If a required task parameter of the first task is missing, the system determines that a second task may be substituted for the first task to achieve the desired result.
[0066] If the system determines that the missing parameters cannot be estimated, the system generates an explanation for the missing parameters (operation 226). The explanation may be a human-understandable explanation. The explanation may be selected from among a set of explanation templates. The system may fill in fields of the explanation template to generate the human-understandable explanation. For example, the explanation may indicate that a particular task cannot be completed without a particular measurement, and that the sensor data associated with the particular measurement cannot be validated. In one embodiment, the human-understandable explanation is presented to the operator via a GUI. If a task is not essential to achieving a goal, the system may give the operator the option of generating a task list without the task. Additionally or alternatively, the system may provide interface elements to enable the operator to provide the necessary information to complete the task. For example, a user may provide the missing parameter information via a GUI.
[0067] If the system determines that the missing parameters can be estimated, the system instantiates the parameters based on the estimated parameter values (operation 228). In an embodiment where the system includes a sensor analytics machine learning model that learns correlations between different sensors, the system estimates the parameters based on the estimated sensor values for the sensor values that cannot be validated. In an example where a first sensor value of a first component is substituted into a formula to calculate an estimate for a second sensor having a second value that cannot be validated, the system may estimate the missing parameters based on the first sensor value instead of the second sensor value. According to an alternative embodiment, the system substitutes one task for another as an alternative to estimating the missing parameters. In an example where two tasks can achieve the desired result, the system may substitute the second task for the first task if the first task includes the missing parameter.
[0068] The system arranges the set of tasks in a task list (operation 230). The system arranges the set of tasks in the list based on one or both of dependencies between the tasks and the priority levels of the tasks. The system arranges the tasks according to dependencies specified in the task template. For example, a task may specify that the task must always be preceded by a particular task. Furthermore, the task may specify that the task must be passed to another particular task. A series of two or more tasks may specify that no intervening tasks may be performed between the two or more tasks. A task may specify a time period for performing the task. A task may specify any related tasks, such as tasks that must be performed simultaneously by another entity. The system may further arrange the tasks in the task list according to the priority level of the task. The priority level may be set by an operator. According to one example, an operator may identify a particular type of task as having a high priority. Alternatively, an operator may identify a task associated with a particular component as having a high priority. The system may arrange two tasks that do not have a dependency according to the identified priority.
[0069] The system begins executing the task list (operation 232). The system may provide the task list to the operator via a user interface GUI, via print media, or by any other means. In an embodiment in which the system provides the task list to the operator via a GUI, the system may display the task list and track completion of the task list. The system may update the display of the task list based on detecting the completion of a task on the task list. During the execution of the task list, the system may execute any computer-implemented tasks without human intervention based on detecting the completion of a nearby predecessor task on the task list.
[0070] The system determines whether the task list can be completed (operation 234). For example, a user may generate an input via the GUI that one or more tasks are missing from the task list. A user may generate an input indicating that two or more tasks on the task list are redundant. The system may identify one or more tasks that cannot be completed. For example, the environment that would require the task to be performed may not be accessible by a human or computer. For example, physical components of a vehicle may not be accessible. Also, a task to validate a set of data may not be accessible due to security protocols.
[0071] Based on determining that the list of tasks cannot be completed, the system may modify one or both of the task parameters and the specific tasks associated with the particular root cause (operation 236). For example, if the user indicates that a task cannot be performed due to the user's inability to access components for taking measurements, the system may attempt to identify alternative measurements to substitute for the initial measurements or alternative tasks to substitute for the initial tasks. If the user indicates that a task is missing, the system may generate new dependencies for an existing task template or an entirely new task template based on the user input. Based on updates to the set of tasks to be performed, the system determines whether the initially identified root cause is still determined to be the root cause of the triggering event. For example, the system may determine that the user's inability to perform a task indicates that another root cause different from the initially identified root cause may be the actual cause of the triggering event. The system repeats the process of identifying root causes (operation 216) and identifying tasks associated with the root causes (operation 218).
[0072] 4. Training the machine learning model 4A and 4B illustrate an example set of operations for training a machine learning model to validate sensor data, to identify sensor data that cannot be validated, and to estimate values for sensor data that cannot be validated, according to one or more embodiments. One or more of the operations illustrated in FIG. 4A and 4B may be modified, rearranged, or omitted altogether. Thus, the particular sequence of operations illustrated in FIG. 4A and 4B should not be construed as limiting the scope of one or more embodiments.
[0073] FIG. 4A illustrates an example set of operations for training a machine learning model, such as a neural network or a support vector machine (SVM), using stochastic optimization of weights, according to one or more embodiments.
[0074] The system obtains a set of historical data including historical sensor values and identified root causes associated with particular combinations of sensor values (operation 402). Examples of sensor values include sensor signals output from physical sensors, such as thermal, optical, strain, and positioning sensors, as well as sensors implemented as software executing on a processing device, such as programs for tracking data volume, bandwidth, data latency, and dropped data packet rates. Examples of root causes include physical malfunction of a component, circuitry failure, electrical component failure, loss of power to a component, and misconfiguration of a software component resulting in a measurable data transmission or data storage failure.
[0075] The system generates a training dataset from the set of historical data (operation 404). The training dataset includes sensor values for a set of sensors monitoring the system. The training dataset may further include labels of root causes in one or more components of the system being monitored. For example, one data point in the set of training data includes a set of sensor data values and a label that identifies one or more causes of anomalous sensor data values in the set of sensor data values. Another data point in the set of training data includes another set of sensor data values and another label that identifies one or more causes of the anomalous sensor data values.
[0076] The system applies a machine learning algorithm to the training data set to train a machine learning model to learn relationships or correlations between the sensors and to predict root causes associated with different combinations of sensor values (operation 406). One or more embodiments include multiple different machine learning models to perform different decisions or predictions. For example, the system may train one machine learning model to predict root causes associated with a particular set of current sensor values. The system may train another machine learning model to identify relationships or correlations between different sensors. The machine learning model may be trained to identify a subset of sensor values in the current set of sensor values that may not be validated. The machine learning model identifies a subset of sensors that may not be validated based on determining that a particular value for the subset of sensors does not match a learned relationship between the sensors. The system may train yet another machine learning model to generate predicted or estimated values for the sensors. The system trains a machine learning model to learn relationships or correlations between sensors. The machine learning model predicts a value for a particular sensor based on values of other sensors associated with the current set of sensor values.
[0077] In one embodiment, training the machine learning model includes receiving feedback on the root cause identification generated by the machine learning model (operation 408). For example, the system may display the root cause identification generated by the machine learning model on a GUI. The system may receive one or more inputs for modifying the identified root cause. The system updates the machine learning model based on the feedback (operation 410).
[0078] FIG. 4B illustrates an example set of operations for training a deterministic-type machine learning (ML) model, such as the MSET model, according to one or more embodiments.
[0079] The system obtains a training data set (operation 412) including ranges of sensor values under non-anomalous conditions for a set of sensors monitoring the monitored system. The set of training data may include time series data. The time series data includes a plurality of signals generated by a plurality of sensors monitoring the monitored system. In one embodiment, the output levels of the plurality of signals in the training data include a range of values that are all non-anomalous values. The set of training data may be selected to include sensor data over time in which the sensor values for each sensor vary within a range of values defined as non-anomalous for the respective sensor.
[0080] The system applies a machine learning algorithm to the training dataset to train a machine learning model to identify (a) a range of non-anomalous values for each sensor and (b) relationships between sensor values for the set of sensors (operation 414). A deterministic type ML model is trained using the non-anomalous training dataset to model non-anomalous operation of the monitored system. In one embodiment, the deterministic ML model includes an abnormal signal prediction model to identify time series signal variations from predefined operating parameters. The abnormal signal prediction model predicts values for the set of sensors based on learned relationships with other sensors and identifies when the predicted value for a sensor differs from the current sensor value for the sensor.
[0081] One or more of the operations illustrated in Figure 4A may be modified, rearranged, or removed altogether, and thus the particular sequence of operations illustrated in Figure 4A should not be construed as limiting the scope of one or more embodiments.
[0082] 5. Example embodiment Detailed examples are described below for clarity. The components and / or operations described below should be understood as one particular example that may not be applicable to certain embodiments. Thus, the components and / or operations described below should not be construed to limit the scope of any of the claims.
[0083] 5A and 5B illustrate a system 500 according to an example embodiment. The system 500 includes a data center 510 that is monitored by a sensor 513. The data center 510 includes computing devices 511a, 511b, and 511c. Cooling units 512a, 512b, and 512c cool the computing devices 511a, 511b, and 511c, respectively.
[0084] Sensors 510 provide sensor data to data center monitor 520. Data center monitor includes MSET ML model 521 for analyzing the sensor data. MSET ML model 521 has been trained with a training dataset of sensor values associated with sensors 513 to model non-anomalous sensor values associated with non-anomalous operation of data center 510.
[0085] MSET ML model 521 generates predicted values for sensor 513. Based on the predicted values, MSET ML model 521 identifies expected current sensor values 522a, such as within a predefined range from the predicted sensor value. Additionally, MSET ML model 521 identifies current sensor values 522b that are anomalous, i.e., have values that are (a) outside a predefined range of predicted values or (b) outside a learned range of non-anomalous values for the sensor. For sensor values identified as anomalous based on being outside a predefined range of predicted values, MSET ML model 521 generates estimated sensor data values 522c by replacing the current sensor value with the predicted current sensor value. For example, if a sensor is drifting such that the output value loses correlation with the sensed characteristic over time, MSET ML model 521 replaces the inaccurate output sensor value resulting from the drift with a predicted sensor value based on learned correlations between the sensor and other sensors.
[0086] Causal analysis engine 523 analyzes sensor data output from MSET ML model 521 to identify a root cause associated with the detected triggering event. If MSET ML model 521 determines, based on learning correlations between sensor values, that a particular current sensor value (a) is consistent with learned relationships with other sensors, and (b) is at an abnormal level outside of non-abnormal levels, causal analysis engine 523 uses the sensor value. Causal analysis engine 523 determines that the sensor is likely to accurately track an abnormal state of a monitored component in data center 510. If MSET ML model 521 determines, based on learning correlations between sensor values, that a particular current sensor value (a) is not consistent with learned relationships with other sensors, causal analysis engine 523 uses a predicted sensor value for the particular sensor as a substitute for the current sensor value. Causal analysis engine 523 determines that the sensor is likely to be in a bad state and that the sensor output is likely not reflective of the state of the monitored component in data center 510.
[0087] In one or more embodiments, the causal analysis engine 523 exploits the deterministic nature of the MSET ML model to perform a traceback operation to identify the root cause of the event indicated by the anomalous signal values. In particular, the MSET ML model implements a mathematical structure that will return the same set of output values for a given set of input values. As a result, the MSET ML model is reversible. A given set of input values may be determined based on a respective set of output values. Thus, the causal analysis engine 523 performs a traceback operation on the signals output by the MSET model to identify a particular subset of the input signals that corresponds to the detected anomaly.
[0088] In the example embodiment of FIG. 5A , causal analysis engine 523 identifies cooling unit failure 524 as a root cause of the detected triggering event. Task list generation engine 525 identifies a set of task templates 531, stored in data repository 530, associated with cooling unit failure 524. In the embodiment of FIG. 5A , task list generation engine 525 references task to cause mapping 532 to identify the particular set of tasks associated with cooling unit failure 524. Task list generation engine 525 generates task list 526, which represents an ordered sequence of tasks 526a, 526b, 526c, and 526d to be accomplished in response to cooling unit failure 524.
[0089] The task list generation engine determines whether the set of tasks 526a-526d includes any missing parameters. For example, the task list generation engine may determine that task 526b “Turn off cooling unit 1” requires a parameter indicating the current temperature of computing unit 511a and a parameter indicating the temperature of cooling unit 1 512a. If the system determines that the sensor data for the current temperature of computing device 511a cannot be validated, task list generation engine 525 replaces the current sensor data for the temperature of computing device 511a with predicted sensor data generated by MSET model 521.
[0090] The task list generation engine 525 arranges a set of tasks 526a-526d based on the dependencies between the tasks and the priority levels of the tasks. For example, task 526a is a computer-executed task that does not have any dependencies on other tasks. The task list generation engine 525 applies rules that instruct the task list generation engine 525 to assign a higher priority to computer-executed tasks than human-executed tasks. Task 526c, for measuring the temperature in cooling unit 2 512b by a human, has a defined dependency relationship to task 526b. Thus, the task list generation engine 525 arranges task 526c to be executed subsequent to task 526b.
[0091] 5B, the task execution engine begins executing the task list 526. The task execution engine 527 causes the user interface 528 to display a GUI depicting the task list 526. As the task is executed, the task execution engine 527 may cause the user interface 528 to modify the depiction of the task in the GUI to indicate that the task is completed. For example, the task execution engine 527 may cause the user interface 528 to modify the depiction of the task in the GUI to minimize the completed task, change the color of the completed task, or display a check mark next to the completed task. Additionally, the task execution engine 527 may cause the user interface 528 to modify the depiction of the task in the GUI to display details associated with the current task while minimizing other tasks. The task execution engine 527 may start the task 526a without human input based on applying a rule that the computer executes when a preceding task is completed without waiting for human input.
[0092] The task execution engine 527 receives user input via the user interface 528 indicating that a task has been completed. The task execution engine 527 may even detect whether a task has been completed independent of user input via the user interface 528. For example, upon detecting that the power to cooling unit 1 512a has been turned off by the sensor 513, the task execution engine 527 may cause the user interface 528 to modify the depiction of the task in the GUI to indicate that task 526b has been completed, without human input via the user interface 528.
[0093] Additionally, if the user determines that the task list 526 requires modification, such as adding or removing a task, or modifying a task element, such as the entity on which a task should be performed, the user may input the changes via a GUI displayed in the user interface 528. The task execution engine 527 may modify the task list 526 based on the user input.
[0094] 6. Computer Networks and Cloud Networks In one or more embodiments, a computer network provides connectivity between a set of nodes. The nodes can be local to each other and / or remote from each other. The nodes are connected by a set of links. Examples of links include coaxial cable, unshielded twisted cable, copper cable, optical fiber, and virtual links.
[0095] A subset of nodes implements a computer network. Examples of such nodes include switches, routers, firewalls, and network address translators (NATs). Another subset of nodes uses a computer network. Such nodes (further referred to as "hosts") may execute client processes and / or server processes. A client process makes a request for a computing service (such as running a particular application and / or storing a particular amount of data). A server process responds by performing the requested service and / or returning the corresponding data.
[0096] A computer network may be a physical network, including physical nodes connected by physical links. A physical node may be any digital device. A physical node may be a function-specific hardware device, such as a hardware switch, a hardware router, a hardware firewall, and a hardware NAT. Additionally or alternatively, a physical node may be a general-purpose machine configured to run various virtual machines and / or applications that perform respective functions. A physical link is a physical medium that connects two or more physical nodes. Examples of links include coaxial cable, unshielded twisted cable, copper cable, and optical fiber.
[0097] A computer network may be an overlay network. An overlay network is a logical network realized on top of another network (such as a physical network). Each node in the overlay network corresponds to a respective node in the underlying network. Thus, each node in the overlay network is associated with both an overlay address (for addressing the overlay node) and an underlay address (for addressing the underlay node that realizes the overlay node). The overlay nodes may be digital devices and / or software processes (such as virtual machines, application instances, or threads). The links connecting the overlay nodes are realized as tunnels through the underlying network. The overlay nodes at either end of the tunnel treat the underlying multi-hop path between them as a single logical link. Tunneling is performed by encapsulation and decapsulation.
[0098] In one embodiment, a client may be local to and / or remote from a computer network. A client may access a computer network over a private network or another computer network, such as the Internet. A client may communicate a request to a computer network using a communication protocol, such as the Hypertext Transfer Protocol (HTTP). The request is communicated through an interface, such as a client interface (such as a web browser), a program interface, or an application programming interface (API).
[0099] In one embodiment, a computer network provides connectivity between clients and network resources. The network resources include hardware and / or software configured to execute server processes. Examples of network resources include processors, data storage, virtual machines, containers, and / or software applications. The network resources are shared among multiple clients. The clients request computing services from the computer network independently of each other. The network resources are dynamically allocated to requests and / or clients on an on-demand basis. The network resources allocated to each request and / or client may be scaled up or down based on, for example, (a) the computing services requested by a particular client, (b) the aggregated computing services requested by a particular tenant, and / or (c) the aggregated computing services requested of the computer network. Such a computer network may be referred to as a "cloud network."
[0100] In one embodiment, a service provider offers a cloud network to one or more end users. A variety of service models may be enabled by the cloud network, including, but not limited to, Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). In SaaS, the service provider offers end users the ability to use the service provider's applications running on the network resources. In PaaS, the service provider offers end users the ability to deploy custom applications onto the network resources. The custom applications may be created using programming languages, libraries, services, and tools supported by the service provider. In IaaS, the service provider offers end users the ability to provision the processing, storage, network, and other basic computing resources provided by the network resources. Any optional applications, including operating systems, may be deployed onto the network resources.
[0101] In one embodiment, a computer network may implement various deployment models, including but not limited to private cloud, public cloud, and hybrid cloud. In a private cloud, network resources are provisioned for exclusive use by a particular group of one or more entities (the term "entity" as used herein refers to a corporation, organization, individual, or other entity). The network resources may be local to and / or remote from the premises of the particular group of entities. In a public cloud, cloud resources are provisioned for multiple entities (further referred to as "tenants" or "customers") that are independent of each other. The computer network and its network resources are accessed by clients corresponding to different tenants. Such a computer network may be referred to as a "multi-tenant computer network." Different tenants may use the same particular network resources at different times and / or at the same time. The network resources may be local to and / or remote from the premises of the tenants. In a hybrid cloud, the computer network includes a private cloud and a public cloud. The interface between the private cloud and the public cloud allows for data and application portability. Data stored in the private cloud and data stored in the public cloud may be exchanged through the interface. Applications implemented in the private cloud and applications implemented in the public cloud may have dependencies on each other. Calls from applications in the private cloud to applications in the public cloud (and vice versa) may be made through the interface.
[0102] In one embodiment, tenants of a multi-tenant computer network are independent of one another. For example, the business or operations of one tenant may be separate from the business or operations of another tenant. Different tenants may desire different network requirements for the computer network. Examples of network requirements include processing speed, amount of data storage, security requirements, performance requirements, throughput requirements, latency requirements, resiliency requirements, quality of service (QoS) requirements, tenant isolation, and / or integrity. The same computer network may be required to achieve different network requirements desired by different tenants.
[0103] In one or more embodiments, in a multi-tenant computer network, tenant isolation is implemented to ensure that applications and / or data of different tenants are not shared with each other. Various tenant isolation techniques may be used.
[0104] In one embodiment, each tenant is associated with a tenant ID. Each network resource in a multi-tenant computer network is tagged with a tenant ID. A tenant is granted access to a particular network resource if the tenant and the particular network resource are associated with the same tenant ID.
[0105] In one embodiment, each tenant is associated with a tenant ID. Each application implemented by the computer network is tagged with a tenant ID. Additionally or alternatively, each data structure and / or data set stored by the computer network is tagged with a tenant ID. A tenant is granted access to a particular application, data structure, and / or data set if the tenant and the particular application, data structure, and / or data set are associated with the same tenant ID.
[0106] As an example, each database implemented by a multi-tenant computer network may be tagged with a tenant ID. Only the tenant associated with the corresponding tenant ID may access the data of a particular database. As another example, each entry in a database implemented by a multi-tenant computer network may be tagged with a tenant ID. Only the tenant associated with the corresponding tenant ID may access the data of a particular entry. However, a database may be shared by multiple tenants.
[0107] In one embodiment, the subscription list indicates which tenants have permission to access which applications. For each application, a list of tenant IDs of tenants that are authorized to access the application is stored. A tenant is granted access to a particular application if the tenant's tenant ID is included in the subscription list that corresponds to the particular application.
[0108] In one embodiment, network resources (such as digital devices, virtual machines, application instances, and threads) corresponding to different tenants are isolated to tenant-specific overlay networks maintained by a multi-tenant computer network. As an example, a packet from any source device in a tenant overlay network may be transmitted to other devices in the same tenant overlay network. An encapsulation tunnel is used to prohibit any transmission from a source device on a tenant overlay network to a device in another tenant overlay network. Specifically, a packet received from a source device is encapsulated in an outer packet. The outer packet is transmitted from a first encapsulation tunnel endpoint (communicating with a source device in a tenant overlay network) to a second encapsulation tunnel endpoint (communicating with a destination device in a tenant overlay network). The second encapsulation tunnel endpoint decapsulates the outer packet to obtain the original packet transmitted by the source device. The original packet is transmitted from the second encapsulation tunnel endpoint to a destination device in the same specific overlay network.
[0109] 7. Miscellaneous; Extensions Embodiments are directed to systems involving one or more devices that include a hardware processor and are configured to perform any of the operations described herein and / or recited in any of the claims below.
[0110] In one embodiment, a non-transitory computer-readable storage medium includes instructions that, when executed by one or more hardware processors, cause the performance of any of the operations described herein and / or recited in any of the claims.
[0111] Any combination of the features and functionality described herein may be used by one or more embodiments. In the foregoing specification, the embodiments have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings should therefore be regarded in an illustrative rather than restrictive sense. The sole and exclusive indication of the scope of the invention, and what is intended by the applicants to be the scope of the invention, is the literal and equivalent scope of the set of claims issuing from this application, in the particular form in which the claims are issued, including any subsequent amendments.
[0112] 8. Hardware Overview According to one embodiment, the techniques described herein are realized by one or more special-purpose computing devices. The special-purpose computing devices may be hardwired to execute the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or network processing units (NPUs) that are permanently programmed to execute the techniques, or may include one or more general-purpose hardware processors that are programmed to execute the techniques according to program instructions in firmware, memory, other storage, or combinations. Such special-purpose computing devices may even combine custom hardwired logic, ASICs, FPGAs, or NPUs with custom programming to accomplish the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices, or any other devices that incorporate hardwired and / or program logic to implement the techniques.
[0113] 6 is a block diagram illustrating an exemplary computer system 600 upon which an embodiment of the present invention may be implemented. Computer system 600 includes a bus 602 or other communication mechanism for communicating information, and a hardware processor 604 coupled with bus 602 for processing information. Hardware processor 604 may be, for example, a general-purpose microprocessor.
[0114] Computer system 600 further includes a main memory 606, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 602 for storing information and instructions to be executed by processor 604. Main memory 606 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 604. Such instructions, when stored in a non-transitory storage medium accessible to processor 604, transform computer system 600 into a special purpose machine that is customized to perform operations specified in the instructions.
[0115] Computer system 600 further includes a read only memory (ROM) 608 or other static storage device coupled to bus 602 for storing static information and instructions for processor 604. A storage device 610, such as a magnetic disk or optical disk, is provided and coupled to bus 602 for storing information and instructions.
[0116] Computer system 600 may be coupled via bus 602 to a display 612, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 614, including alphanumeric and other keys, is coupled to bus 602 for communicating information and command selections to processor 604. Another type of user input device is a cursor control 616, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to processor 604, as well as for controlling cursor movement on display 612. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify a position in a plane.
[0117] Computer system 600 may implement the techniques described herein using customized hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with a computer system, causes or programs computer system 600 to be a special purpose machine. According to one embodiment, the techniques herein are performed by computer system 600 in response to processor 604 executing one or more sequences of one or more instructions contained in main memory 606. Such instructions may be read into main memory 606 from another storage medium, such as storage device 610. Execution of the sequences of instructions contained in main memory 606 causes processor 604 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.
[0118] The term "storage medium" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 610. Volatile media include dynamic memory, such as main memory 606. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid state drives, magnetic tape, or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROM, and EPROM, flash EPROM, NVRAM, any other memory chip or cartridge, content addressable memory (CAM), and ternary content addressable memory (TCAM).
[0119] Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise bus 602. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.
[0120] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 604 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 600 may receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector may receive the data carried in the infrared signal and appropriate circuitry may place the data on the bus 602. The bus 602 carries the data to main memory 606, from which the processor 604 retrieves and executes the instructions. The instructions received by main memory 606 may optionally be stored on a storage device 610 either before or after execution by the processor 604.
[0121] Computer system 600 further includes a communication interface 618 coupled to bus 602. The communication interface 618 provides a two-way data communication coupling to a network link 620 that is connected to a local network 622. For example, the communication interface 618 may be an Integrated Services Digital Network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, the communication interface 618 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. A wireless link may also be implemented. In any such implementation, the communication interface 618 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
[0122] Network link 620 typically provides data communication through one or more networks to other data devices. For example, network link 620 may provide a connection through local network 622 to a host computer 624 or to data equipment operated by an Internet Service Provider (ISP) 626. ISP 626 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the "Internet" 628. Local network 622 and Internet 628 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 620 and through communication interface 618, which carry the digital data to and from computer system 600, are example forms of transmission media.
[0123] Computer system 600 may send messages and receive data, including program code, through the network(s), network link 620 and communication interface 618. In the Internet example, a server 630 might transmit a requested code for an application program through Internet 628, ISP 626, local network 622 and communication interface 618.
[0124] The received code may be executed by processor 604 as it is received, and / or stored in storage device 610, or other non-volatile storage for later execution.
[0125] In the foregoing specification, embodiments of the present invention have been described with reference to numerous specific details that may vary from one implementation to another. The specification and drawings should therefore be regarded in an illustrative rather than restrictive sense. The sole and exclusive indication of the scope of the invention, and what is intended by the applicants to be the scope of the invention, is the literal and equivalent scope of the set of claims issuing from this application, in the particular form in which the claims are issued, including any subsequent amendments.
Claims
1. A non-transitory computer-readable medium containing instructions that, when executed by one or more hardware processors, cause operations to be performed, the operations including: training a first machine learning model based on historical sensor data obtained from a plurality of data sources; applying the first trained machine learning model to current sensor data detected by a plurality of data sources to identify a particular subset of the current sensor data that may not be validated based on data relationships corresponding to the historical sensor data; applying a second trained machine learning model to generate estimated sensor data for substituting for the particular subset of the current sensor data that may not be validated; and and analyzing the current sensor data with the estimated sensor data substituted for the particular subset of the current sensor data to generate a task list for remediating anomalous events.
2. 2. The non-transitory computer-readable medium of claim 1, wherein the task list specifies, for a particular task, (a) an entity that is to perform the particular task, (b) an action to be performed, and (c) a component of a monitored system on which the action is to be performed.
3. The non-transitory computer-readable medium of claim 1 , wherein the task list comprises an ordered sequence of two or more tasks.
4. At least one first task of the two or more tasks is a task to be performed by a first entity; 4. The non-transitory computer-readable medium of claim 3, wherein at least one second task among the two or more tasks is a task to be performed by a second entity different from the first entity.
5. the two or more tasks further specify a dependency of one task among the two or more tasks on another task among the two or more tasks; at least one task among the two or more tasks includes a dependency on another task among the two or more tasks; The non-transitory computer-readable medium of claim 3 , wherein the two or more tasks are arranged in a sequence according to the dependency relationship.
6. At least one first task of the two or more tasks is a task to be performed by a human; at least one second task of the two or more tasks is a task to be performed by a computer without human intervention; The instruction:
4. The non-transitory computer-readable medium of claim 3, further comprising: in response to detecting a completion of the at least one second task, generating a human-readable notification associated with the completion of the at least one second task.
7. The operation includes:
2. The non-transitory computer-readable medium of claim 1, further comprising identifying the anomalous event based on the current sensor data with the estimated sensor data substituted for the particular subset of the current sensor data.
8. 2. The non-transitory computer-readable medium of claim 1 , wherein the first trained machine learning model and the second trained machine learning model correspond to a same machine learning model.
9. 8. The non-transitory computer-readable medium of claim 7, wherein the same machine learning model is a Multivariate State Estimation Technique (MSET) model.
10. The operation includes: obtaining a training data set from the historical sensor data; training a second machine learning model to identify correlations between the plurality of data sources based on the training dataset; applying the second trained machine learning model to generate estimated sensor data for substituting for the particular subset of the current sensor data that may not be validated; the second trained machine learning model identifying the correlation between the particular subset of the current sensor data and another subset of the current sensor data being validated; and generating the estimated sensor data based on the correlation.
11. The non-transitory computer-readable medium of claim 1 , wherein the task list for repairing the anomalous event includes a task for repairing a root cause of the anomalous event.
12. The operation includes: In response to receiving an input to modify two or more tasks associated with the root cause, adding a new task to the two or more tasks; and Removing at least one task from among the two or more tasks. modifying the two or more tasks by performing one or both of: and regenerating the task list based on modifying the two or more tasks.
13. generating the task list, generating at least one query based on the root cause; and querying a set of stored task templates to identify two or more tasks that satisfy a query condition associated with the at least one query.
14. A non-transitory computer-readable medium containing instructions that, when executed by one or more hardware processors, cause operations to be performed, the operations including: training a machine learning model based on historical sensor data obtained from a plurality of sensors; applying the trained machine learning model to current sensor data detected by a plurality of sensors to identify a particular subset of the current sensor data that may not be validated based on data relationships corresponding to the historical sensor data; filtering out the particular subset of the current sensor data that may not be validated to obtain a filtered set of current sensor data that includes validated sensor data; performing an analysis on the filtered set of current sensor data that does not include the particular subset of current sensor data to identify issues to be remediated; and generating recommendations for remediating the identified problem areas based on the filtered set of the current sensor data.
15. The instruction: performing further operations including estimating a second subset of the current sensor data for use in the analysis in place of the particular subset of the current sensor data, the estimating being based on the data relationships corresponding to the historical sensor data; 15. The non-transitory computer-readable medium of claim 14, wherein the analysis is further performed on the estimated second subset of the current sensor data in addition to the filtered set of the current sensor data to identify the problem areas to be remedied.
16. Generating recommendations for remediating the issues includes:
20. The non-transitory computer-readable medium of claim 15, further comprising generating a task list including a plurality of tasks to be performed in sequence to repair the issue.
17. generating the task list, 17. The non-transitory computer-readable medium of claim 16, further comprising identifying a set of parameters required to generate the task list, the set of parameters including at least one value from the filtered set of current sensor data and at least one value from the estimated second subset of the current sensor data.
18. A method comprising the acts of any of claims 1 to 17.
19. one or more processors; and a memory storing instructions that, when executed by said one or more processors, cause said system to perform the operations recited in any of claims 1 to 17.
20. A system comprising means for performing the operations of any of claims 1 to 17.
Citation Information
Patent Citations
Inference Signal Generator for Instrumented Equipment and Processes
JP2004523735A
Abnormality detection / diagnostic method, abnormality detection / diagnostic system, abnormality detection / diagnostic program and company asset management / facility asset management system
JP2012137934A
