Operational data anomaly detection and repair

Abstract

Techniques for detecting and repairing abnormal intervals in time series data of a monitored device are disclosed. The system trains a machine learning model on a combination of real data obtained from a monitoring device and false data generated by adding noise to the real data. The model predicts operating values of the device at various intervals of the time series data set. The system identifies anomalies in the time series data based on a difference between the predicted value and the true value. If the difference between the predicted value generated by the machine learning model and the true value exceeds a threshold, the system identifies a particular data point, such as a meter reading, as anomalies. The system rates the exception to perform a repair operation based on the rating.

Description

[0001] Incorporation by reference; disclaimer

[0002] The following applications are hereby incorporated by reference: Application No. 18 / 069,534, filed on December 21, 2022; Application No. 63 / 383,176, filed on November 10, 2022, which are hereby incorporated by reference. The applicant hereby disclaims any disclaimer of claim scope in the (one or more) parent applications or their prosecution histories, and notifies the USPTO that the claims in this application may be broader than any claims in the (one or more) parent applications. Technical field

[0003] The present disclosure relates to detecting and remediating anomalies detected in operational data. In particular, the present disclosure relates to training a machine learning model to predict target operating values for a monitored device to identify anomalies associated with the monitored device. Background art

[0004] A remote system monitoring platform monitor obtains system characteristics from sensors in real time and analyzes the sensor data to identify possible problems in the system. One remote system monitoring platform is the Advanced Metering Infrastructure (AMI). AMI uses utility meters to monitor the utility usage of entities. A transmitter transmits the utility usage data to the utility provider. For example, a home equipped with an AMI power meter transmits power data to the power utility provider in real time or at regular intervals. The utility provider collects the usage data for a specific time period to bill the customer for the customer's utility usage. In addition, the utility provider can analyze the usage data to identify usage requirements for customers and regions. Anomalies may occur in the AMI system due to theft, cyberattacks, meter failures, device or equipment malfunctions, data corruption, or other problems. Inaccurate usage data may lead to inaccurate forecasts and customer billing.

[0005] The methods described in this section are methods that can be taken, but not necessarily methods that have been previously envisioned or taken. Therefore, unless otherwise stated, no method described in this section should be considered prior art solely because it is included in this section. Description of the drawings

[0006] In the figures of the drawings, embodiments are illustrated by way of example and not by way of limitation. It should be noted that references to "embodiment" or "an embodiment" in the present disclosure do not necessarily refer to the same embodiment, and they mean at least one. In the drawings:

[0007] Figure 1A and Figure 1B illustrates a system in accordance with one or more embodiments;

[0008] Figures 2A - 2C illustrates a set of example operations for operating data anomaly detection and repair in accordance with one or more embodiments;

[0009] Figure 3A and Figure 3B illustrates example embodiments; and

[0010] Figure 4 shows a block diagram illustrating a computer system in accordance with one or more embodiments. DETAILED DESCRIPTION

[0011] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding. One or more embodiments may be practiced without these specific details. Features described in one embodiment may be combined with features described in different embodiments. In some instances, well-known structures and devices are described in block diagram form to avoid unnecessarily obscuring the present invention.

[0012] 1. General Overview

[0013] 2. System Architecture

[0014] 3. Operating Data Anomaly Detection and Repair

[0015] 4. Example Embodiments

[0016] 5. Computer Networks and Cloud Networks

[0017] 6. Other Matters; Extensions

[0018] 7. Hardware Overview

[0019] 1. General Overview

[0020] Utility providers collect utility usage data from meters to plan load distribution, future modifications to the utility network, and to bill customers for utility usage. However, many different events can cause interruptions in accurate utility monitoring, resulting in inaccurate planning and billing.

[0021] One or more embodiments include training a machine learning model on a combination of real data of a device and false data generated by adding noise to the real data to predict an operating value of the device at various intervals of a time series dataset. The system identifies anomalies in the time series data based on the difference between the predicted value and the real value. If the difference between the predicted value generated by the machine learning model and the real value exceeds a threshold, then the system identifies a particular data point (such as a meter reading) as anomalous. The system ranks the anomalies to perform a remediation operation.

[0022] According to an example embodiment, the system trains a deep learning long short-term memory (LSTM) encoder machine learning model to predict an electricity usage value for an interval in a time series dataset. The LSTM encoder ML model receives a set of time series data as input data and predicts the electricity usage level of the device. The system can calculate an anomaly value for each interval in the time series data. If the anomaly value exceeds a threshold, then the system identifies that interval as anomalous. The system can rank the anomalous intervals based on various criteria to determine an appropriate action for remediating the anomaly. According to one example, the system compares the pattern of the anomalous intervals with patterns associated with known problems (such as a new device in a home or the installation of a piece of equipment in a commercial or industrial environment). The pattern can be associated with theft, associated with a meter malfunction, or associated with a device or equipment failure. The system can rank the anomaly based on the pattern that the system associates with the anomaly. According to another example, the system ranks the anomaly based on the severity of the anomaly. For example, if the real electricity usage value is 100% of the predicted electricity usage value, then the system ranks that anomaly higher than another anomaly where the real electricity usage value is 10% higher than the predicted electricity usage value.

[0023] One or more embodiments described in this specification and / or recited in the claims may not be included in this general overview section.

[0024] 2. System Architecture

[0025] FIG. 1 illustrates a system 100 according to one or more embodiments. As shown in FIG. 1, the system 100 includes a device operation monitoring platform 110, a data repository 120, a monitored device 130, and a network 140. In one or more embodiments, the system 100 may include more or fewer components than those shown in FIG. 1. The components shown in FIG. 1 may be local to each other or remote from each other. The components shown in FIG. 1 may be implemented in software and / or hardware. Each component may be distributed across multiple applications and / or machines. Multiple components may be combined into one application and / or machine. Operations described with respect to one component may alternatively be performed by another component.

[0026] The device operation monitoring platform 110 collects operation data from the monitored device 130 and stores the operation data as historical device operation data 121. The monitored device 130 may include meters, such as utility meters. For example, the monitored device 130 may be an electricity meter that measures the amount of electricity used at a specific location (such as a home, business, farm, etc.). The monitored device 130 may transmit electricity usage data to the device operation monitoring platform 110 at regular intervals.

[0027] The device operation monitoring platform 110 includes a machine learning model engine 111. The machine learning model engine 111 is trained on a training data set 122 to predict target operation values for the monitored device 130. The machine learning model engine 111 generates a training data set of true operation data 123 from the historical device operation data. Each data point in the data set includes (a) a device operation value (such as the usage amount within a defined time interval), (b) an attribute associated with the value (including the time interval associated with the value and the weather condition when the value was recorded), and (c) a label indicating that the value corresponds to the true operation data. The machine learning model engine 111 includes a fake training data generator 112. The fake training data generator 112 generates a set of fake operation data 124 based on the true operation data 123 training data set. For example, the fake training data generator 112 may randomly select a number of data points between 20% and 40% of the data points in the true operation data training data set 123. The fake training data generator 112 adds noise to the selected data points to generate the fake operation data training data set 124. The fake training data generator 112 may randomly add noise, such as by randomly determining whether to increase or decrease the usage value. In addition, the fake training data generator 112 may add noise by randomly modifying the usage value of the data points between 5% and 500%. According to an example embodiment, the fake training data generator 112 selects clusters of sequentially occurring time series data points to which noise is to be added.

[0028] Figure 1B Illustrated is a fake training data generator 112 applied to an embodiment where the data set is a time series data set. The fake training data generator 112 receives, for example, authentic time series operation data 151 corresponding to the true operation data 123 as input data. The noise generator 152 adds noise to the authentic time series operation data 151 to generate a data set corresponding to Figure 1AThe false operation data 124 corresponds to the false time series operation data 124. The noise generator 152 includes a noise application selection engine 153 and a noise magnitude determination engine 154. The noise application selection engine 153 determines which data points to select from the data points of the true time series operation data 151 to add noise. The noise application selection engine 153 selects a specific number of data points according to a specific pattern. The specific number of data points can include, for example, a specific percentage of the true time series operation data points. For example, if the true time series operation data 151 includes 1000 data points, then the noise application selection engine 153 can select 10% of the data points to add noise. In addition, the noise application selection engine 153 can add noise to the true time series operation data 151 by removing data points from the operation data 151. For example, if the time series data includes data points at one-hour time increments, then the noise application selection engine 153 can remove the data points for a specific hour. According to one embodiment, the noise application selection engine 153 applies a randomization function to randomly select data points in the true time series operation data 151 to add noise until a termination condition is met. For example, the noise application selection engine 153 can select data points to add noise according to the randomization function until 10% of the data points have been selected.

[0029] For the selected data points, the noise magnitude determination engine 154 determines (a) the magnitude of the noise to be added to the data points, and (b) the sign of the noise. The magnitude of the noise can be a percentage or an absolute value. For example, the noise magnitude determination engine 154 can apply a randomization function to randomly apply a noise amount in the range of 20% to 500% of the value of the data point. Alternatively, in an example where the data point represents kilowatt-hours, the noise magnitude determination engine 154 can randomly apply a noise amount in the range of 0.5 kWh to 200 kWh. The noise magnitude determination engine 154 also determines whether to add noise by applying a positive sign to the noise or a negative sign to the noise. According to one embodiment, the noise magnitude determination engine 154 applies a randomization function to determine whether to apply a positive sign or a negative sign to the noise. For example, if the data point includes a value of 100, and if the noise magnitude determination engine 154 determines that a noise with a magnitude of 75 will be added to the data point, then the noise magnitude determination engine 154 can also randomly apply a positive sign to the noise, resulting in a data point value of 175, or apply a negative sign to the noise, resulting in a data point value of 25.

[0030] The training data set engine 155 creates a training data set 158 for training a machine learning model (corresponding to Figure 1ACombined training dataset 122). The training dataset engine 155 combines the true time series operation data 151 with the false time series operation data 124 to create combined time series data 156. The combined time series data 156 includes the true time series operation data 151, where the selected data points are replaced with false data points generated by the false training data generator 112. For example, if a segment of data includes hourly data points for a particular day, the false training data generator 112 can create false data points for the 10:00 AM data point and the 2:00 PM data point. The combined time series data 156 includes the true data points from 12:00 AM to 9:00 AM of the day, the false data point at 10:00 AM, the true data points from 11:00 AM to 1:00 PM, the false data point at 2:00 PM, and the true data points from 3:00 PM to 11:00 PM.

[0031] The training dataset engine 155 also associates additional time series attribute data with the data points of the combined time series data 156 to generate a training dataset 158 of combined time series operation data. Examples of the additional time series attribute data 157 include weather conditions associated with a set of operation data 151 and calendar information (such as a date or a specific event) associated with a set of operation data 151. The training dataset engine 155 provides the training dataset 158 of combined time series operation data to the machine learning model engine 111 to train the machine learning model.

[0032] The machine learning model engine 111 uses the combined training dataset 122 that includes both the true operation data 123 and the false operation data 124 (corresponding to Figure 1BThe training data set in ( ) 158) trains the machine learning model 113. The machine learning model engine 111 trains the machine learning model to identify (a) relationships between attributes within the same data point, and (b) relationships between attributes of different data points in the same set of time series data. According to one embodiment, the machine learning model is a deep learning long short-term memory (LSTM) autoencoder machine learning algorithm. The LSTM autoencoder machine learning algorithm is configured with a set of LSTM "cells". Each "cell" includes a "cell state" and gates with parameters that are adjusted during training to teach the machine learning model the relationships between data points in the time series data. Each cell receives data via an input, outputs data via an output, and includes a "forget" gate. The input receives the data value associated with the current cell. For example, when training the LSTM autoencoder algorithm using time series data, one data point is associated with one cell, and subsequent time series data points are associated with subsequent cells. The input gate receives the data value associated with the current cell. The "forget" gate specifies the parameter that learns which information from the previous cell should be forgotten or ignored. The autoencoder structure of the machine learning algorithm maps the input data from a high-dimensional state to a low-dimensional state and then back to the original high-dimensional state. According to one or more embodiments, the LSTM autoencoder machine learning model includes tens of thousands of parameters that are adjusted during training. For example, the LSTM autoencoder machine learning model can include between 60,000 and 70,000 parameters that are adjusted during the training of the machine learning model.

[0033] Although the LSTM algorithm is described above by way of example, any machine learning model capable of processing time series data as input data and identifying the characteristics of specific intervals within the time series data can be utilized.

[0034] In some examples, one or more elements of the machine learning model engine 111 can use machine learning algorithms to learn the target operation data values for time series data points. The machine learning algorithm is an algorithm that can be iterated to learn the target model f that optimally maps a set of input variables to output variables using a set of training data. The machine learning algorithm can include a supervised component and / or an unsupervised component. Various types of algorithms can be used, such as linear regression, logistic regression, linear discriminant analysis, classification and regression trees, naive Bayes, k-nearest neighbor, learning vector quantization, support vector machines, bagging and random forests, boosting, backpropagation, and / or clustering.

[0035] In an embodiment, a set of training data includes a data set and associated labels. The data set is associated with input variables for a target model f (e.g., device operation values, time data, weather data, site data (e.g., single-family homes, apartments, businesses, farms, factories, etc.)). Each data point is associated with a label that indicates whether the data point is real operation data or fake operation data. Training the model involves auto-encoding an input vector representing the input data, reducing the dimension within the hidden layer of the model, and expanding the dimension of the hidden layer of the model such that the number of dimensions of the output layer is the same as that of the input layer. Training the model involves adjusting parameters to cause the values at the output layer to be the same as the values at the input layer. The training data can be updated based on, for example, feedback on the accuracy of the current target model f. The updated training data is fed back into a machine learning algorithm, which in turn updates the target model f.

[0036] The machine learning algorithm generates a target model f such that the target model f best fits the data set of the training data to the labels of the training data. Additionally or alternatively, the machine learning algorithm generates a target model f such that when the target model f is applied to the data set of the training data, the largest number of results determined by the target model f match the labels of the training data.

[0037] In an embodiment, the machine learning algorithm can be iterated to predict device operation values for a time interval in time series data. In an embodiment, a set of training data includes real operation data 123 and fake operation data 124. The training data set 122 is associated with labels that indicate whether a particular data point in the training data set corresponds to real operation data or fake operation data.

[0038] The device operation monitoring platform 110 receives operation data 125 from the monitored device 130. For example, a utility provider can receive real-time, minute-by-minute, hourly, or daily updates on electricity usage measured by an electricity meter. After receiving the operation data 125, the monitored device attribute data collection engine 114 collects additional attribute data 126 associated with the monitored device. For example, the monitored device attribute data collection engine 114 can identify the weather station closest to the device 130 that generated the operation data. The device operation monitoring platform 110 can store the weather data together with the received operation data values for a particular time interval. Other examples of attribute data that can be stored as data point attributes together with the operation data values include supplementary utility data, such as whether the location includes a backup generator. The attribute data can include information about the type of structure associated with the meter, such as a single-family home, apartment, hotel, industrial site, farm, factory, warehouse, or storefront. The attribute data can include the size of the structure associated with the meter, such as the number of bedrooms in a home or the number of square feet of the structure.

[0039] The machine learning model engine 111 embeds the monitored device operation data 125 and the attribute data 126 as vectors of a set of time series data. The machine learning model engine 111 feeds the time series data into the machine learning model 113 to generate predicted target operation values 115 for each sub-interval within the time series data. For example, a set of time series data can include 30 days of power data, which is divided into 30 sub-intervals, each corresponding to the power usage and additional attributes for each day in the 30 days.

[0040] The anomaly detection engine 116 analyzes the predicted target operation values 115 to detect anomalies among the sub-intervals in the time series data. The anomaly detection engine 116 identifies data points with outlier values that correspond to sub-intervals of time within the set of time series data. In particular, the machine learning model 113 generates predicted target values for the data points in the time series data based on (a) the correlations between the attributes within the data points, and (b) the correlations between the attributes of different data points in the time series data. The anomaly detection engine 116 compares the predicted target operation values 115 with the actual values of the monitored device operation data 125. The anomaly detection engine 116 calculates an anomaly score for each data point in the time series data set based on the difference between the predicted target value of the data point and the actual value associated with the data point. If the difference between the predicted target value and the actual value exceeds a threshold, then the system identifies the specific data point as anomalous.

[0041] The anomaly scoring engine 117 analyzes the anomalous data points in the time series data to assign a rating or weight to the anomalous data points. For example, the anomaly scoring engine 117 can assign a relatively greater weight to data points with a higher anomaly score than to data points with a lower anomaly score. Alternatively, the anomaly scoring engine 117 can assign a greater rating or weight to a cluster of data points that includes a specific pattern. The anomaly scoring engine 117 can identify patterns associated with meter failures, device failures, utility theft, utility transmission failures, and data transmission failures. The anomaly scoring engine 117 can assign different rating values to different identified patterns based on the severity of the corresponding failures.

[0042] The repair engine 118 selects repair actions associated with the anomalies detected in a set of time series data for execution. Examples of repair operations include generating notifications to customers and / or utility providers, remotely resetting meters, and adjusting the value calculation of utility bills for customers. The repair engine 118 can select a repair action based on the rating or weight of the detected anomaly. For example, if the system detects an anomaly associated with a resolved data transmission failure in a set of time series data, then the system can avoid performing further repair actions. If the system detects an anomaly associated with a utility transmission failure in a set of time series data, then the system can trigger a notification to the utility service provider that a repair may be required.

[0043] Additional embodiments and / or examples related to computer networks are described below in Part 5 entitled "Computer Networks and Cloud Networks".

[0044] In one or more embodiments, the data repository 120 is any type of storage unit and / or device for storing data (e.g., a file system, a database, a collection of tables, or any other storage mechanism). Additionally, the data repository 120 can include multiple different storage units and / or devices. The multiple different storage units and / or devices may or may not have the same type or be located at the same physical site. Additionally, the data repository 120 can be implemented or executed on the same computing system as the device operation monitoring platform 110. Alternatively or additionally, the data repository 120 can be implemented or executed on a computing system separate from the device operation monitoring platform 110. The data repository 104 can be communicatively coupled to the device operation monitoring platform 110 via a direct connection or via a network.

[0045] Information describing the training data set, the monitored device operation data, and the monitored device attribute data can be implemented across any component within the system 100. However, for purposes of clarity and explanation, the information is illustrated as being within the data repository 120.

[0046] In one or more embodiments, the device operation monitoring platform 110 refers to the hardware and / or software configured to perform the operations described herein for collecting operation data, analyzing the operation data by applying a trained machine learning model to the operation data, and using the predictions generated by the trained machine learning model to identify and repair anomalies in the system. Examples of operations for identifying and repairing anomalies based on the monitored device operation data are described below with reference to Figures 2A - 2C Describe examples of operations for identifying and repairing anomalies based on the monitored device operation data.

[0047] In an embodiment, the device operation monitoring platform 110 is implemented on one or more digital devices. The term "digital device" generally refers to any hardware device that includes a processor. A digital device can refer to a physical device that executes an application or a virtual machine. Examples of digital devices include computers, tablet computers, laptop computers, desktop computers, netbooks, servers, web servers, network policy servers, proxy servers, general-purpose machines, function-specific hardware devices, hardware routers, hardware switches, hardware firewalls, hardware network address converters (NATs), hardware load balancers, mainframes, televisions, content receivers, set-top boxes, printers, mobile handsets, smartphones, personal digital assistants ("PDAs"), wireless receivers and / or transmitters, base stations, communication management devices, routers, switches, controllers, access points, and / or client devices.

[0048] In one or more embodiments, the interface 119 refers to the hardware and / or software configured to facilitate communication between a user and the device operation monitoring platform 110. The interface 119 renders user interface elements and receives input via the user interface elements. Examples of interfaces include graphical user interfaces (GUIs), command line interfaces (CLIs), haptic interfaces, and voice command interfaces. Examples of user interface elements include checkboxes, radio buttons, dropdown lists, list boxes, buttons, toggles, text fields, date and time pickers, command lines, sliders, pages, and forms.

[0049] In an embodiment, different components of the interface 119 are specified in different languages. The behavior of user interface elements is specified in a dynamic programming language such as JavaScript. The content of user interface elements is specified in a markup language such as Hypertext Markup Language (HTML) or XML User Interface Language (XUL). The layout of user interface elements is specified in a style sheet language such as Cascading Style Sheets (CSS). Alternatively, the interface 119 is specified in one or more other languages such as Java, C, or C++.

[0050] 3. Identifying and Fixing Anomalies Based on Operational Data

[0051] Figures 2A - 2C Illustrated are a set of example operations for identifying and fixing anomalies based on operational data according to one or more embodiments. Figures 2A - 2C One or more of the operations shown may be modified, rearranged, or omitted. Thus, Figures 2A - 2C the particular order of operations shown should not be construed as limiting the scope of one or more embodiments.

[0052] The system obtains a set of historical data (operation 202). The historical data includes attributes and operation data values associated with the operation of one or more monitored devices. Examples of operation data values include power or other utility usage levels and calendar data, such as timestamps associated with usage levels. Examples of attributes associated with operation data values include weather data, location data, data describing the type of facility or structure utilizing the utility (e.g., commercial, retail, industrial), data describing the size of the structure (such as the number of bedrooms, rooms, or square feet), data describing how the structure is used (e.g., private home, hotel, data center, farmland, factory, storefront, warehouse), and data specifying whether the location is associated with relevant features (such as solar panels, wind turbines, or swimming pools).

[0053] The system generates a training data set from the historical operation data (operation 204). The training data set includes data points specifying operation data values and attributes associated with the monitored devices that generated the operation values. For example, a data point can include kWh power consumption over the course of an hour and the weather at the location associated with the meter that generated the kWh power consumption value.

[0054] The system generates a training data set of fake operation data using the training data set of real historical operation data (operation 204). The system selects data points from a training subset of the real historical operation data. The system introduces noise into the selected data points to generate a data set of fake operation data. For example, the system can copy the training data set of real operation data and introduce noise into each copy. Alternatively, the system can select a predetermined number of data points (such as 50%) from the training data set to copy and introduce noise to generate a training data set of fake operation values. According to yet another alternative, the system iteratively performs the following process: (a) select real historical operation data, (b) generate fake operation data by adding noise, (c) apply a machine learning algorithm to the combined data, (d) determine the accuracy of the machine learning model obtained by applying the algorithm to the combined data set, and (e) if the predicted accuracy is less than a threshold, then repeat (a)-(d).

[0055] According to an example embodiment, the system replaces a random number of data points in the time series data with spurious data points including noise. For example, in a time series dataset including 30 time series data points, the system may replace 10 data points randomly located among the 30 time series data points with spurious data points including noise. Additionally or alternatively, the spurious data points may be introduced into the dataset in clusters. For example, in a dataset including 30 time series data points, the system may introduce the spurious data points in three groups: 3 time series data points, 3 time series data points, and 4 time series data points. These groups may include consecutive time series data points. The system may calculate the noise to be added to the cluster according to a randomization formula. For example, the noise value for each data point in the cluster may be random within a predetermined range. Alternatively, the noise value for each data point in the cluster may be random in magnitude but have the same positive or negative sign. In other words, while the system may add or subtract a value from a true operating value to obtain a spurious operating value, each data point in one cluster will have a value added to create noise. Each data point in another cluster will have a value subtracted to add noise.

[0056] According to one or more embodiments, the system may add noise to the cluster in a specific pattern. For example, the system may randomly determine that a specific cluster of 3 time series data points will receive a positive noise operating value of 10 kWh, where 10 kWh is a value randomly selected from a range of values between 1 kWh and 50 kWh. The system may select one of the 3 time series data points to have an operating value of 10 kWh. The system may apply a gradation formula to set the operating values of the other 2 data points. For example, the system may apply a formula that sets the noise value adjacent to the peak random noise value to be 10% smaller in magnitude relative to the peak random operating value. Alternatively, the system may add noise to the cluster by applying a bell curve formula.

[0057] According to one or more embodiments, adding noise to time series data points includes: (a) selecting a certain number of data points in the time series data set to which noise is to be added, (b) selecting whether to add noise by increasing the magnitude of the operating value of the data point or decreasing the magnitude of the operating value of the data point, (c) selecting the magnitude by which the operating value is to be increased / decreased. The system may select a certain number of data points in the time series data set to which noise is to be randomly added. For example, if a particular time series data set includes 360 data points, then the system may randomly select data points from among the 360 data points to which noise is to be added. Alternatively, the system may randomly select data points within a threshold number of data points. For example, the system may apply a formula that calculates data points having values between 20% and 50% of the data set to add noise. If a particular time series data set includes 360 data points, then the system may randomly select a number of data points within the range of 72 to 180 from among the 360 data points to which noise is to be added. Selecting a certain number of data points in the time series data set to which noise is to be added may include adding noise to randomly distributed sites (i.e., operating data value generators, such as meters) from among the sites providing the data constituting the data set. For example, the system may apply a rule to add noise to the time series data of 20% of the sites randomly selected from among the sites providing the data constituting the data set. According to one or more embodiments, the system adds noise to the data points in the data set according to a particular distribution (such as: Poisson distribution, F distribution, Chi-squared distribution, Student's t distribution, normal distribution, and uniform distribution).

[0058] Selecting whether to increase or decrease the magnitude of the operating value of the data point may be randomized such that any particular data point has a 50% chance of increasing the operating value magnitude and a 50% chance of decreasing the operating value magnitude.

[0059] Selecting the magnitude by which to increase or decrease the operating value to add noise may be random within a particular value range. For example, the system may apply a formula that specifies that the operating value should be randomly selected within an operating value range that varies between 5% and 500% of the true operating value of historical data points. Additionally or alternatively, the system may apply a rule to change the magnitude of the operating data value by a particular unit range. For example, if the data set includes power data measured in kilowatt-hours (kWh), then the system may apply a rule to add / subtract a value between 1 kWh and 10 kWh to the data points to which noise is to be added.

[0060] According to one or more embodiments, the system uses operation data values from multiple sites over a specific time period to train a machine learning model. For example, the training dataset can include power usage data from 300 individual power meters respectively associated with 300 individual sites (such as residences). The training dataset can include time series data spanning weeks, months, or years. For example, a training dataset for training a machine learning model to identify outliers in a time series data segment of a month divided into daily intervals can span multiple different sites over two years or longer. The system can add noise to data points within a specific date range in the training dataset, or can add noise to data points across the entire date range of the entire training dataset.

[0061] The system applies a machine learning algorithm to the combined training set to train a machine learning model to predict a target operation value (operation 208). According to one embodiment, the machine learning algorithm accepts time series data obtained over a specified time period as input data. The time period of the time series segment provided to the machine learning algorithm is divided into incremental intervals. Each interval has its own operation data value and its own additional attributes. For example, the algorithm can accept a time series segment including 30 days of data points as input data. Each data point specifies the amount of power consumed in kWh on a specific day, the weather condition associated with that specific day, and any additional attributes included in the time series data. Based on (a) the relationships between the attributes within a specific data point and (b) the relationships between the attributes between different data points in the same time series segment, the system adjusts the parameters of the machine learning algorithm to train the machine learning model.

[0062] According to one embodiment, the machine learning algorithm is a deep learning long short-term memory (LSTM) autoencoder machine learning algorithm. The LSTM autoencoder machine learning algorithm is configured with a set of LSTM "cells". Each "cell" includes a "cell state" and gates with parameters that are adjusted during training to teach the machine learning model the relationships between data points in time series data. Each cell receives data via an input, outputs data via an output, and includes a "forget" gate. The input receives the data value associated with the current cell. For example, when training the LSTM autoencoder algorithm with time series data, one data point is associated with one cell, and subsequent time series data points are associated with subsequent cells. The input gate receives the data value associated with the current cell. The "forget" gate specifies the parameter that learns which information from the previous cell should be forgotten or ignored. The autoencoder structure of the machine learning algorithm maps the input data from a high-dimensional state to a low-dimensional state and then back to the original high-dimensional state. According to one or more embodiments, the LSTM autoencoder machine learning model includes tens of thousands of parameters that are adjusted during training. For example, the LSTM autoencoder machine learning model can include between 60,000 and 70,000 parameters that are adjusted during the training of the machine learning model.

[0063] Although the LSTM algorithm is described above by way of example, any machine learning model capable of processing time series data as input data and identifying the characteristics of specific intervals within the time series data can be utilized.

[0064] The system receives time series data including operation values from one or more monitored devices (operation 210). For example, the system can obtain power consumption data generated by an electricity meter associated with a residence. According to another example, the system can obtain water consumption data generated by a water meter associated with an industrial facility. The system can obtain the data in real time when the data is generated. Alternatively, the system can batch request or upload the data at intervals such as daily intervals, weekly intervals, or monthly intervals.

[0065] The system applies a trained machine learning model to time series data to generate predicted values for intervals in the time series data (operation 212). In particular, the system provides a set of time series data spanning a specific time period as input data to a machine learning model engine that stores and runs the machine learning model. The time series data includes time intervals within the time period. For example, the time period can be a week, a month, or multiple months. The time intervals can be, for example, a few minutes, a few hours, or a few days. The system identifies data points with outliers that correspond to the time intervals within the set of time series data. In particular, the machine learning model generates predicted target values for the data points in the time series data based on the learned (a) correlations between the attributes within the data points, and (b) correlations between the attributes of different data points in the time series data.

[0066] The system identifies outlier data points within the time series data (operation 214). For each data point associated with an interval of the time series data, the system compares the predicted target value generated by the machine learning model based on analyzing multiple data points of the time series data with the actual value of the interval in the time series data. The system calculates an outlier score for each data point in the time series dataset based on the difference between the predicted target value of the data point and the actual value associated with the data point. In other words, the greater the difference between the actual operational data value of the data point and the predicted target operational value, the greater the outlier score. If the difference between the predicted target value and the actual value exceeds a threshold, then the system identifies the specific data point as an outlier.

[0067] Reference Figure 2B The system analyzes the outlier data points in the time series data to assign an outlier score or weight to the outlier data points (operation 216). For example, the system can assign a relatively greater weight to data points with a higher outlier score than to data points with a lower outlier score. Alternatively, the system can assign a greater weight to a cluster of data points that contain a specific pattern. The system can store utility usage patterns associated with specific events such as meter failures, device failures, utility theft, and transmission failures (such as in the case of a water utility, a water supply pipe break, or in the case of an electric utility, a power supply line short circuit). As an example, the system can identify a pattern of a drop in electricity usage to a stable low usage rate that is not related to weather conditions as an outlier associated with a data transmission failure from solar panels installed on a structure. The system can assign different rating values to different identified patterns of outlier data points based on the severity of the corresponding failure. An anomaly detected corresponding to a pattern associated with a failed data transmission may receive a lower rating than an anomaly corresponding to a pattern associated with a damaged power line.

[0068] The system determines whether the anomaly score exceeds a threshold (operation 218). The threshold can include one or both of the following: (a) a threshold difference between the predicted value and the actual value of a data point, and (b) a threshold number of anomalous data points within a set of data points. For example, the threshold can specify that 50% or more of 20 data points in a set are anomalous by more than 10% of the predicted value. A data set where 40% of the data points are anomalous by more than 10% of the predicted value will not meet the threshold. The threshold can include multiple levels. For example, the threshold can specify (a) that 50% or more of 20 data points in a set are anomalous by more than 10% of the predicted value, or (b) that 10% or more of 20 data points in a set are anomalous by more than 30% of the predicted value. In other words, the threshold can be set to define a sliding scale, thus requiring more anomalous data points of lower severity or fewer anomalous data points of higher severity.

[0069] If the anomalous data point or set of data points exceeds the threshold, then the system selects a remediation action associated with the anomaly detected in a set of time series data to perform (operation 220). The system can select the remediation action based on the rating or weight of the detected anomaly. For example, if the system detects an anomaly associated with a resolved data transfer failure in a set of time series data, then the system can avoid performing further remediation actions. If the system detects an anomaly associated with a utility transmission failure in the set of time series data, then the system can trigger a notification to the utility service provider that repairs may be required. According to one example embodiment, the system analyzes the utility usage for a particular billing period to determine whether the amount billed to a customer is accurate. If the system detects an anomaly within the time series data for the billing period that has a pattern associated with theft of the utility (such as unauthorized use of electricity from a particular location), then the system can generate a notification to the customer advising the customer to review the charges. Additionally or alternatively, the system can avoid including charges associated with the anomalous usage in the customer's bill.

[0070] If the system determines that the anomalous data point or set of anomalous data points does not exceed the threshold, then the system selects the next data point corresponding to the next time interval in a set of time series data for analysis (operation 222).

[0071] Figure 2C Illustrated is a set of operations that can be performed as an addition to or an alternative to the set of operations shown as Figure 2B in.

[0072] Similar to Figure 2B as discussed above, the system analyzes the anomalous data points to generate a score and / or weight associated with the anomalous data points (operation 216).

[0073] Based on one or more scores in a set of time series data, the system classifies anomalies (operation 224). For example, the system may classify a set of anomaly scores for data points in a set of time series data as: meter failure, device failure, solar panel failure, utility theft, new device installation, utility provider failure, data transmission error, and increase / decrease in utility usage associated with an increase / decrease in occupants in a residence or a change in operations at a commercial facility.

[0074] The system determines whether the anomaly classification corresponds to a meter failure (operation 226). If the classification does not correspond to a meter failure, then the system selects the next data point for analysis (operation 230). If the classification corresponds to a meter failure, then the system stores or transmits the predicted value(s) of the time series interval data points corresponding to the meter failure instead of the measured values of the time series interval data points corresponding to the meter failure. For example, the system may detect a meter failure on two days out of thirty. Instead of storing the measured values for these two days or in addition to it, the system stores the predicted values generated by a machine learning model. According to one example embodiment, the remediation action includes sending a notification about the meter failure to a service center. The operator can contact the customer to inspect the meter or to schedule a time to repair the meter. According to another example embodiment, a bill showing utility usage at different time intervals within a set period of time (such as daily electricity usage within a month) may display the actual measured usage value of the anomaly as a dashed line and display the predicted usage value for the same time interval as superimposed on the actual measured usage value.

[0075] 4. Example Embodiments

[0076] For clarity, detailed examples are described below. The components and / or operations described below should be understood as a specific example, which may not be applicable to some embodiments. Therefore, the components and / or operations described below should not be construed as limiting the scope of any claims.

[0077] Figure 3A System 300 for monitoring electricity usage using advanced metering infrastructure (AMI) technology is illustrated. System 300 includes residences 330a - 330n. Residences 330a - 330n are connected to an electric utility network. Electricity usage at residences 330a - 330n is monitored by meters 333a - 333n. Meters 333a - 333n transmit electricity usage data to a meter monitoring platform 310 via network 340. The network may include a global data network such as the Internet.

[0078] A machine learning model training data generator 311 generates a training data set consisting of true time series data and false time series data obtained from residences 330. As Figure 3BAs shown in [figure], the machine learning model training data generator 311 obtains the true time series meter data 351 from the residence 330 and provides the true time series meter data 351 to the fake training data generator 312.

[0079] The noise generator 352 adds noise to the true time series meter data 351 to generate the fake time series meter data 324. The noise generator 352 includes a noise application selection engine 353 and a noise amplitude determination engine 354. The noise application selection engine 353 determines which data points among the data points of the true time series meter data 351 to add noise to. The noise application selection engine 353 selects a specific number of data points according to a specific pattern. The specific number of data points includes a specific percentage of the true time series operation data points. In Figure 3B the example embodiment shown in [figure], the true time series meter data 351 includes one month of meter data divided into one-hour increments. Each increment includes (a) a value corresponding to the power consumption measured by the corresponding utility meter within the corresponding hour, and (b) a timestamp indicating the hour and date when the meter measures the power consumption. The noise application selection engine 353 selects 20% of the data points in a set of true time series meter data 351, or approximately 144 data points corresponding to 144 hours within a 30-day month of 720 hours (depending on the number of days in the given month). The noise application selection engine 353 further removes 5% of the data points in the set of true time series meter data 351, or approximately 36 data points corresponding to 36 hours within a 30-day month of 720 hours. According to one embodiment, the noise application selection engine 353 applies a randomization function to randomly select data points in the true time series operation data 351 to add noise until a termination condition is met. For example, the noise application selection engine 353 can randomly select data points corresponding to the hour increments among the 720-hour increments of the true time series meter data 351 until 144 data points have been selected.

[0080] For the selected data points, the noise amplitude determination engine 354 determines (a) the amplitude of the noise to be added to the data point, and (b) the sign of the noise. The amplitude of the noise can be a percentage or an absolute value. In Figure 3B the example shown in [figure], the noise amplitude determination engine generates a random value between 20% and 100% of the amplitude of the power usage value of the data point. In addition, the noise amplitude determination engine 354 also randomly applies a positive or negative sign to the random value.

[0081] The training dataset compilation engine 355 creates a training dataset 358 of combined time series meter data for training a machine learning model. The training dataset compilation engine 355 combines the true time series operational data 351 with the false time series operational data 324 to create combined time series data 356. The combined time series data 356 includes the true time series operational data 351, where selected data points are replaced with false data points generated by the false training data generator 312.

[0082] The training dataset compilation engine 355 further retrieves additional time series attribute data using the data points of the combined time series data 356 to generate a training dataset 358 of combined time series operational data. The additional time series attribute data 357 includes the weather conditions at the time corresponding to the timestamps of the true time series meter data 351 and the calendar information associated with the timestamps of the true time series meter data 351.

[0083] The meter monitoring platform 310 provides the training dataset 358 of combined time series operational data to the machine learning model engine 312 to train a machine learning model 362 to predict the target operational value of the meter. The machine learning model can be applied to the time series data generated by one of the meters 333a - 333n, or to another meter determined to have characteristics similar to those of the meters 333a - 333n. For example, the meter monitoring platform 310 can apply the machine learning model to any meter within a specified geographical area and associated with a single - family residence.

[0084] The machine learning model engine 311 trains the machine learning model 362 to identify (a) the relationships between attributes within the same data point, and (b) the relationships between attributes of different data points within the same set of time series data. In Figure 3A and Figure 3C the example shown, the machine learning model 362 is a deep learning long short - term memory (LSTM) auto - encoder machine learning model.

[0085] After training the machine learning model 362, the meter monitoring platform 310 obtains target time series meter data 361, as shown in Figure 3C The target time series meter data 361 is data generated by an electric meter (such as the electric meter 333a of the residence 330a). The meter 333a associated with the target time series meter data 361 can be among the group of meters that provided the data for training the machine learning model 362. Alternatively, the meter 333a can not be among the group of meters used for training the machine learning model 362.

[0086] The target time series metering data 361 can be net Advanced Metering Infrastructure (AMI) data. The net AMI data includes values that reflect not only the electricity supplied from a utility provider to a customer but also the electricity generated by the customer (such as by the solar panel 331). For example, in some solar systems, the solar panels do not power the residence in which they are installed. Instead, the utility company provides all the electricity to the residence, the solar panels feed electricity back into the grid, and the utility company deducts the cost of that electricity from the utility bill associated with the residence. The target time series metering data 361 can include a metering value that includes the electricity provided from the utility provider minus the electricity generated by the solar panels and sold back to the utility provider. Although solar panels are described in the example embodiments associated with Figure 3A additional power sources that can generate electricity at a residence that can be sold to the utility company include wind turbines and geothermal generators.

[0087] The target time series metering data 361 corresponds to a one-month duration. The target time series metering data 361 includes individual data points for each hourly interval within the one-month time period. For example, if the month has 30 days, then the target time series data 361 includes 720 individual data points, each data point including (a) a metering value (e.g., electricity consumed in kWh), (b) a timestamp, and (c) additional attribute data 367, such as weather data or attribute data about the monitored location (such as residence 330a includes solar panel 331, or residence 330b includes a swimming pool 332).

[0088] In Figure 3C the embodiment shown, the meter location identifier 363 determines the location of the meter that generates the target time series metering data 361. The location can be an address, coordinates, or a region. The additional data collection engine 364 includes a weather station locator 365 for locating a weather station 335a or 335n near the meter location. The weather station can be the weather station closest to the meter that generates the target time series metering data 361. The facility attribute data collection engine 366 collects additional data about the facility being monitored by the meter, including unique electricity consumption attributes. For example, the facility attribute data collection engine 366 can determine whether residence 330a includes a solar panel 331, a swimming pool, is associated with a high electricity consumption operation (such as a data center), or is a multi-residence building.

[0089] The instrument monitoring platform 310 provides target time series instrument data 361 and additional time series attribute data 367 to the machine learning model 362 to generate power usage prediction values for each data point of the target time series instrument data 361. This set of prediction values for the target time series instrument data 361 is the predicted time series instrument data 368.

[0090] The anomaly detection engine 313 analyzes the predicted time series instrument data 368 to detect anomalies among the intervals in the data. The anomaly detection engine 313 identifies data points with outlier values that correspond to hour times within a one-month time period. The anomaly scoring engine 314 calculates an anomaly score for each data point in the time series dataset based on the difference between the predicted target value of the data point and the actual value associated with the data point. If the difference between the predicted target value and the actual value exceeds a threshold, then the system identifies the specific data point as an anomaly. A larger difference between the predicted instrument value and the measured instrument value corresponds to a higher anomaly score. A smaller difference between the predicted instrument value and the measured instrument value corresponds to a lower anomaly score.

[0091] The anomaly repair engine 315 selects repair actions associated with the anomalies detected in the target time series instrument data 361 to execute. Examples of repair operations include generating notifications to customers and / or utility providers, remotely resetting the meter, and adjusting the value calculation for the customer's utility bill.

[0092] The system 300 includes a utility service platform 318. Based on detecting an anomaly score that exceeds a threshold, the anomaly repair engine 315 can transmit data associated with the anomaly to the utility service platform 318. For example, the anomaly repair engine 315 can detect a power usage pattern corresponding to power theft in the target time series instrument data 361. The instrument monitoring platform 310 sends location data associated with the meter 333a and potential theft to the utility service platform 318. The utility service platform 318 generates a ticket. Utility staff can inspect the meter associated with the ticket to determine if theft is occurring or if any other faults or issues can be observed at the meter 333a.

[0093] System 300 includes a utility billing platform 319. The utility billing platform 319 utilizes a machine learning model 362 to analyze target time series meter data 361 to determine whether the amount billed to a customer is accurate. Over a billing period (such as a particular month), meter 333n transmits net AMI utility usage data to a meter monitoring platform 310 maintained by a utility provider. The utility provider stores the utility usage data in a data repository. The meter monitoring platform compiles net AMI time series meter data for a particular billing period from the data repository before sending a bill to the customer. The meter monitoring platform 310 applies model 362 to the net AMI time series meter data for the month to identify specific days and hours with abnormal usage values. The utility billing platform 319 initiates a remediation action based on the severity of the abnormal usage value. For example, if the anomaly is associated with a lower rating or severity, then the utility billing platform 319 can generate a notification to the customer. If the anomaly rating corresponds to potential theft of the utility, then the utility billing platform 319 can generate a warning to an anti-theft unit. If the anomaly rating is associated with a meter malfunction, then the utility billing platform 319 can prompt a utility representative to omit the charges for one or more days from the bill pending review for the customer.

[0094] Although Figures 3A - 3CThe embodiments described are described in terms of a set of time series data corresponding to a one-month duration at an hourly interval, but the embodiments include different durations and different intervals of the time series data sets. For example, according to one example embodiment, the meter monitoring platform 310 analyzes utility usage data from meters to identify equipment failures and data transmission failures. The meter monitoring platform 310 can analyze the utility usage data received from the utility meters in real time or near real time. For example, a machine learning model 362 can be trained to receive a one-week time series data segment consisting of hourly intervals as input data, as individual data points within the one-week time series data segment. Each day, the meter monitoring platform 310 can provide the machine learning model with the utility usage data for the previous seven days. The machine learning model 362 generates predictions of target utility usage values for each hourly interval in the one-week time series data segment. The meter monitoring platform 310 generates an anomaly score for each hourly interval in the one-week time series data segment based on the difference between the predicted target utility usage value and the actual utility usage value. The meter monitoring platform 310 can identify a particular anomaly as corresponding to an equipment and / or transmission failure. For example, if a particular hourly interval corresponds to a period of extremely hot or cold weather and also corresponds to a decrease in electricity usage, then the meter monitoring platform 310 can determine that an equipment failure has occurred based on historical patterns associated with electricity usage and extreme weather conditions. As another example, if a series of one-hour long intervals maintain the same electricity usage level when the system expects a change in the electricity usage level, then the meter monitoring platform 310 can determine that a data transmission failure has occurred.

[0095] Based on detecting anomalous time series data in the target time series meter data 361, the anomaly repair engine 315 can perform a repair operation of replacing the anomalous time series values with the predicted time series values generated by the machine learning model 362. The meter monitoring platform 310 can use the utility usage data for a variety of different purposes, such as planning the future development of the utility network, predicting the load on the utility network, and billing customers for utility usage. When equipment failures, utility transmission failures, or meter data transmission failures interrupt a series of time series data points through anomalous data points, the system may be unable to accurately plan future development or bill customers. The anomaly repair engine 315 repairs the detected anomalous values by replacing the detected anomalous values with the values predicted by the machine learning model 362 in the data storage or data transmission. According to one example, the meter monitoring platform 310 may detect anomalous data points for three days in a one-month time series data segment. The meter monitoring platform 310 can replace the anomalous data point values in the time series data with the target values generated by the machine learning model. The meter monitoring platform 310 can use the data set including the replaced data point values instead of the anomalous data point values to identify trends. As another example, the utility billing platform 319 can replace the currency value corresponding to the anomalous utility usage value with the replacement currency value corresponding to the utility usage value predicted by the machine learning model 362. The utility usage value predicted by the machine learning model 362 is more likely to reflect the actual utility usage than the anomalous measured utility usage value. Therefore, the utility billing platform 319 can charge customers an amount that more closely corresponds to the actual utility usage than the utility usage reflected in the anomalous data point values.

[0096] 5. Computer Networks and Cloud Networks

[0097] In one or more embodiments, a computer network provides connectivity between a set of nodes. The nodes can be local to each other and / or remote from each other. The nodes are connected by a set of links. Examples of links include coaxial cables, unshielded twisted pairs, copper cables, optical fibers, and virtual links.

[0098] A subset of the nodes implements the computer network. Examples of such nodes include switches, routers, firewalls, and network address translators (NATs). Another subset of the nodes uses the computer network. Such nodes (also referred to as “hosts”) can execute client processes and / or server processes. The client processes make requests for computing services, such as the execution of a particular application and / or the storage of a particular amount of data. The server processes respond by executing the requested service and / or returning the corresponding data.

[0099] A computer network can be a physical network, including physical nodes connected by physical links. A physical node is any digital device. A physical node can be a function-specific hardware device, such as a hardware switch, a hardware router, a hardware firewall, and a hardware NAT. Additionally or alternatively, a physical node can be a general-purpose machine configured to execute various virtual machines and / or applications performing corresponding functions. A physical link is a physical medium connecting two or more physical nodes. Examples of links include coaxial cables, unshielded twisted pairs, copper cables, and optical fibers.

[0100] A computer network can be an overlay network. An overlay network is a logical network implemented on top of another network, such as a physical network. Each node in the overlay network corresponds to a corresponding node in the underlying network. Thus, each node in the overlay network is associated with both an overlay address (for addressing to the overlay node) and an underlying address (for addressing the underlying node implementing the overlay node). An overlay node can be a digital device and / or a software process, such as a virtual machine, an application instance, or a thread. The link connecting overlay nodes is implemented as a tunnel through the underlying network. The overlay nodes at either end of the tunnel view the underlying multi-hop path between them as a single logical link. Tunneling is performed through encapsulation and decapsulation.

[0101] In an embodiment, a client can be local to and / or remote from a computer network. The client can access the computer network through other computer networks, such as a private network or the Internet. The client can use a communication protocol, such as the Hypertext Transfer Protocol (HTTP), to send requests to the computer network. The requests are sent through an interface, such as a client interface (such as a web browser), a program interface, or an Application Programming Interface (API).

[0102] In an embodiment, a computer network provides a connection between a client and network resources. Network resources include hardware and / or software configured to execute server processes. Examples of network resources include processors, data storage devices, virtual machines, containers, and / or software applications. Network resources are shared among multiple clients. The clients independently request computing services from the computer network. Network resources are dynamically allocated to requests and / or clients on a demand basis. The network resources allocated to each request and / or client can be scaled up or down based on, for example, (a) the computing services requested by a specific client, (b) the aggregated computing services requested by a specific tenant, and / or (c) the aggregated computing services requested by the computer network. Such a computer network can be referred to as a "cloud network".

[0103] In an embodiment, a service provider provides a cloud network to one or more end users. The cloud network can implement various service models, including but not limited to Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). In SaaS, the service provider provides the end user with the ability to use an application that is executing on network resources of the service provider. In PaaS, the service provider provides the end user with the ability to deploy a customized application onto the network resources. The customized application can be created using programming languages, libraries, services, and tools supported by the service provider. In IaaS, the service provider provides the end user with the ability to provision processing, storage, network, and other basic computing resources provided by the network resources. Any arbitrary application, including an operating system, can be deployed on the network resources.

[0104] In an embodiment, a computer network can implement various deployment models, including but not limited to private cloud, public cloud, and hybrid cloud. In a private cloud, network resources are provisioned for exclusive use by a specific group of one or more entities (as used herein, the term "entity" refers to a company, organization, person, or other entity). The network resources can be local to the premises of the specific group of entities and / or remote from the premises of the specific group of entities. In a public cloud, cloud resources are provisioned for multiple entities (also referred to as "tenants" or "customers") that are independent of each other. The computer network and its network resources are accessed by clients corresponding to different tenants. Such a computer network can be referred to as a "multi-tenant computer network". Several tenants can use the same specific network resources at different times and / or at the same time. The network resources can be local to the premises of the tenants and / or remote from the premises of the tenants. In a hybrid cloud, the computer network includes a private cloud and a public cloud. The interface between the private cloud and the public cloud allows for portability of data and applications. Data stored at the private cloud and data stored at the public cloud can be exchanged through the interface. Applications implemented at the private cloud and applications implemented at the public cloud can be dependent on each other. Calls can be made from an application at the private cloud to an application at the public cloud (and vice versa) through the interface.

[0105] In an embodiment, the tenants of a multi-tenant computer network are independent of each other. For example, the business or operations of one tenant can be separated from the business or operations of another tenant. Different tenants may have different network requirements for the computer network. Examples of network requirements include processing speed, data storage volume, security requirements, performance requirements, throughput requirements, latency requirements, elasticity requirements, Quality of Service (QoS) requirements, tenant isolation, and / or consistency. The same computer network may need to implement different network requirements required by different tenants.

[0106] In one or more embodiments, in a multi-tenant computer network, tenant isolation is implemented to ensure that the applications and / or data of different tenants are not shared with each other. Various tenant isolation methods can be used.

[0107] In an embodiment, each tenant is associated with a tenant ID. Each network resource of a multi-tenant computer network is labeled with the tenant ID. A tenant is allowed to access a particular network resource only if the tenant and the particular network resource are associated with the same tenant ID.

[0108] In an embodiment, each tenant is associated with a tenant ID. Each application implemented by a computer network is labeled with the tenant ID. Additionally or alternatively, each data structure and / or data set stored by the computer network is labeled with the tenant ID. A tenant is allowed to access a particular application, data structure, and / or data set only if the tenant and the particular application, data structure, and / or data set are associated with the same tenant ID.

[0109] As an example, each database implemented by a multi-tenant computer network can be labeled with a tenant ID. Only a tenant associated with the corresponding tenant ID can access the data of a particular database. As another example, each entry in a database implemented by a multi-tenant computer network can be labeled with a tenant ID. Only a tenant associated with the corresponding tenant ID can access the data of a particular entry. However, a database can be shared by multiple tenants.

[0110] In an embodiment, a subscription list indicates which tenants have authorization to access which applications. For each application, a list of tenant IDs of the tenants authorized to access the application is stored. A tenant is allowed to access a particular application only if the tenant ID of the tenant is included in the subscription list corresponding to the particular application.

[0111] In an embodiment, network resources (such as digital devices, virtual machines, application instances, and threads) corresponding to different tenants are isolated into tenant-specific overlay networks maintained by a multi-tenant computer network. As an example, packets from any source device in a tenant overlay network can only be transmitted to other devices within the same tenant overlay network. Encapsulation tunnels are used to prohibit any transmission from a source device on a tenant overlay network to a device in another tenant overlay network. Specifically, a packet received from a source device is encapsulated within an outer packet. The outer packet is transmitted from a first encapsulation tunnel endpoint (communicating with the source device in the tenant overlay network) to a second encapsulation tunnel endpoint (communicating with the destination device in the tenant overlay network). The second encapsulation tunnel endpoint de-encapsulates the outer packet to obtain the original packet transmitted by the source device. The original packet is transmitted from the second encapsulation tunnel endpoint to the destination device within the same particular overlay network.

[0112] 6. Other Matters; Extensions

[0113] The embodiments relate to a system having one or more devices, the one or more devices including a hardware processor and configured to perform any of the operations described herein and / or recited in any of the following claims.

[0114] In an embodiment, a non-transitory computer-readable storage medium includes instructions that, when executed by one or more hardware processors, cause any of the operations described herein and / or recited in any of the claims to be performed.

[0115] According to one or more embodiments, any combination of the features and functions described herein may be used. In the foregoing specification, embodiments have been described with reference to numerous specific details that may vary depending on implementation. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention and what the applicant intends to be the scope of the invention is the literal and equivalent scope of the issued claims in the specific form of a set of claims issued from this application, including any subsequent amendments.

[0116] 7. Hardware Overview

[0117] According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing device can be hard-wired to perform the techniques, or can include digital electronic devices (such as one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or network processing units (NPUs)) that are persistently programmed to perform the techniques, or can include one or more general-purpose hardware processors programmed to perform the techniques according to program instructions in firmware, memory, other storage devices, or a combination. Such special-purpose computing devices can also combine custom hard-wired logic, ASICs, FPGAs, or NPUs with custom programming to implement the techniques. The special-purpose computing device can be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that combines hard-wired and / or program logic to implement the techniques.

[0118] For example, Figure 4 is a block diagram of a computer system 400 on which embodiments of the invention may be implemented. Computer system 400 includes a bus 402 or other communication mechanism for conveying information and a hardware processor 404 coupled to bus 402 for processing information. For example, hardware processor 404 can be a general-purpose microprocessor.

[0119] The computer system 400 also includes a main memory 406 coupled to the bus 402 for storing information and instructions to be executed by the processor 404, such as random access memory (RAM) or other dynamic storage devices. The main memory 406 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 404. When such instructions are stored in a non-transitory storage medium accessible to the processor 404, such instructions cause the computer system 400 to become a special-purpose machine customized to perform the operations specified in the instructions.

[0120] The computer system 400 also includes a read-only memory (ROM) 408 or other static storage device coupled to the bus 402 for storing static information and instructions for the processor 404. A storage device 410, such as a magnetic disk or optical disk, is provided and coupled to the bus 402 for storing information and instructions.

[0121] The computer system 400 may be coupled via the bus 402 to a display 412 for displaying information to a computer user, such as a cathode ray tube (CRT). An input device 414, including alphanumeric keys and other keys, is coupled to the bus 402 for transmitting information and command selections to the processor 404. Another type of user input device is a cursor control 416 for transmitting direction information and command selections to the processor 404 and for controlling the movement of a cursor on the display 412, such as a mouse, trackball, or cursor direction keys. Such input devices typically have two degrees of freedom along two axes (a first axis (e.g., x) and a second axis (e.g., y)), which allows the device to specify a position in a plane.

[0122] The computer system 400 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic, which in combination with the computer system causes the computer system 400 to become a special-purpose machine or programs the computer system 400 as a special-purpose machine. According to one embodiment, the computer system 400 performs the techniques herein in response to execution of one or more sequences of one or more instructions contained in the main memory 406 by the processor 404. Such instructions may be read into the main memory 406 from another storage medium, such as the storage device 410. Execution of the sequence of instructions contained in the main memory 406 causes the processor 404 to perform the processing steps described herein. In an alternative embodiment, hardwired circuitry may be used in place of or in combination with software instructions.

[0123] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. For example, non-volatile media includes optical or magnetic disks, such as storage device 410. Volatile media includes dynamic memory, such as main memory 406. By way of example, common forms of storage media include floppy disks, flexible disks, hard disks, solid state drives, magnetic tape, or any other magnetic data storage media, CD-ROM, any other optical data storage media, any physical media with hole patterns, RAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, content addressable memory (CAM), and ternary content addressable memory (TCAM).

[0124] Storage media is different from transmission media but can be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that comprise bus 402. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0125] Carrying one or more sequences of one or more instructions to processor 404 for execution can involve various forms of media. For example, the instructions can initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 400 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector can receive the data carried in the infrared signal, and appropriate circuitry can place the data on bus 402. Bus 402 carries the data to main memory 406, from which processor 404 retrieves and executes the instructions. The instructions received by main memory 406 can optionally be stored on storage device 410 before or after being executed by processor 404.

[0126] The computer system 400 also includes a communication interface 418 coupled to the bus 402. The communication interface 418 provides two-way data communication coupled to a network link 420 that is connected to a local network 422. For example, the communication interface 418 can be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection with a corresponding type of telephone line. As another example, the communication interface 418 can be a LAN card for providing a data communication connection with a compatible local area network (LAN). A wireless link can also be implemented. In any such implementation, the communication interface 418 transmits and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0127] The network link 420 typically provides data communication to other data devices through one or more networks. For example, the network link 420 can provide a connection to a host computer 424 or to a data device operated by an Internet Service Provider (ISP) 426 through the local network 422. The ISP 426 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the "Internet" 428. Both the local network 422 and the Internet 428 use electrical, electromagnetic, or optical signals that carry digital data streams. Signals through the various networks and signals on the network link 420 and through the communication interface 418 are example forms of transmission media that carry digital data to and from the computer system 400.

[0128] The computer system 400 can send messages and receive data, including program code, through the (one or more) networks, the network link 420, and the communication interface 418. In the Internet example, a server 430 can transmit the requested code for an application through the Internet 428, the ISP 426, the local network 422, and the communication interface 418.

[0129] The received code can be executed by the processor 404 when it is received, and / or stored in the storage device 410 or other non-volatile storage means for later execution.

[0130] In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary depending on the implementation. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indication of the scope of the invention and what the applicant intends to be the scope of the invention is the literal and equivalent scope of the issued claims in the specific form of a set of claims issued from this application, including any subsequent amendments.

Claims

1. A non-transitory computer-readable medium comprising instructions that, when executed by one or more hardware processors, cause performance of operations comprising: Training a machine learning model to predict target operating values for monitored devices, the training comprising: Obtaining a first subset of training data comprising historical operating data for one or more monitored devices, each first subset of training data comprising: Time series operating values for the one or more monitored devices; and For each subset in the first subset of training data, identifying that subset as a label of real operating data; Generating a second subset of training data at least by the following steps: Selecting a second subset of training data from among the first subset of training data; Applying noise to the second subset of training data; and For each subset in the second subset of training data, applying a label that identifies that subset as fake operating data; Training the machine learning model based on the first subset of training data and the second subset of training data; Receiving specific time series operating data associated with a first monitored device; Applying the machine learning model to the specific time series operating data to generate predicted target operating data; and Comparing a first value of a first data point of the received specific time series operating data with a second value corresponding to a predicted target operating data value associated with the first data point; and Based on determining that the difference between the first value and the second value exceeds a threshold, identifying the first value as an anomaly.

2. The non-transitory computer-readable medium of claim 1, wherein the operations further comprise: Comparing a third value of a second data point of the received specific time series operating data with a fourth value corresponding to a predicted target operating data value associated with the second data point; Based on determining that the difference between the third value and the fourth value exceeds the threshold, identifying the third value as an anomaly; In response to determining that the difference between the first value and the second value exceeds the first amount of the threshold, assigning a first weight to the first value; In response to determining that the difference between the third value and the fourth value exceeds the second amount of the threshold, assigning a second weight to the third value; Based on determining that the first weight meets a repair criterion, performing a repair operation associated with the first value; and Based on determining that the second weight does not meet the repair criterion, avoiding performing any repair operation associated with the third value.

3. The non-transitory computer-readable medium of claim 1, wherein each first subset of training data further comprises attributes associated with the one or more monitored devices, the attributes comprising at least weather conditions near the one or more monitored devices.

4. The non-transitory computer-readable medium of claim 3, wherein the attributes associated with the one or more monitored devices further comprise at least one of the following: temperature data, dew point data, residential type data, and demographic data.

5. The non-transitory computer-readable medium of claim 1, wherein applying noise to the second subset of training data comprises: Selecting a set of data points from among the first subset of training data; Randomly selecting an addition operation or a subtraction operation to perform; and Apply a random positive change or a random negative change to the value of each data point in the selected set of data points within a varying threshold level, based on a randomly selected addition or subtraction operation.

6. The non-transitory computer-readable medium of claim 1, wherein receiving target time series operation data comprises: Receiving location data and time series operation values associated with a first monitored device; Identifying a weather sensor within a threshold distance of the first monitored device based on the location data; Obtaining weather data generated by the weather sensor associated with the received time series operation values; and Generating a vector comprising the time series operation values and the weather data, wherein the machine learning model is applied to the vector.

7. The non-transitory computer-readable medium of claim 1, wherein the machine learning model is based on a deep learning long short-term memory (LSTM) type model.

8. The non-transitory computer-readable medium of claim 1, wherein the operation further comprises: Comparing a third value of a second data point of the received specific time series operation data with a fourth value corresponding to a predicted target operation data value associated with the second data point; and Identifying the third value as non-abnormal based on determining that the difference between the third value and the fourth value does not exceed the threshold.

9. The non-transitory computer-readable medium of claim 1, wherein the operation further comprises: Associating a monetary value with a specific user account corresponding to the first monitored device based on the operation values of the first monitored device over a defined time period; and Based on determining that the first value of the first data point is abnormal: Omitting a first monetary value associated with the first value of the first data point from the monetary value associated with the specific user account.

10. The non-transitory computer-readable medium of claim 1, wherein the time series operation data comprises power data measured by an electric utility meter at a specific location.

11. A method comprising the operations recited in any one of claims 1 to 10.

12. A system, comprising: One or more processors; and A memory storing instructions which, when executed by the one or more processors, cause the system to perform the operations recited in any one of claims 1 to 10.

13. A system comprising means for performing the operations recited in any one of claims 1 to 10.

Images