Operational data anomaly detection and correction

Abstract

Techniques are disclosed for detecting and correcting anomalous intervals in time series data for monitored devices. A system trains a machine learning model on a combination of actual data obtained from the monitoring device and fake data generated by adding noise to the actual data. The model predicts operational values ​​of the device at individual intervals in the time series data set. The system identifies anomalies in the time series data based on the difference between the predicted value and the actual value. If the difference between the predicted value generated by the machine learning model and the actual value exceeds a threshold, the system identifies the particular data point, such as a meter reading, as anomalous. The system ranks the anomalies and takes corrective action based on the ranking.

Description

[Technical Field]

[0001] Incorporation by Reference; Disclaimer The following applications are incorporated herein by reference: U.S. Patent Application No. 18 / 069,534, filed December 21, 2022, and U.S. Patent Application No. 63 / 383,176, filed November 10, 2022, which are incorporated herein by reference. Applicant hereby withdraws any disclaimer of claim scope in the parent application or its prosecution history, and reports to the USPTO that the claims in this application may be broader than any claims in the parent application.

[0002] Technical Field The present disclosure relates to detecting anomalies in operational data and correcting the detected anomalies. In particular, the present disclosure relates to training machine learning models to predict target operational values ​​of monitored devices and to identify anomalies associated with the monitored devices. [Background technology]

[0003] background A remote system monitoring platform monitors system characteristics in real time from sensors and analyzes the sensor data to identify possible problems in the system. One remote system monitoring platform is the Advanced Metering Infrastructure (AMI). AMI monitors an entity's utility usage using utility meters. A transmitter transmits utility usage data to a utility provider. For example, a home equipped with an AMI power meter transmits power data to an electricity utility provider in real time or at regular intervals. The utility provider collects usage data over a specific period and bills customers for utility usage. Additionally, the utility provider can analyze the usage data to identify customer and local usage requirements. Anomalies in an AMI system may occur as a result of theft, cyberattacks, meter failure, appliance or device failure, data corruption, or other issues. Inaccurate usage data may result in inaccurate forecasts and customer billing. Summary of the Invention

[0004] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

[0005] Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings. It should be noted that when reference is made to "one" or "an" embodiment in this disclosure, this does not necessarily refer to the same embodiment, but rather means at least one. [Brief explanation of the drawings]

[0006] [Figure 1A] FIG. 1 illustrates a system according to one or more embodiments. [Figure 1B]FIG. 1 illustrates a system according to one or more embodiments. [Figure 2A] FIG. 1 illustrates a set of example operations for operational data anomaly detection and correction, according to one or more embodiments. [Figure 2B] FIG. 1 illustrates a set of example operations for operational data anomaly detection and correction, according to one or more embodiments. [Figure 2C] FIG. 1 illustrates a set of example operations for operational data anomaly detection and correction, according to one or more embodiments. [Figure 3A] FIG. 1 illustrates an exemplary embodiment. [Figure 3B] FIG. 1 illustrates an exemplary embodiment. [Figure 4] FIG. 1 is a block diagram illustrating a computer system according to one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0007] Detailed Description In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding. One or more embodiments may be practiced without these specific details. Features described in one embodiment may be combined with features described in a different embodiment. In some instances, well-known structures and devices are described with reference to block diagram form in order to avoid unnecessarily obscuring the present invention.

[0008] 1.Overview 2. System Architecture 3. Operational data anomaly detection and correction 4. Exemplary Embodiments 5. Computer Networks and Cloud Networks 6. Miscellaneous; Extensions 7. Hardware Overview 1.Overview Utility providers collect utility usage data from meters to balance loads, plan future changes to the utility network, and bill clients for utility usage. However, many different events can prevent accurate utility monitoring, resulting in inaccurate planning and billing.

[0009] One or more embodiments include training a machine learning model on a combination of actual device data and fake data generated by adding noise to the actual data to predict device operational values ​​at individual intervals of a time series data set. The system identifies anomalies in the time series data based on the difference between the predicted value and the actual value. If the difference between the predicted value generated by the machine learning model and the actual value exceeds a threshold, the system identifies the particular data point, such as a meter reading, as an anomaly. The system ranks the anomaly and takes corrective action.

[0010] According to an exemplary embodiment, a system trains a deep learning long short-term memory (LSTM) encoder machine learning model to predict power usage values ​​for intervals in a time series set of data. The LSTM encoder ML model receives a set of time series data as input data and predicts power usage levels for a device. The system can calculate an anomaly value for each interval in the time series data. If the anomaly value exceeds a threshold, the system identifies the interval as anomalous. The system can rank the anomalous intervals based on various criteria and determine appropriate actions to correct the anomaly. According to one example, the system compares anomalous interval patterns to patterns associated with known issues, such as the installation of a new appliance in a home or equipment in a commercial or industrial environment. The patterns can be associated with theft, meter failure, or appliance or equipment failure. The system can rank the anomalies according to the patterns the system associates with the anomalies. According to another example, the system ranks the anomalies according to the severity of the anomaly. For example, if the actual power usage value is 100% of the predicted power usage value, the system will rank this anomaly higher than another anomaly whose actual power usage value exceeds the predicted power usage value by 10%.

[0011] One or more embodiments described and / or claimed herein may not be included in this General Summary section.

[0012] 2. System Architecture FIG. 1 illustrates a system 100 according to one or more embodiments. As shown in FIG. 1, the system 100 includes a device behavior monitoring platform 110, a data repository 120, monitored devices 130, and a network 140. In one or more embodiments, the system 100 may include more or fewer components than those illustrated in FIG. 1. The components illustrated in FIG. 1 may be local or remote from one another. The components illustrated in FIG. 1 may be implemented in software and / or hardware. Each component may be distributed across multiple applications and / or machines. Multiple components may be combined into a single application and / or machine. Operations described with respect to one component may instead be performed by another component.

[0013] The device performance monitoring platform 110 collects performance data from the monitored devices 130 and stores the performance data as historical device performance data 121. The monitored devices 130 may include meters such as utility meters. For example, the monitored devices 130 may be electricity meters that measure the amount of electricity used in a particular location, such as a home, business, farm, etc. The monitored devices 130 may transmit the electricity usage data to the device performance monitoring platform 110 at regular intervals.

[0014] The device behavior monitoring platform 110 includes a machine learning model engine 111. The machine learning model engine 111 is trained on a training dataset 122 to predict target behavior values ​​of the monitored device 130. The machine learning model engine 111 generates a training dataset of actual behavior data 123 from historical device behavior data. Each data point in the dataset includes (a) a device behavior value (e.g., usage within a defined time interval), (b) attributes associated with the value, including the time interval associated with the value and the weather conditions at the time the value was recorded, and (c) a label indicating that the value corresponds to actual behavior data. The machine learning model engine 111 includes a fake training data generator 112. The fake training data generator 112 generates a set of fake behavior data 124 based on the training dataset of the actual behavior data 123. For example, the fake training data generator 112 may randomly select a plurality of data points that are 20% to 40% of the data points in the actual behavior data training dataset 123. The fake training data generator 112 adds noise to the selected data points to generate the fake operational data training dataset 124. The fake training data generator 112 may add noise randomly, such as by randomly determining whether to add or subtract from the utilization values. Additionally, the fake training data generator 112 may add noise by randomly varying the utilization values ​​of the data points by between 5% and 500%. According to an exemplary embodiment, the fake training data generator 112 selects a cluster of consecutively occurring time-series data points to add noise to.

[0015] 1B illustrates a fake training data generator 112 applied to an embodiment in which the dataset is a time-series dataset. The fake training data generator 112 receives, for example, genuine time-series motion data 151 corresponding to the actual motion data 123 as input data. The noise generator 152 adds noise to the genuine time-series motion data 151 to generate fake time-series motion data 124 corresponding to the fake motion data 124 of FIG. 1A. The noise generator 152 includes a noise application selection engine 153 and a noise intensity determination engine 154. The noise application selection engine 153 determines which data points to select from the genuine time-series motion data 151 for noise addition. The noise application selection engine 153 selects a specific number of data points according to a specific pattern. The specific number of data points may include, for example, a specific percentage of genuine time-series motion data points. For example, if the authentic time-series motion data 151 includes 1000 data points, the noise application selection engine 153 may select 10% of the data points to add noise to. Additionally, the noise application selection engine 153 may add noise to the authentic time-series motion data 151 by removing data points from the motion data 151. For example, if the time-series data includes data points in one-hour time increments, the noise application selection engine 153 may remove data points for a specific time. According to one embodiment, the noise application selection engine 153 provides a randomization function that randomly selects data points in the authentic time-series motion data 151 to add noise to until a termination condition is met. For example, the noise application selection engine 153 may select data points to add noise to according to the randomization function until 10% of the data points are selected.

[0016] For a selected data point, the noise intensity determination engine 154 determines (a) the intensity of noise to add to the data point and (b) the positive or negative sign of the noise. The intensity of the noise can be either a percentage or an absolute value. For example, the noise intensity determination engine 154 can apply a randomization function that randomly applies an amount of noise within a range of 20% to 500% of the value of the data point. Alternatively, in an example where the data point represents kilowatt-hours, the noise intensity determination engine 154 can randomly apply an amount of noise within a range of 0.5 kWh to 200 kWh. The noise intensity determination engine 154 further determines whether to add noise by applying a positive sign to the noise or by applying a negative sign to the noise. According to one embodiment, the noise intensity determination engine 154 applies a randomization function to determine whether to apply a positive sign or a negative sign to the noise. For example, if a data point contains a value of 100, and the noise intensity determination engine 154 determines that noise having an intensity of 75 is to be added to the data point, the noise intensity determination engine 154 may further randomly apply a positive sign to the noise, resulting in a data point value of 175, or a negative sign to the noise, resulting in a data point value of 25.

[0017] The training dataset engine 155 generates a training dataset 158 ​​(corresponding to the combined training dataset 122 of FIG. 1A ) for training a machine learning model. The training dataset engine 155 combines the authentic time-series motion data 151 with the fake time-series motion data 124 to generate combined time-series data 156. The combined time-series data 156 includes the authentic time-series motion data 151, but with selected data points replaced with fake data points generated by the fake training data generator 112. For example, if a segment of data includes hourly data points for a particular day, the fake training data generator 112 can create fake data points for the 10:00 AM data point and the 2:00 PM data point. The combined time series data 156 includes, for that day, a true data point from 12:00 AM to 9:00 AM, a false data point from 10:00 AM, a true data point from 11:00 AM to 1:00 PM, a false data point from 2:00 PM, and a true data point from 3:00 PM to 11:00 PM.

[0018] The training dataset engine 155 further associates additional time-series attribute data with the data points of the combined time-series data 156 to generate a training dataset of combined time-series motion data 158. Examples of additional time-series attribute data 157 include weather conditions associated with the set of motion data 151 and calendar information (such as dates or specific events) associated with the set of motion data 151. The training dataset engine 155 provides the training dataset of combined time-series motion data 158 to the machine learning model engine 111 to train a machine learning model.

[0019] The machine learning model engine 111 trains the machine learning model 113 using a combined training dataset 122 (corresponding to training dataset 158 ​​in FIG. 1B ) that includes both real behavior data 123 and fake behavior data 124. The machine learning model engine 111 trains the machine learning model 113 to identify relationships (a) between attributes at the same data point and (b) between attributes of different data points within the same set of time-series data. According to one embodiment, the machine learning model is a deep learning long short-term memory (LSTM) autoencoder machine learning algorithm. The LSTM autoencoder machine learning algorithm is constructed using a set of LSTM “cells.” Each “cell” includes a “cell state” and a gate with parameters that are adjusted during training to teach the machine learning model relationships between data points in the time-series data. Each cell receives data via an input, outputs data via an output, and includes a “forget” gate. The input receives the data value associated with the current cell. For example, when an LSTM autoencoder algorithm is trained using time series data, one data point is associated with one cell, and subsequent time series data points are associated with subsequent cells. An input gate receives the data value associated with the current cell. A "forget" gate specifies parameters that determine which information from previous cells should be forgotten or ignored. The autoencoder structure of the machine learning algorithm maps input data from a high-dimensional state to a lower-dimensional state and then back to the original high-dimensional state. According to one or more embodiments, the LSTM autoencoder machine learning model includes 10,000 parameters that are tuned during training. For example, the LSTM autoencoder machine learning model may include 60,000 to 70,000 parameters that are tuned while training the machine learning model.

[0020] Although the LSTM algorithm is described above as an example, any machine learning model that can process time series data as input data and identify characteristics of specific intervals within the time series data can be used.

[0021] In some examples, one or more elements of the machine learning model engine 111 can use a machine learning algorithm to learn target behavioral data values ​​for the time-series data points. A machine learning algorithm is an algorithm that can iterate using a set of training data to learn a target model f that best maps a set of input variables to an output variable. The machine learning algorithm can include supervised and / or unsupervised components. Various types of algorithms can be used, such as linear regression, logistic regression, linear discriminant analysis, classification and regression trees, naive Bayes, k-nearest neighbors, learning vector quantization, support vector machines, bagging and random forests, boosting, backpropagation, and / or clustering.

[0022] In an embodiment, the set of training data includes a dataset and associated labels. The dataset is associated with input variables for the target model f (e.g., device operational values, time data, weather data, site data (e.g., single-family home, apartment, business, farm, factory, etc.)). Each data point is associated with a label indicating whether the data point is real or fake operational data. Training the model involves auto-encoding input vectors representing the input data, reducing the dimensionality in the model's hidden layer, and expanding the dimensionality of the model's hidden layer so that the dimensionality of the output layer is the same as that of the input layer. Training the model involves adjusting parameters so that the values ​​in the output layer are the same as the values ​​in the input layer. The training data can be updated, for example, based on feedback on the accuracy of the current target model f. The updated training data is fed back to the machine learning algorithm, and the machine learning algorithm updates the target model f.

[0023] The machine learning algorithm generates a target model f such that the target model f best fits the dataset of training data to the labels in the training data. Additionally, or alternatively, the machine learning algorithm generates a target model f such that when the target model f is applied to the dataset of training data, the greatest number of results determined by the target model f match the labels in the training data.

[0024] In an embodiment, the machine learning algorithm can iterate to predict device operating values ​​for time intervals in the time series data. In an embodiment, the set of training data includes actual operating data 123 and fake operating data 124. The training data set 122 is associated with a label that indicates whether a particular data point in the training data set corresponds to actual operating data or fake operating data.

[0025] The device performance monitoring platform 110 receives performance data 125 from monitored devices 130. For example, a utility provider may receive updates from an electric meter regarding the amount of electricity usage measured by the meter in real time, minute by minute, hourly, or daily. Upon receiving the performance data 125, the monitored device attribute data collection engine 114 collects additional attribute data 126 associated with the monitored device. For example, the monitored device attribute data collection engine 114 may identify the weather station closest to the device 130 generating the performance data. The device performance monitoring platform 110 may store weather data along with received performance data values ​​for a particular time interval. Other examples of attribute data that may be stored as data points along with performance data values ​​include supplemental utility data, such as whether a location includes an alternative power generator. The attribute data may include information about the type of building associated with the meter, such as a single-family home, apartment building, hotel, industrial facility, farm, factory, warehouse, or store. The attribute data may include the size of the building associated with the meter, such as the number of bedrooms in a home or the square footage of a building.

[0026] The machine learning model engine 111 embeds the monitored device's operational data 125 and attribute data 126 as vectors in a set of time series data. The machine learning model engine 111 feeds the time series data to the machine learning model 113 to generate predicted target operational values ​​115 for each subinterval in the time series data. For example, the set of time series data may include 30 days of power data divided into 30 subintervals, each corresponding to daily power usage and additional attributes for each of the 30 days.

[0027] The anomaly detection engine 116 analyzes the predicted target operational values ​​115 to detect anomalies among subintervals of the time series data. The anomaly detection engine 116 identifies data points corresponding to subintervals in the set of time series data that have anomalous values. In particular, the machine learning model 113 generates predicted target values ​​for data points in the time series data based on learned correlations (a) between attributes within the data points and (b) between attributes of different data points in the time series data. The anomaly detection engine 116 compares the predicted target operational values ​​115 with actual values ​​of the monitored device operational data 125. The anomaly detection engine 116 calculates an anomaly score for each data point in the time series data set based on the difference between the predicted target value of the data point and the actual value associated with the data point. If the difference between the predicted target value and the actual value exceeds a threshold, the system identifies the particular data point as anomalous.

[0028] The anomaly scoring engine 117 analyzes anomalous data points in the time series data and assigns rankings or weights to the anomalous data points. For example, the anomaly scoring engine 117 may assign a relatively higher weight to data points with higher anomaly scores than to data points with lower anomaly scores. Alternatively, the anomaly scoring engine 117 may assign a higher rank or weight to a cluster of data points that includes a particular pattern. The anomaly scoring engine 117 may identify patterns associated with meter failures, appliance failures, utility theft, utility transmission failures, and data transmission failures. The anomaly scoring engine 117 may assign different ranking values ​​to different identified patterns based on the severity of the corresponding failures.

[0029] The remediation engine 118 selects a remediation action to perform that is associated with the detected anomaly in the set of time series data. Examples of remediation actions include generating a notification to the customer and / or utility provider, remotely resetting a meter, and adjusting a price calculation on the customer's utility bill. The remediation engine 118 may select a remediation action according to a ranking or weight of the detected anomaly. For example, if the system detects an anomaly in the set of time series data that is associated with a resolved data transmission failure, the system may refrain from performing further remediation actions. If the system detects an anomaly in the set of time series data that is associated with a utility transmission failure, the system may trigger a notification to the utility service provider that repairs may be necessary.

[0030] Further embodiments and / or examples of computer networks are described below in Section 5 entitled "Computer Networks and Cloud Networks."

[0031] In one or more embodiments, data repository 120 is any type of storage unit and / or device for storing data (e.g., a file system, a database, a collection of tables, or any other storage mechanism). Furthermore, data repository 120 can include multiple different storage units and / or devices. The multiple different storage units and / or devices may or may not be of the same type and may or may not be located in the same physical location. Furthermore, data repository 120 may be implemented or executed on the same computing system as device behavior monitoring platform 110. Alternatively, or in addition, data repository 120 may be implemented or executed on a computing system separate from device behavior monitoring platform 110. Data repository 104 can be communicatively coupled to device behavior monitoring platform 110 via a direct connection or via a network.

[0032] The information describing training data sets, monitored device operation data, and monitored device attribute data may be implemented across any of the components in system 100. However, this information is presented in data repository 120 for purposes of clarity and explanation.

[0033] In one or more embodiments, device operation monitoring platform 110 refers to hardware and / or software configured to collect operation data, analyze the operation data by applying trained machine learning models to the operation data, and perform operations to identify and correct anomalies in the system using predictions generated by the trained machine learning models, as described herein. Examples of operations for identifying and correcting anomalies based on monitored device operation data are described below with reference to Figures 2A-2C.

[0034] In embodiments, the device behavior monitoring platform 110 is implemented in one or more digital devices. The term "digital device" generally refers to any hardware device that includes a processor. A digital device may refer to a physical device or a virtual machine that runs an application. Examples of digital devices include computers, tablets, laptops, desktops, netbooks, servers, web servers, network policy servers, proxy servers, general-purpose machines, specific-function hardware devices, hardware routers, hardware switches, hardware firewalls, hardware network address translators (NATs), hardware load balancers, mainframes, televisions, content receivers, set-top boxes, printers, mobile handsets, smartphones, personal digital assistants ("PDAs"), wireless receivers and / or transmitters, base stations, communication managers, routers, switches, controllers, access points, and / or client devices.

[0035] In one or more embodiments, interface 119 refers to hardware and / or software configured to facilitate communication between a user and device behavior monitoring platform 110. Interface 119 renders user interface elements and receives input via user interface elements. Examples of interfaces include graphical user interfaces (GUIs), command line interfaces (CLIs), tactile interfaces, and voice command interfaces. Examples of user interface elements include check boxes, radio buttons, drop-down lists, list boxes, buttons, toggles, text fields, date and time selectors, command lines, sliders, pages, and forms.

[0036] In an embodiment, different components of interface 119 are specified in different languages. The behavior of user interface elements can be specified in a dynamic programming language such as JavaScript. The content of user interface elements is specified in a markup language such as HyperText Markup Language (HTML) or XML User Interface Language (XUL). The layout of user interface elements is specified in a style sheet language such as Cascading Style Sheets (CSS). Alternatively, interface 119 is specified in one or more other languages, such as Java, C, or C++.

[0037] 3. Identifying and correcting anomalies based on operational data 2A-2C illustrate an example set of operations for identifying and correcting anomalies based on operational data, according to one or more embodiments. One or more of the operations illustrated in FIGS. 2A-2C may be modified, reordered, or omitted entirely. Thus, the particular order of the operations illustrated in FIGS. 2A-2C should not be construed as limiting the scope of one or more embodiments.

[0038] The system obtains a set of historical data (operation 202). The historical data includes operational data values ​​and attributes associated with the operation of one or more monitored devices. Examples of operational data values ​​include power or other utility usage levels and calendar data, such as timestamps associated with usage levels. Examples of attributes associated with operational data values ​​include weather data, location data, data describing the type of facility or building using the utility (e.g., commercial, retail, industrial), data describing the size of the building, such as number of bedrooms, number of rooms, square feet, etc., data describing the use of the building (e.g., private residence, hotel, data center, farm, factory, store, warehouse), and data specifying whether the location is associated with related features such as solar panels, wind turbines, swimming pools, etc.

[0039] The system generates a training data set from the historical operational data (operation 204). The training data set includes data points that specify operational data values ​​and attributes associated with the monitored devices that generate the operational values. For example, the data points may include kWh electricity consumption over an hour and the weather at a location associated with a meter that generates the kWh electricity consumption value.

[0040] The system generates a training dataset of fake operational data using the training dataset of actual historical operational data (operation 204). The system selects data points from the training subset of actual historical operational data. The system introduces noise to the selected data points to generate the dataset of fake operational data. For example, the system may copy the training dataset of actual operational data and introduce noise into each copy. Alternatively, the system may select a predetermined number of data points, such as 50%, from the training dataset and introduce noise to generate the training dataset of fake operational values. According to yet another alternative, the system (a) selects actual historical operational data, (b) generates fake operational data by adding noise, (c) applies a machine learning algorithm to the combined data, (d) determines the accuracy of the machine learning model obtained by applying the algorithm to the combined dataset, and (e) iteratively performs the process of repeating (a) through (d) if the accuracy of the prediction is below a threshold.

[0041] According to an exemplary embodiment, the system replaces a random number of data points in time series data with noisy false data points. For example, in a time series dataset containing 30 time series data points, the system can replace 10 data points randomly located among the 30 time series data points with noisy false data points. Additionally or alternatively, false data points may be introduced into the dataset on a cluster-by-cluster basis. For example, in a dataset containing 30 time series data points, the system may introduce false data points into three sets: three time series data points, three time series data points, and four time series data points. The sets may include consecutive time series data points. The system may calculate the noise to add to the clusters according to a randomization formula. For example, the noise value for each data point in a cluster may be random within a predetermined range. Alternatively, the noise value for each data point in a cluster may be random in intensity but have the same positive or negative sign. In other words, the system can add or subtract values ​​from the actual operating value to get the false operating value, but each data point in one cluster adds a value to create noise, and each data point in another cluster subtracts a value to add noise.

[0042] According to one or more embodiments, the system can add noise to a cluster in a specific pattern. For example, the system can randomly determine that a particular cluster of three time-series data points will receive a positive noise operating value of 10 kWh, where 10 kWh is a randomly selected value from a value range of 1 kWh to 50 kWh. The system can select one of the three time-series data points to have an operating value of 10 kWh. The system can apply a gradation formula to set operating values ​​for the other two data points. For example, the system can apply a formula that sets a noise value adjacent to a peak randomized noise value that is 10% lower in intensity than the peak randomized operating value. Alternatively, the system can add noise to a cluster by applying a bell curve formula.

[0043] According to one or more embodiments, adding noise to time series data points includes (a) selecting a number of data points within the time series dataset to add noise to; (b) selecting whether to add noise by increasing the magnitude of the data point's operational value or by decreasing the magnitude of the data point's operational value; and (c) selecting the magnitude by which to increase or decrease the operational value. The system can select the number of data points within the time series dataset to add noise to randomly. For example, if a particular time series dataset includes 360 data points, the system can randomly select data points from the 360 ​​data points to add noise to. Alternatively, the system can randomly select data points within a threshold number of data points. For example, the system can apply a formula that calculates the value of data points in the dataset to add noise at between 20% and 50%. If a particular time series dataset includes 360 data points, the system can randomly select the number of data points from the 360 ​​data points to add noise to within a range of between 72 and 180. Selecting the number of data points in the time series dataset to add noise to can include adding noise to a random distribution of multiple locations (i.e., operational data value generators, such as meters) that provide data that make up the dataset. For example, the system can apply a rule to add noise to time series data from a random selection of 20% of the locations that provide data that make up the dataset. According to one or more embodiments, the system adds noise to data points in the dataset according to a particular distribution, such as a Poisson distribution, an F distribution, a chi-squared distribution, a Student's t distribution, a normal distribution, and a uniform distribution.

[0044] The choice of whether to increase or decrease the magnitude of the operational value of a data point can be randomized so that any particular data point has a 50% chance of having an increased operational value magnitude and a 50% chance of having a decreased operational value magnitude.

[0045] The selection of the magnitude by which the noise is added to the operating value can be random within a specified range of values. For example, the system can apply a formula specifying that the operating value be randomly selected within a range of operating values ​​that vary from 5% to 500% of the actual operating value of the historical data point. Additionally or alternatively, the system can apply a rule that modifies the magnitude of the operating data value by a specified range of units. For example, if a data set contains power data measured in kilowatt-hours (kWh), the system can apply a rule that adds or subtracts 1 kWh to 10 kWh from the data point to which noise is added.

[0046] According to one or more embodiments, the system trains a machine learning model using operational data values ​​from multiple locations over a specific time period. For example, a training dataset can include electricity usage data from 300 separate power meters associated with 300 separate locations, such as residences. The training dataset can include time series data spanning weeks, months, or years. For example, a training dataset for training a machine learning model to identify outliers in one-month segments of time series data divided into daily intervals might span multiple different locations over two or more years. The system can add noise to data points within a specific date range in the training dataset or across the entire date range of the entire training dataset.

[0047] The system may apply a machine learning algorithm to the combined training set to train a machine learning model to predict the target operating value (operation 208). According to one embodiment, the machine learning algorithm receives as input data time series data acquired over a particular time period. The time period of the time series segments provided to the machine learning algorithm is separated into incremental intervals. Each interval has its own operating data value and its own additional attributes. For example, the algorithm may accept as input data a time series segment including 30 days of data points. Each data point specifies the amount of electricity consumed in kWh on a particular day, the weather conditions associated with the particular day, and any additional attributes included in the time series data. Based on the relationships (a) between the attributes at a particular data point and (b) between the attributes of different data points in the same time series segment, the system adjusts parameters of the machine learning algorithm to train the machine learning model.

[0048] According to one embodiment, the machine learning algorithm is a deep learning long short-term memory (LSTM) autoencoder machine learning algorithm. The LSTM autoencoder machine learning algorithm is constructed using a set of LSTM "cells." Each "cell" includes a "cell state" and a gate with parameters that are adjusted during training to teach the machine learning model relationships between data points in the time-series data. Each cell receives data through an input, outputs data through an output, and includes a "forget" gate. The input receives the data value associated with the current cell. For example, when the LSTM autoencoder algorithm is trained using time-series data, one data point is associated with one cell, and subsequent time-series data points are associated with subsequent cells. The input gate receives the data value associated with the current cell. The "forget" gate specifies parameters that determine which information from previous cells should be forgotten or ignored. The autoencoder structure of the machine learning algorithm maps input data from a high-dimensional state to a lower-dimensional state and then back to the original high-dimensional state. According to one or more embodiments, the LSTM autoencoder machine learning model includes 10,000 parameters that are adjusted during training. For example, the LSTM autoencoder machine learning model may include 60,000 to 70,000 parameters that are adjusted while training the machine learning model.

[0049] Although the LSTM algorithm is described above as an example, any machine learning model that can process time series data as input data and identify characteristics of specific intervals within the time series data can be used.

[0050] The system receives time-series data including operational values ​​from one or more monitored devices (operation 210). For example, the system may obtain electricity consumption data generated by an electric meter associated with a residence. According to another example, the system may obtain water consumption data generated by a water meter associated with an industrial facility. The system may obtain data in real time as the data is generated. Alternatively, the system may request or upload data in batches, such as at daily, weekly, or monthly intervals.

[0051] The system applies the trained machine learning model to the time series data to generate predicted values ​​for intervals in the time series data (operation 212). In particular, the system provides a set of time series data spanning a particular time period as input data to a machine learning model engine that stores and executes the machine learning model. The time series data includes time intervals within the time period. For example, the time period can be a week, a month, or multiple months. The time interval can be, for example, minutes, hours, or days. The system identifies data points corresponding to subintervals within the set of time series data that have outliers. In particular, the machine learning model generates predicted target values ​​for data points in the time series data based on learned correlations (a) between attributes within the data points and (b) between attributes of different data points in the time series data.

[0052] The system identifies anomalous data points in the time series data (operation 214). For each data point associated with an interval of the time series data, the system compares a predicted target value generated by the machine learning model based on an analysis of multiple data points in the time series data with the actual value of the interval in the time series data. The system calculates an anomaly score for each data point in the time series data set based on the difference between the data point's predicted target value and the actual value associated with the data point. In other words, the more the data point's actual operational data value differs from the predicted target operational value, the greater the anomaly score. If the difference between the predicted target value and the actual value exceeds a threshold, the system identifies the particular data point as anomalous.

[0053] Referring to FIG. 2B , the system analyzes anomalous data points in the time series data and assigns anomaly scores or weights to the anomalous data points (operation 216). For example, the system may assign a relatively higher weight to data points with higher anomaly scores than to data points with lower anomaly scores. Alternatively, the system may assign a higher weight to a cluster of data points that contain a particular pattern. The system may store utility usage patterns associated with meter failures, appliance failures, utility theft, and transmission failures (e.g., broken water mains for water utilities or shorted power lines for electric utilities). As an example, the system may identify a pattern of electricity usage dropping to a constant low utilization rate regardless of weather conditions as an anomaly associated with data transmission failures from solar panels installed on a building. The system may assign different ranking values ​​to different identified patterns of anomalous data points based on the severity of the corresponding failures. A detected anomaly corresponding to a pattern associated with data transmission failures may receive a lower ranking than an anomaly corresponding to a pattern associated with a power line failure.

[0054] The system determines whether the anomaly score exceeds a threshold (operation 218). The threshold may include one or both of (a) a threshold difference between the predicted value of a data point and the data point's actual value, and (b) a threshold number of data points in the set of anomalous data points. For example, the threshold may specify that 50% or more of a set of 20 data points are anomalous by more than 10% of the predicted value. A data set in which 40% of the data points are anomalous by more than 10% of the predicted value does not meet the threshold. The threshold may include multiple tiers. For example, the threshold may specify that (a) 50% or more of a set of 20 data points are anomalous by more than 10% of the predicted value, or (b) 10% or more of a set of 20 data points are anomalous by more than 30% of the predicted value. In other words, the threshold may be set to define a sliding scale requiring more anomalous data points of lower severity or fewer anomalous data points of higher severity.

[0055] If an anomalous data point or set of data points exceeds a threshold, the system selects a corrective action to perform associated with the detected anomaly in the set of time series data (operation 220). The system may select the corrective action according to the ranking or weight of the detected anomaly. For example, if the system detects an anomaly in the set of time series data associated with a resolved data transmission failure, the system may refrain from performing further corrective actions. If the system detects an anomaly in the set of time series data associated with a utility transmission failure, the system may trigger a notification to the utility service provider that repairs may be necessary. According to one exemplary embodiment, the system analyzes utility usage over a particular billing period to determine whether the amounts billed to the customer are accurate. If the system detects an anomaly in the time series data for a billing period that has a pattern associated with utility theft (such as unauthorized power use from a particular location), the system may generate a notification to the customer suggesting that the customer review their bill. Additionally or alternatively, the system may refrain from including charges associated with the anomalous usage on the customer's bill.

[0056] If the system determines that the anomalous data point or set of anomalous data points does not exceed the threshold, the system selects the next data point corresponding to the next time interval in the set of time series data for analysis (operation 222).

[0057] FIG. 2C illustrates a set of operations that may be performed in addition to, or instead of, the set of operations illustrated in FIG. 2B.

[0058] Similar to FIG. 2B, the system analyzes the anomalous data points to generate scores and / or weights associated with the anomalous data points, as discussed above (operation 216).

[0059] Based on one or more scores in the set of time series data, the system classifies the anomaly (operation 224). For example, the system may classify a set of anomaly scores for data points in the set of time series data as a meter failure, an appliance failure, a solar panel failure, utility theft, a new appliance installation, a utility provider failure, a data transmission error, and an increase or decrease in utility usage associated with an increase or decrease in occupancy at a residence or an operational change at a business facility.

[0060] The system determines whether the anomaly classification corresponds to a meter failure (operation 226). If the classification does not correspond to a meter failure, the system selects the next data point for analysis (operation 230). If the classification corresponds to a meter failure, the system stores or transmits predicted values ​​for the time series interval data points corresponding to the meter failure rather than measured values ​​for the time series interval data points corresponding to the meter failure. For example, the system may detect a meter failure for two days out of 30. Instead of or in addition to storing measured values ​​for the two days, the system stores predicted values ​​generated by the machine learning model. According to one exemplary embodiment, corrective action includes sending a notification about the meter failure to a service center. An operator can contact the customer to check the meter and schedule a time to service the meter. According to another exemplary embodiment, a bill displaying utility usage at different time intervals over a set period (such as daily electricity usage for a month) can display anomalous measured usage values ​​with a dashed line and display predicted usage values ​​for the same time interval overlaid on the measured usage values.

[0061] 4. Exemplary Embodiments Detailed examples are described below for clarity. The components and / or operations described below should be understood as specific examples that may not be applicable to a particular embodiment. Therefore, the components and / or operations described below should not be construed as limiting the scope of any of the claims.

[0062] 3A illustrates a system 300 for monitoring electricity usage using advanced metering infrastructure (AMI) technology. The system 300 includes residences 330a-330n. The residences 330a-330n are connected to an electricity utility network. Electricity usage at the residences 330a-330n is monitored by meters 333a-333n. The meters 333a-333n transmit electricity usage data to a meter monitoring platform 310 over a network 340. The network may include a global data network such as the Internet.

[0063] The machine learning model training data generator 311 generates a training data set consisting of genuine time series data and fake time series data acquired from the residence 330. As shown in FIG. 3B , the machine learning model training data generator 311 acquires genuine time series meter data 351 from the residence 330 and provides the genuine time series meter data 351 to the fake training data generator 312.

[0064] The noise generator 352 adds noise to the genuine time-series operational data 351 to generate the fake time-series meter data 324. The noise generator 352 includes a noise application selection engine 353 and a noise intensity determination engine 354. The noise application selection engine 353 determines which data points in the genuine time-series meter data 351 to select for noise application. The noise application selection engine 353 selects a specific number of data points according to a specific pattern. The specific number of data points includes a specific percentage of genuine time-series operational data points. In the exemplary embodiment shown in FIG. 3B , the genuine time-series meter data 351 includes one month of meter data divided into one-hour units. Each unit includes (a) a value corresponding to the amount of power consumption measured by the corresponding utility meter within the respective hour, and (b) a timestamp indicating the date and time when the power consumption was measured by the meter. The noise application selection engine 353 selects 20% of the data points in the set of authentic time-series meter data 351, or approximately 144 data points (depending on the number of days in a given month) corresponding to 144 hours in a 30-day month with 720 hours. The noise application selection engine 353 further removes 5% of the data points in the set of authentic time-series meter data 351, or approximately 36 data points corresponding to 36 hours in a 30-day month with 720 hours. According to one embodiment, the noise application selection engine 353 provides a randomization function to randomly select data points in the authentic time-series operational data 351 to add noise to until a termination condition is met. For example, the noise application selection engine 353 may randomly select data points corresponding to one-hour units from among the 720-hour units of the authentic time-series meter data 351 until 144 data points are selected.

[0065] For a selected data point, the noise intensity determination engine 354 determines (a) the intensity of the noise to add to the data point and (b) the sign of the noise. The noise intensity can be either a percentage or an absolute value. In the example shown in FIG. 3B, the noise intensity determination engine generates a random value between 20% and 100% of the magnitude of the power usage value of the data point. In addition, the noise intensity determination engine 354 further randomly applies a positive or negative sign to the random value.

[0066] The training dataset compilation engine 355 generates a training dataset 358 of combined time-series behavior data for training the machine learning model. The training dataset compilation engine 355 combines the authentic time-series behavior data 351 with the fake time-series behavior data 324 to generate combined time-series data 356. The combined time-series data 356 includes the authentic time-series behavior data 351, but with selected data points replaced with fake data points generated by the fake training data generator 312.

[0067] The training dataset compilation engine 355 further retrieves additional time-series attribute data along with the data points of the combined time-series data 356 to generate a training dataset of combined time-series operational data 358. The additional time-series attribute data 357 includes weather conditions at times corresponding to the timestamps of the authentic time-series meter data 351 and calendar information associated with the timestamps of the authentic time-series meter data 351.

[0068] The meter monitoring platform 310 provides a training dataset of the combined time-series operating data 358 to the machine learning model engine 312 to train a machine learning model 362 to predict a target operating value for the meter. The machine learning model may be applied to time-series data generated by one of the meters 333a-333n or another meter determined to have similar characteristics to the meters 333a-333n. For example, the meter monitoring platform 310 may apply the machine learning model to any meters within a specified geographic area and associated with a single-family dwelling.

[0069] The machine learning model engine 311 trains the machine learning model 362 to identify relationships (a) between attributes of the same data point and (b) between attributes of different data points within the same set of time series data. In the example shown in Figures 3A-3C, the machine learning model 362 is a deep learning long short-term memory (LSTM) autoencoder machine learning model.

[0070] 3C , when training machine learning model 362, meter monitoring platform 310 obtains target time-series meter data 361. Target time-series meter data 361 is data generated by one electricity meter, such as electricity meter 333a at residence 330a. Meter 333a associated with target time-series meter data 361 may be included in the set of meters that provide data for training machine learning model 362. Alternatively, meter 333a may not be included in the set of meters used to train machine learning model 362.

[0071] The target time-series meter data 361 can be net advanced metering infrastructure (AMI) data. Net AMI data includes values ​​that reflect not only the power provided to the customer by the utility provider, but also the power generated by the customer, such as the solar panels 331. For example, in some solar systems, the solar panels do not power the residence to which they are attached. Instead, the utility company provides all of the power to the residence, the solar panels send the power back to the power grid, and the utility company deducts the cost of the electricity from the utility bill associated with the residence. The target time-series meter data 361 can include meter values ​​that include the power provided by the utility provider minus the power generated by the solar panels and sold back to the utility provider. While solar panels are described in connection with FIG. 3A in the exemplary embodiment, additional power sources at the residence that generate power that can be sold to the utility company include wind turbines and geothermal generators.

[0072] Target time series meter data 361 corresponds to a one-month duration. Target time series meter data 361 includes a separate data point for each time interval within the one-month period. For example, if a month has 30 days, target time series data 361 includes 720 separate data points, each including (a) a meter value (e.g., power consumption in kWh), (b) a timestamp, and (c) additional attribute data 367, such as weather data or attribute data about the monitored location (e.g., that residence 330a includes solar panels 331, or that residence 330b includes swimming pool 332).

[0073] In the embodiment shown in FIG. 3 , a meter location identifier 363 determines the location of the meter generating the target time-series meter data 361. The location can be an address, coordinates, or area. An additional data collection engine 364 includes a weather station locator 365 that locates a weather station 335 a or 335 n in the vicinity of the meter location. The weather station can be the weather station closest to the meter generating the time-series meter data 361. A facility attribute data collection engine 366 collects additional data about the facility monitored by the meter, including unique power consumption characteristics. For example, the facility attribute data collection engine 366 can determine whether the residence 330 a includes solar panels 331, a swimming pool, is associated with a high-power consumption operation such as a data center, or is a multi-residential building.

[0074] The meter monitoring platform 310 provides the target time series meter data 361 and the additional time series attribute data 367 to the machine learning model 362 to generate a predicted electricity usage value for each data point of the target time series meter data 361. The set of predicted values ​​for the target time series meter data 361 is predicted time series meter data 368.

[0075] The anomaly detection engine 313 analyzes the predicted time series meter data 368 to detect anomalies within intervals of data. The anomaly detection engine 313 identifies data points within a monthly period that correspond to times with anomalous values. The anomaly scoring engine 314 calculates an anomaly score for each data point in the time series dataset based on the difference between the data point's predicted target value and the actual value associated with the data point. If the difference between the predicted target value and the actual value exceeds a threshold, the system identifies the particular data point as anomalous. A larger difference between the predicted and measured meter values ​​corresponds to a higher anomaly score. A smaller difference between the predicted and measured meter values ​​corresponds to a lower anomaly score.

[0076] The anomaly correction engine 315 selects a corrective action to perform that is associated with a detected anomaly in the target time series meter data 361. Examples of corrective actions include generating a notification to the customer and / or utility provider, remotely resetting the meter, and adjusting the price calculation on the customer's utility bill.

[0077] The system 300 includes a utility service delivery platform 318. Based on detecting an anomaly score that exceeds a threshold, the anomaly correction engine 315 can send data associated with the anomaly to the utility service delivery platform 318. For example, the anomaly correction engine 315 can detect a power usage pattern corresponding to power theft in the target time-series meter data 361. The meter monitoring platform 310 sends location data associated with the meter 333a and the potential theft to the utility service delivery platform 318. The utility service delivery platform 318 generates a ticket. A utility worker can investigate the meter associated with the ticket to determine whether theft is occurring or whether any other malfunctions or problems may be observed with the meter 333a.

[0078] The system 300 includes a utility billing platform 319. The utility billing platform 319 utilizes machine learning model 362 analysis of target time-series meter data 361 to determine whether the amount billed to the customer is accurate. Throughout a billing period, such as a particular month, meters 333n transmit net AMI utility usage data to a meter monitoring platform 310 maintained by the utility provider. The utility provider stores the utility usage data in a data repository. The meter monitoring platform compiles the net AMI time-series meter data for a particular billing period from the data repository before sending bills to customers. The meter monitoring platform 310 applies the model 362 to the monthly net AMI time-series meter data to identify specific dates and times with anomalous usage values. The utility billing platform 319 initiates corrective action according to the severity of the anomalous usage values. For example, the utility billing platform 319 can generate a notification to the customer if the anomaly is associated with a low ranking or severity. The utility billing platform 319 can generate an alert to a theft prevention unit if the anomaly ranking corresponds to potential utility theft. If the anomaly ranking is associated with a meter malfunction, the utility billing platform 319 can prompt a utility representative to exclude one or more days of charges from the customer's bill pending the outcome of a review of the charges.

[0079] While the embodiments described in FIGS. 3A-3C are described in terms of sets of time series data corresponding to one-hour intervals of one-month duration, embodiments include sets of time series data of different durations and different intervals. For example, according to one exemplary embodiment, the meter monitoring platform 310 analyzes utility usage data from meters to identify equipment failures and data transmission failures. The meter monitoring platform 310 can analyze utility usage data received from utility meters in real time or near real time. For example, the machine learning model 362 can be trained to receive as input data one-week segments of time series data consisting of one-hour intervals as separate data points within the one-week segments of time series data. Each day, the meter monitoring platform 310 can provide the machine learning model with the utility usage data for the previous seven days. The machine learning model 362 generates a prediction of a target utility usage value for each time interval in the one-week segments of time series data. The meter monitoring platform 310 generates an anomaly score based on the difference between the predicted target utility usage value and the actual utility usage value for each time interval in the one-week segments of time series data. The meter monitoring platform 310 can identify certain anomalies as corresponding to equipment and / or transmission failures. For example, if a particular time interval corresponds to a period of excessively hot or cold weather and also corresponds to a drop in power usage, the meter monitoring platform 310 can determine that an equipment failure has occurred based on historical patterns associated with power usage and extreme weather conditions. As another example, if a sequence of hourly intervals maintains the same power usage level when the system expects fluctuating power usage levels, the meter monitoring platform 310 can determine that a data transmission failure has occurred.

[0080] Based on the detection of anomalous time series data in the target time series meter data 361, the anomaly correction engine 315 can perform a corrective action using predicted time series values ​​generated by the machine learning model 362 to replace the anomalous time series values. The meter monitoring platform 310 can use the utility usage data for several different purposes, such as planning future development of the utility network, predicting load on the utility network, and billing customers for utility usage. When an equipment failure, utility transmission failure, or meter data transmission failure interrupts a series of time series data points with an anomalous data point, the system may not be able to accurately plan future development or bill customers. The anomaly correction engine 315 corrects the detected anomalous values ​​by replacing values ​​in the data storage or data transmission with values ​​predicted by the machine learning model 362. According to one example, the meter monitoring platform 310 may detect three anomalous data points in a one-month segment of the time series data. The meter monitoring platform 310 can replace the anomalous data point values ​​in the time series data with target values ​​generated by the machine learning model. The meter monitoring platform 310 can use a dataset including the substituted data point values ​​in place of the anomalous data point values ​​to identify trends. As another example, the utility billing platform 319 can replace a monetary value corresponding to the anomalous utility usage value with a substitute monetary value corresponding to the utility usage value predicted by the machine learning model 362. The utility usage value predicted by the machine learning model 362 is more likely to reflect actual utility usage than the anomalous measured utility usage value. Thus, the utility billing platform 319 can bill clients an amount that more closely corresponds to actual utility usage than the utility usage reflected in the anomalous data point value.

[0081] 5. Computer Networks and Cloud Networks In one or more embodiments, a computer network provides connectivity between a set of nodes. These nodes may be local and / or remote from one another. The nodes are connected by a set of links. Examples of links include coaxial cable, unshielded twisted cable, copper cable, optical fiber, and virtual links.

[0082] A subset of nodes implements computer networks. Examples of such nodes include switches, routers, firewalls, and network address translators (NATs). Another subset of nodes uses computer networks. Such nodes (also called "hosts") can run client processes and / or server processes. A client process makes a request for a computing service (such as running a particular application and / or storing a particular amount of data). A server process responds by performing the requested service and / or returning corresponding data.

[0083] A computer network may be a physical network including physical nodes connected by physical links. A physical node is any digital device. A physical node may be a hardware device with a specific function, such as a hardware switch, a hardware router, a hardware firewall, and a hardware NAT. Additionally or alternatively, a physical node may be a general-purpose machine configured to run various virtual machines and / or applications that perform their respective functions. A physical link is a physical medium connecting two or more physical nodes. Examples of links include coaxial cable, unshielded twisted cable, copper cable, and optical fiber.

[0084] A computer network may be an overlay network. An overlay network is a logical network implemented on top of another network (e.g., a physical network). Each node in the overlay network corresponds to a respective node in the underlying network. Thus, each node in the overlay network is associated with both an overlay address (for addressing the overlay node) and an underlay address (for addressing the underlay node that implements the overlay node). An overlay node may be a digital device and / or a software process (e.g., a virtual machine, an application instance, or a thread). Links connecting overlay nodes are implemented as tunnels through the underlying network. Overlay nodes at both ends of the tunnel treat the underlying multi-hop path between the overlay nodes as a single logical link. Tunneling is performed through encapsulation and decapsulation.

[0085] In embodiments, a client may be local and / or remote to a computer network. A client may access a computer network through a private network or another computer network, such as the Internet. A client may communicate a request to the computer network using a communication protocol, such as Hypertext Transfer Protocol (HTTP). The request is communicated through an interface, such as a client interface (e.g., a web browser), a program interface, or an application programming interface (API).

[0086] In an embodiment, a computer network provides connectivity between clients and network resources. The network resources include hardware and / or software configured to run server processes. Examples of network resources include processors, data storage devices, virtual machines, containers, and / or software applications. The network resources are shared among multiple clients. The clients request computing services from the computer network independently of each other. The network resources are dynamically allocated to requests and / or clients on an on-demand basis. The network resources allocated to each request and / or client may be scaled up or down based on, for example, (a) the computing services requested by a particular client, (b) the aggregated computing services requested by a particular tenant, and / or (c) the aggregated computing services requested from the computer network. Such a computer network may also be referred to as a "cloud network."

[0087] In an embodiment, a service provider offers a cloud network to one or more end users. Various service models can be implemented by the cloud network, including, but not limited to, Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). In SaaS, the service provider offers end users the ability to use the service provider's applications running on the network resources. In PaaS, the service provider offers end users the ability to deploy custom applications on the network resources. The custom applications can be created using programming languages, libraries, services, and tools supported by the service provider. In IaaS, the service provider offers end users the ability to provision the processing, storage, network, and other basic computing resources provided by the network resources. Any application, including an operating system, can be deployed on the network resources.

[0088] In embodiments, a computer network may implement various deployment models, including, but not limited to, private clouds, public clouds, and hybrid clouds. In a private cloud, network resources are provisioned for exclusive use by a specific group of one or more entities (the term "entity" as used herein refers to a company, organization, person, or other entity). The network resources may be local and / or remote to the premises of the specific group of entities. In a public cloud, cloud resources are provisioned for multiple entities (also referred to as "tenants" or "customers") that are independent of each other. The computer network and its network resources are accessed by clients corresponding to different tenants. Such a computer network may be referred to as a "multi-tenant computer network." Several tenants may use the same specific network resources at different times and / or at the same time. The network resources may be local and / or remote to the tenant's premises. In a hybrid cloud, the computer network includes a private cloud and a public cloud. An interface between the private cloud and the public cloud enables data and application portability. Data stored in the private cloud and data stored in the public cloud may be exchanged through the interface. Applications implemented in a private cloud and applications implemented in a public cloud may have dependencies on each other, and calls from applications in a private cloud to applications in a public cloud (and vice versa) may be made through an interface.

[0089] In embodiments, tenants of a multi-tenant computer network are independent of one another. For example, the business or operations of one tenant may be separate from the business or operations of another tenant. Different tenants may require different network requirements from the computer network. Examples of network requirements include processing speed, data storage, security requirements, performance requirements, throughput requirements, latency requirements, resilience requirements, quality of service (QoS) requirements, tenant isolation, and / or consistency. The same computer network may be required to implement the different network requirements required by different tenants.

[0090] In one or more embodiments, tenant isolation is implemented in a multi-tenant computer network to ensure that applications and / or data of different tenants are not shared with each other. Various tenant isolation approaches may be used.

[0091] In an embodiment, each tenant is associated with a tenant ID. Each network resource in the multi-tenant computer network is tagged with a tenant ID. A tenant is granted access to a particular network resource only if the tenant and the particular network resource are associated with the same tenant ID.

[0092] In an embodiment, each tenant is associated with a tenant ID. Each application implemented by the computer network is tagged with a tenant ID. Additionally or alternatively, each data structure and / or dataset stored by the computer network is tagged with a tenant ID. A tenant is granted access to a particular application, data structure, and / or dataset only if the tenant and the particular application, data structure, and / or dataset are associated with the same tenant ID.

[0093] As one example, each database implemented by a multi-tenant computer network may be tagged with a tenant ID. Only the tenant associated with the corresponding tenant ID may access the data in a particular database. As another example, each entry in a database implemented by a multi-tenant computer network may be tagged with a tenant ID. Only the tenant associated with the corresponding tenant ID may access the data in a particular entry. However, a database may be shared by multiple tenants.

[0094] In an embodiment, the subscription list indicates which tenants have authorization to access which applications. For each application, a list of tenant IDs of tenants authorized to access the application is stored. A tenant is granted access to a particular application only if the tenant ID of the tenant is included in the subscription list corresponding to the particular application.

[0095] In an embodiment, network resources (such as digital devices, virtual machines, application instances, and threads) corresponding to different tenants are separated into tenant-specific overlay networks maintained by a multi-tenant computer network. As an example, packets from any source device in a tenant overlay network can be sent only to other devices within the same tenant overlay network. To prohibit any transmission from a source device on a tenant overlay network to a device in another tenant overlay network, an encapsulation tunnel is used. Specifically, a packet received from a source device is encapsulated within an outer packet. The outer packet is sent from a first encapsulation tunnel endpoint (communicating with the source device in the tenant overlay network) to a second encapsulation tunnel endpoint (communicating with a destination device in the tenant overlay network). The second encapsulation tunnel endpoint decapsulates the outer packet to obtain the original packet sent by the source device. The original packet is sent from the second encapsulation tunnel endpoint to a destination device in the same specific overlay network.

[0096] 6. Miscellaneous; Extensions Embodiments are directed to systems that include one or more devices that include a hardware processor and are configured to perform any of the operations described herein and / or recited in any of the appended claims.

[0097] In an embodiment, a non-transitory computer-readable storage medium includes instructions that, when executed by one or more hardware processors, cause any of the operations described herein and / or recited in any of the claims to be performed.

[0098] Any combination of the features and functions described herein may be used in accordance with one or more embodiments. In the foregoing specification, embodiments have been described with reference to numerous specific details that may vary from implementation to implementation. Accordingly, the specification and drawings should be considered in an illustrative rather than a restrictive sense. The sole and exclusive indication of the scope of the present invention and what the applicants intend to be the scope of the present invention is the literal and equivalent scope of the set of claims issuing from this application, in the specific form from such claims, including any subsequent amendments.

[0099] 7. Hardware Overview According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. These special-purpose computing devices may be hardwired to perform these techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or network processing units (NPUs) permanently programmed to perform these techniques, or may include one or more general-purpose hardware processors programmed to perform these techniques according to program instructions in firmware, memory, other storage, or a combination thereof. Such special-purpose computing devices may also combine custom hardwired logic, ASICs, FPGAs, or NPUs with custom programming to perform these techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices, or any other devices incorporating hardwired and / or program logic to implement these techniques.

[0100] For example, Figure 4 is a block diagram illustrating a computer system 400 in which embodiments of the present invention may be implemented. Computer system 400 includes a bus 402 or other communication mechanism for communicating information, and a hardware processor 404 coupled with bus 402 for processing information. Hardware processor 404 may be, for example, a general-purpose microprocessor.

[0101] Computer system 400 also includes a main memory 406, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 402 for storing information and instructions executed by processor 404. Main memory 406 may also be used for storing temporary variables or other intermediate information during execution of instructions executed by processor 404. Such instructions, when stored on a non-transitory storage medium accessible to processor 404, render computer system 400 a special-purpose machine customized to perform the operations specified in the instructions.

[0102] Computer system 400 further includes a read only memory (ROM) 408 or other static storage device coupled to bus 402 for storing static information and instructions for processor 404. A storage device 410, such as a magnetic disk or optical disk, is provided and coupled to bus 402 for storing information and instructions.

[0103] Computer system 400 can be coupled via bus 402 to a display 412, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 414, including alphanumeric and other keys, is coupled to bus 402 for communicating information and command selections to processor 404. Another type of user input device is a cursor control 416, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to processor 404 and for controlling cursor movement on display 412. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify a position in a plane.

[0104] Computer system 400 may implement the techniques described herein using customized hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, when combined with the computer system, makes computer system 400 a special-purpose machine or programs it to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 400 in response to processor 404 executing one or more sequences of one or more instructions contained in main memory 406. Such instructions may be read into main memory 406 from another storage medium, such as storage device 410. Execution of the sequences of instructions contained in main memory 406 causes processor 404 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0105] The term "storage medium," as used herein, refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such storage media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 410. Volatile media include dynamic memory, such as main memory 406. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, content-addressable memory (CAM), and ternary content-addressable memory (TCAM).

[0106] Storage media are distinct from but may be used in conjunction with transmission media. Transmission media involves transferring information between storage media. For example, transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise bus 402. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0107] Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 404 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 400 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector can receive the data carried in the infrared signal and appropriate circuitry can place the data on bus 402. Bus 402 carries the data to main memory 406, from which processor 404 retrieves and executes the instructions. The instructions received by main memory 406 may optionally be stored on storage device 410 either before or after execution by processor 404.

[0108] Computer system 400 also includes a communication interface 418 coupled to bus 402. Communication interface 418 provides a two-way data communication coupling to a network link 420 that is connected to a local network 422. For example, communication interface 418 may be an Integrated Services Digital Network (ISDN) card, cable modem, satellite modem, or modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 418 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. A wireless link may also be implemented. In any such implementation, communication interface 418 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0109] Network link 420 typically provides data communication through one or more networks to other data devices. For example, network link 420 may provide a connection through local network 422 to a host computer 424 or to data equipment operated by an Internet Service Provider (ISP) 426. ISP 426 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the "Internet" 428. Local network 422 and Internet 428 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 420 and through communication interface 418, which carry the digital data to and from computer system 400, are exemplary forms of transmission media.

[0110] Computer system 400 can send messages and receive data, including program code, through the network(s), network link 420 and communication interface 418. In the Internet example, a server 430 might transmit a requested code for an application program through Internet 428, ISP 426, local network 422 and communication interface 418.

[0111] The received code may be executed by processor 404 as it is received, and / or stored in storage device 410, or other non-volatile storage for later execution.

[0112] In the above specification, embodiments of the present invention have been described with reference to numerous specific details that may vary from embodiment to embodiment. Accordingly, the specification and drawings should be considered in an illustrative rather than a restrictive sense. The sole and exclusive indication of the scope of the present invention and what the applicants intend to be the scope of the present invention is the literal and equivalent scope of the set of claims originating from this application, in the specific form derived from such claims, including any subsequent amendments.

Claims

1. A non-transitory computer-readable medium containing instructions that, when executed by one or more hardware processors, cause operations to be performed, the operations including: training a machine learning model to predict a target operational value of the monitored device, said training comprising: obtaining first subsets of training data comprising historical operational data of one or more monitored devices, each first subset of training data comprising: a time series of operational values ​​of the one or more monitored devices; for each subset of the first subset of training data, a label identifying the subset as actual motion data; Including, at least, selecting a second subset of the training data from the first subset of the training data; applying noise to a second subset of the training data; applying, to each subset of the second subset of training data, a label that identifies the subset as erroneous motion data; generating a second subset of the training data; training the machine learning model based on the first subset of training data and the second subset of training data; and the operations include: receiving specific time series operational data associated with a first monitored device; applying the machine learning model to the particular time-series motion data to generate predicted target motion data; comparing a first value of a first data point of the received particular time-series motion data to a second value corresponding to a predicted target motion data value associated with the first data point; identifying the first value as anomalous based on determining that the difference between the first value and the second value exceeds a threshold; 10. A non-transitory computer-readable medium, further comprising:

2. The operation is comparing a third value of a second data point of the received particular time-series motion data to a fourth value corresponding to a predicted target motion data value associated with the second data point; identifying the third value as anomalous based on determining that a difference between the third value and the fourth value exceeds the threshold; assigning a first weight to the first value in response to determining that the difference between the first value and the second value exceeds the threshold by a first amount; assigning a second weight to the third value in response to determining that the difference between the third value and the fourth value exceeds the threshold by a second amount; performing a corrective action associated with the first value based on determining that the first weight satisfies a correction criterion; and Discontinuing execution of any corrective action associated with the third value based on determining that the second weight does not satisfy the correction criteria; and The non-transitory computer-readable medium of claim 1 , further comprising:

3. 10. The non-transitory computer-readable medium of claim 1, wherein each first subset of training data further includes attributes associated with the one or more monitored devices, the attributes including at least weather conditions in a vicinity of the one or more monitored devices.

4. 4. The non-transitory computer-readable medium of claim 3, wherein the attributes associated with the one or more monitored devices further include at least one of temperature data, dew point data, home type data, and demographic data.

5. Applying the noise to a second subset of the training data includes: selecting a set of data points from the first subset of training data; randomly selecting an addition or subtraction operation to be performed; applying a random positive or negative variation to the value of each data point in the set of selected data points based on the randomly selected addition or subtraction operation, within a threshold variation level; 10. The non-transitory computer-readable medium of claim 1, comprising:

6. Receiving target time-series motion data includes: receiving time series operational values ​​and location data associated with the first monitored device; identifying weather sensors within a threshold distance of the first monitored device based on the location data; obtaining weather data generated by the weather sensor associated with the received time series of operating values; generating a vector including the time series of operating values ​​and the weather data; Including, The non-transitory computer-readable medium of claim 1 , wherein the machine learning model is applied to the vector.

7. 10. The non-transitory computer-readable medium of claim 1, wherein the machine learning model is based on a deep learning long short-term memory (LSTM) type model.

8. The operation is comparing a third value of a second data point of the received particular time-series motion data to a fourth value corresponding to a predicted target motion data value associated with the second data point; identifying the third value as normal based on determining that the difference between the third value and the fourth value does not exceed the threshold; and The non-transitory computer-readable medium of claim 1 , further comprising:

9. The operation is Associating a monetary value with a particular user account corresponding to the first monitored device based on operational values ​​of the first monitored device over a defined period of time; based on determining that the first value of the first data point is anomalous, removing from the monetary value associated with the particular user account a first monetary value associated with the first value of the first data point; The non-transitory computer-readable medium of claim 1 , further comprising:

10. The non-transitory computer-readable medium of claim 1 , wherein the time-series operational data comprises power data measured by an electric utility meter at a particular location.

11. A method comprising the operations of any of claims 1 to 10.

12. 1. A system comprising: one or more processors; a memory having stored thereon instructions that, when executed by the one or more processors, cause the system to perform the operations of any one of claims 1 to 10; A system comprising:

12. A system comprising means for performing the operations of any of claims 1 to 10.

Images