Performing anomaly detection using a set of detection models

Abstract

Techniques for anomaly detection are described. An example method includes receiving a request to detect anomalies in time series data using one or more detectors using an anomaly detection service, configuring the anomaly detection service by generating a configuration for the anomaly detection service based at least in part on one or more of: the request, the time series data, and metadata, where the configuration identifies at least one particular detector of the one or more detectors, and configuring the anomaly detection service using the generated configuration, evaluating the time series data for anomalies using the configured anomaly detection service by observing potential anomalous behavior using the identified at least one particular detector of the one or more detectors, and generating an anomaly indication.

Description

Background Technology

[0001] Anomaly detection is a fundamental problem in data mining. Some systems in the market generate anomalies in time series based on predictive models such as Holt-Winters. The output of these models is a probability distribution of future values ​​of the time series. Anomalies are generated by these models when the time series input has a lower probability than the prediction. Attached Figure Description

[0002] Various embodiments according to this disclosure will be described with reference to the accompanying drawings, in which:

[0003] Figure 1 An implementation scheme is shown that includes a network of providers offering anomaly detection services.

[0004] Figure 2 Examples of anomalies that can be detected by an anomaly detection service according to some implementation schemes are shown.

[0005] Figure 3 Examples of anomalies that can be detected by an anomaly detection service according to some implementation schemes are shown.

[0006] Figure 4 An implementation scheme for an anomaly detection service, such as this anomaly detection service, is shown.

[0007] Figure 5 This is a flowchart illustrating the operation of a method for anomaly detection according to some implementation schemes.

[0008] Figure 6 An exemplary provider network environment is shown according to some implementation schemes.

[0009] Figure 7 This is a block diagram of an exemplary provider network that provides storage services and hardware virtualization services to customers according to some implementation schemes.

[0010] Figure 8 This is a block diagram illustrating an exemplary computer system that can be used in some implementations. Detailed Implementation

[0011] This disclosure relates to methods, apparatus, systems, and non-transitory computer-readable storage media for anomaly detection services. According to some embodiments, the anomaly detection service uses a set of models to observe potential anomalous behavior. Generally, potential anomalous behavior may be interpretable as anomaly. Domain knowledge is utilized to select which models(s) to use to observe potential anomalous behavior in time series data for specific problems (e.g., change point problems, threshold violations, missing values, etc.).

[0012] Figure 1An implementation scheme of a provider network including anomaly detection services is shown. Anomaly detection service 110(A) or 110(B) incorporates operational domain knowledge of metrics into model selection and model training / use. In some implementations, anomaly detection service 110(A) or 110(B) generates interpretable anomalies from time-series data provided by data sources such as internal data source 114 and external data source 116. Note that in some implementations, as shown, the anomaly detection service is part of metrics service 112 (anomaly detection service 110(A)), and in some implementations, the anomaly detection service is external to metrics service 112 (anomaly detection service 110(B)). Metrics service 112 may perform one or more of the following non-anomaly detection functions: collecting and storing logs, collecting metrics (including time-series data) that can be default or custom, aggregating metrics, generating alerts, automatically scaling resources, etc.

[0013] Data sources 114 and 116 provide time-series data for evaluation by anomaly detection services 110(A) or 110(B) for potential anomalies. It is important to note that an anomaly does not necessarily equate to an alert, as not all requesters want alerts for every anomaly, but only for "critical" issues. For example, an anomaly that does not indicate an unhealthy system may be of interest but not necessarily something requiring action. Therefore, in some implementations, the captured anomalies are a superset of the anomalies that generate alerts.

[0014] Anomaly detection services 110(A) or 110(B) include or have access to multiple different machine learning models (sets) or other anomaly detection mechanisms (e.g., statistical). These models / algorithms analyze metrics of systems and applications, determine normal baselines, and reveal anomalies with minimal user intervention. Implementing anomaly detection for metrics involves using past data of the metric to induce machine learning algorithms or models in the application to create models of the expected values ​​of the metric. The models evaluate both the trend of the metric and hourly, daily, and weekly patterns.

[0015] Based on the metric to be evaluated (time series data), the anomaly detection service 110(A) or 110(B) is configured to use the best model, etc. Anomaly types and alert types vary depending on the metric type, application type, etc. Annotating anomalies with types allows for flexible selection of the correct anomaly type based on metrics and / or user feedback. User feedback and unique metric requirements are used to adjust which model to use (e.g., annotations based on datasets, anomalies, etc.), retrain the model, set different reporting thresholds, etc. For example, user feedback can be used to: adjust the importance or insights of anomalies, adjust what types of anomalies are used to create insights for metrics, etc. For example, if user feedback indicates that seasonal anomalies are useless for a certain metric, those anomalies are silenced. To obtain better feedback, anomaly types are used to generate explanations to be displayed to the user. For example, if an anomaly is caused by a trend change, an explanation of how the trend changed can be shown to the user.

[0016] Domain knowledge is used to annotate metrics and select which models the anomaly detection service 110(A) or 110(B) uses to create anomalies and the rules for threshold-based anomalies. Domain knowledge can also be used to determine when to issue an alert for an anomaly.

[0017] To handle unknown metrics, the anomaly detection service 110(A) or 110(B) can start conservatively, generating alerts only for high threshold violations and selecting alternative detectors based on metadata annotator analysis. For example, if the detector configuration generator classifies time series as continuous and stationary, change point and missing value detectors are typically selected. Alerts are tuned based on user feedback. For instance, for custom metrics, users can provide feedback that threshold violations at the lower end of the metric are not alarm-prone, and therefore no new alerts are generated for such events.

[0018] Provider Network 100 (or “cloud” provider network) provides users with the ability to utilize one or more of various types of computing-related resources, such as computing resources (e.g., executing virtual machine (VM) instances and / or containers, executing batch jobs, executing code without provisioning servers), data / storage resources (e.g., object storage, block-level storage, data archive storage, databases and database tables, etc.), network-related resources (e.g., configuring virtual networks (including multiple sets of computing resources), content delivery networks (CDN), domain name services (DNS)), application resources (e.g., databases, application build / deployment services), access policies or roles, identity policies or roles, machine images, routers, and other data processing resources, etc. These and other computing resources may be provided as services, such as hardware virtualization services that can execute computing instances, storage services that can store data objects, etc. Users of Provider Network 100 (or “customers”) may utilize one or more user accounts associated with a customer account, but these items may be used somewhat interchangeably depending on the context of use. Users can interact with provider network 100 across one or more intermediate networks 106 (e.g., the Internet) via one or more interfaces, such as through application programming interface (API) calls, or via a console implemented as a website or application. An API refers to an interface and / or communication protocol between a client and a server, such that if a client issues a request in a predefined format, the client should receive a response in a specific format or initiate a defined action. In the context of a cloud provider network, an API provides a gateway for customers to access cloud infrastructure by allowing customers to obtain data from or initiate actions within the cloud provider network, thereby enabling the development of applications that interact with resources and services hosted within the cloud provider network. APIs can also enable different services within the cloud provider network to exchange data with each other. One or more interfaces may be part of or act as the front end of the control plane of provider network 100, which includes "back-end" services that support and implement services that can be provided more directly to customers.

[0019] For example, a cloud provider network (or simply "the cloud") typically refers to a large pool of accessible virtualized computing resources, such as compute, storage, and networking resources, applications, and services. The cloud provides convenient, on-demand network access to a shared pool of configurable computing resources that can be programmatically provisioned and deployed in response to client commands. These resources can be dynamically provisioned and reconfigured to suit variable loads. Therefore, cloud computing can be viewed as applications delivered as a service over a publicly accessible network (e.g., the Internet, cellular networks) and the hardware and software in the cloud provider's data centers that provide those services.

[0020] Cloud provider networks can be structured into multiple zones, where a zone is a geographical area in which the cloud provider clusters its data centers. Each zone comprises multiple (e.g., two or more) Availability Zones (AZs) interconnected via a dedicated high-speed network (e.g., fiber optic communication connection). An AZ (also called an Availability Domain, or simply a "zone") provides an isolated fault domain and includes one or more data center facilities that have separate power, networking, and cooling from those data center facilities in another AZ. A data center refers to a physical building or rack that houses and provides power and cooling to the servers of the cloud provider's network. Preferably, AZs within a zone are far enough apart that a natural disaster (or other fault-inducing event) will not simultaneously affect or take down more than one AZ.

[0021] Customers can connect to the cloud provider's Availability Zones (AZs) via publicly accessible networks (e.g., the Internet, cellular networks), such as through a Switching Center (TC). The TC is the primary backbone location connecting customers to the cloud provider's network and can be co-located with other network provider facilities (e.g., Internet Service Providers, telecommunications providers) and securely connected (e.g., via VPN or direct connection) to the AZ. Two or more TCs can operate in each zone for redundancy. Zones are connected to a global network comprising dedicated network infrastructure (e.g., fiber optic connections controlled by the cloud provider) that connects each zone to at least one other zone. The cloud provider's network delivers content from points of presence (or "POPs") outside but networked with these zones, via edge locations and zone edge caching servers. This partitioning and geographical distribution of computing hardware enables the cloud provider's network to provide customers with low-latency resource access globally with high fault tolerance and stability.

[0022] To provide these and other computing resource services, provider network 100 typically relies on virtualization technology. For example, virtualization technology can be used to provide users with the ability to control or utilize computing resources (e.g., “computing instances,” such as VMs using a guest operating system (O / S) that operate using a hypervisor that may or may not operate on top of the underlying host O / S, containers that may or may not operate within the VM, and computing instances that can execute on “bare metal” hardware without an underlying hypervisor), where one or more computing resources can be implemented using a single electronic device. Thus, users can directly utilize computing resources hosted by the provider network (e.g., provided by a hardware virtualization service) to perform various computing tasks. Alternatively or additionally, users can indirectly utilize computing resources by submitting code executed by the provider network (e.g., via an on-demand code execution service), which in turn utilizes one or more computing resources to execute the code, typically without the user having any control or knowledge of the one or more underlying computing instances involved.

[0023] For example, in various implementations, a "serverless" function may include on-demand executable code provided by a user or other entity, such as the provider network itself. The serverless function may be maintained within provider network 100 by an on-demand code execution service and may be associated with a specific user or account or generally accessible to multiple users / accounts. The serverless function may be associated with a Uniform Resource Locator (URL), Uniform Resource Identifier (URI), or other references that can be used to invoke the serverless function. The serverless function may be executed by computing resources (such as virtual machines, containers, etc.) when triggered or invoked. In some implementations, the serverless function may be invoked via an Application Programming Interface (API) call or a specially formatted Hypertext Transfer Protocol (HTTP) request message. Therefore, users can define on-demand executable serverless functions without requiring users to maintain dedicated infrastructure to execute them. Alternatively, serverless functions may be executed on demand using resources maintained by provider network 100. In some implementations, these resources may be maintained in a "ready" state (e.g., having a pre-initialized runtime environment configured to execute serverless functions), allowing near real-time execution of serverless functions.

[0024] The circles containing numbers indicate exemplary activity flows. At circle 1, requesting device 118 sends a request to provider network 10 to configure anomaly detection service 110(B). An exemplary content of such a request will be described in detail later.

[0025] At circle 2, after configuring the anomaly detection service 110(B) (including selecting one or more ML models for anomaly detection, etc.), data source 116 provides time-series data to be evaluated. In this illustration, this time-series data is consumed by metrics service 112 and then provided to the anomaly detection service 110(B) at circle 3. The anomaly detection service 110(B) evaluates the time-series data at circle 4 and determines whether an anomaly exists. The anomaly detection service 110(B) sends an indication of the anomaly to metrics service 112, making it available to the user (e.g., requesting device 118) at circle 6.

[0026] Figure 2 Examples of anomalies that can be detected by an anomaly detection service 110(A) or (B) according to some implementations are shown. Specifically, what is detected are change points. In this example, two change points are detected. In this illustration, the anomaly detection model selected to determine when a change point occurs learns new behavior in 500 data points. The first "event" begins at t = 1000 and is detected at t = 1050. Specifically, the average increase in the detected change point is significant. The first event ends at t = 1500. The second "event" begins at t = 1900 and is detected at t = 2050. Specifically, the average decrease in the detected change point is significant. The second event ends at t = 2500. The output of the anomaly detection service 110(A) or (B) includes the event duration (start / stop time) and type. In some implementations, a score is also output. The event is typically the thing of interest, the score is the degree of difference between the event and the expected value, and the type is the type of anomaly.

[0027] Figure 3 Examples of anomalies that can be detected by anomaly detection service 110(A) or (B) according to some implementation schemes are shown. Specifically, the detected anomalies are threshold-based. In this example, the "event" begins at t = 24000 and is detected at t = 24003. The event type is above the threshold.

[0028] Figure 4 An implementation scheme of an anomaly detection service such as the anomaly detection service 110(A) or 110(B) is shown. This anomaly detection service 110(A) or 110(B) typically consists of one or more of the following: indicator browser 401, detector configuration generator 405, data ingestor 403, and anomaly generator 411.

[0029] In some implementations, the metrics browser 401 scans metrics (e.g., from the metrics service 112) to determine which metrics need to be added or removed for anomaly detection. In this way, the metrics browser 401 determines which metrics to add / remove for data ingestion and how to configure the anomaly generator 411. It should be noted that in some implementations, metrics are added / removed at the request of a specific user. In some implementations, the metrics browser 401 generates metric metadata.

[0030] Data ingestor 403 acquires time-series data from a data source for evaluation by anomaly generator 411. In some implementations, data ingestor 403 formats the time-series data before presenting it to anomaly generator 411. Data sources include databases, streams (such as video, audio, and data), etc.

[0031] The detector configuration generator 405 receives metric metadata (domain knowledge, such as known latency metrics), time series data (metrics), system update statistics 413 (global usage knowledge, such as analysis of data usage in the provider network), and / or information from the metric browser 401 to determine which detector 407 of the anomaly generator 411 should be used to detect potential anomalies from the time series data. In some implementations, the detector configuration generator 405 analyzes the metrics and determines the attributes (annotations) of the metric metadata for use by the anomaly generator 411. Examples of annotations include, but are not limited to: metric resource type (stream, database, etc.); metric type, such as restraint, request rate, latency, error rate, custom, etc.; metric characteristics, such as one or more seasonal cycles, stationarity, etc.; metric limit thresholds (such as Disk = {Medium = 70%, High = 80%}, ProvisionedLimit = {Medium = 150%, High = 175}); and metric domain knowledge (such as Alarm_Threshold-upper, Interest_Threshold_lower, Interest_Changepoint, Interest_missing_values, etc.). The detector configuration generator 405 uses this information (along with information about available detectors) to generate the configuration for the anomaly detector 411. The configuration may include one or more of the following: the metrics to be evaluated, one or more detectors to be used, how to configure the detectors, domain knowledge metadata of interest (e.g., thresholds, aggregation information), time-series data sources, and indications of aggregators in one or more aggregators 409 to be used to determine when an anomaly exists. The detector configuration generator 405 can also receive user feedback to adjust model sensitivity, what items are not of interest (which can then be used to retrain the detector or notify the aggregator to ignore observations), etc.

[0032] Anomaly generator 411 receives time-series data (and in some cases, indicator metadata) as input and generates anomaly indications as events (event start / end) and types, as output, according to its configuration. In some implementations, the anomaly indications include scores. Anomaly types may include, but are not limited to, change points, thresholds, and missing data. As shown, anomaly generator 411 is decomposed into multiple subsystems. Detector 407 is a machine learning model, a statistically based detector, and / or a rule-based detector that evaluates the time-series data to generate observations of potential anomalies.

[0033] These observations are received by one or more aggregators 409, which determine when sufficient observations have been generated to indicate the presence of an event that can be considered anomalous. Typically, one aggregator exists per model and / or metric. Aggregator 409 generates anomaly indicators that include one or more of event information (start, stop, detection), score, and type. Typically, the ratio of aggregators 409 to detectors is 1:N. In some embodiments, detectors 407 and aggregators 409 are combined.

[0034] Additionally, in some implementations, the anomaly generator 411 includes an update mechanism 415. For example, if the detector 407 is observing an abnormal number of events (too many or too few), the configuration of the anomaly generator 411 may need to be changed. Therefore, an update request will be generated and sent to the detector configuration generator 405, including information such as observation time and observation type. The update mechanism 415 can also alert the user to potential updates.

[0035] To help understand how an implementation of the anomaly detection service 110(A) or 110(B) can work, an exemplary latency use case is described. The detector configuration generator 405 uses metadata about metrics that are interested in what types of things are relevant to the latency, such as upper thresholds, lower thresholds, change points, missing values, etc. When the data ingestor 403 begins ingesting metrics, it is sent to the anomaly generator 411 for processing by the configured detector. The detector that discovers a potential anomaly is activated, generating an observation consisting of an event type and a start / end / detection time (if applicable). In some cases, a score is also generated. When the aggregator receives this observation, it determines whether it is sufficient to generate an anomaly to be presented to the user, etc.

[0036] Figure 5This is a flowchart illustrating the operation of a method for anomaly detection according to some embodiments. Some or all of the operations (or other processes described herein, or variations and / or combinations thereof) are executed under the control of one or more computer systems configured with executable instructions, and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that commonly executes on one or more processors, implemented by hardware, or implemented by a combination thereof. The code is stored on a computer-readable storage medium, for example, in the form of a computer program including instructions executable by one or more processors. The computer-readable storage medium is non-transitory. In some embodiments, one or more (or all) of the operations are performed by anomaly detection services 110(A) or 110(B) as shown in other figures.

[0037] At position 502, a request is received to detect anomalies in time series data using one or more machine learning models with the anomaly detection service. The request may include one or more of the following: indications of the time series data source to be evaluated, the location of the detected anomaly, indications of which users are allowed to provide feedback, initial configuration (such as sensitivity, indications of what is of interest or disinterest as an anomaly, thresholds, etc.), the types of anomalies to be detected, when anomalies will be presented, who will receive indications of anomalies or their presence, including the namespace of the metrics, the evaluation period, the statistics or percentiles to be used, the data points for alarms and / or anomaly indications, and instructions for handling missing data (considered acceptable, ignored, considered unacceptable, etc.).

[0038] Configure the anomaly detection service at 504. This configuration may include several different behaviors. In some implementations, at 506, the metrics browser participates in determining which metrics to add or remove for evaluation. For example, if the request includes an indication of what type of anomaly to detect, the metrics browser may determine what time series data will be used for that detection. Similarly, for existing configurations, the metrics browser may be used to determine which data to evaluate based on feedback (from the user or anomaly generator 411).

[0039] At point 508, the anomaly generator is configured with information about which detectors to use, based on one or more of the following: the request, time-series data, metadata about the domain of interest (e.g., thresholds, aggregation information, etc.), global usage knowledge, time-series data sources, and / or the aggregator to be used. Note that in some implementations, feedback is incorporated into this configuration generation. At point 510, the anomaly generator is configured using the generated configuration.

[0040] At 512, time-series data is evaluated for anomalies. At 514, time-series data is ingested. Ingestion may include formatting, decryption, etc. At 516, potential anomalous behaviors are observed using one or more configured detectors. At 518, potential anomalous behaviors are aggregated with other observed potential anomalous behaviors. At 520, an anomaly indication is generated when the aggregated potential anomalous behaviors prove such an indication. For example, when a specific number of change points exist within a set time period (according to configuration), when a threshold has been exceeded a set number of times, etc. It should be noted that in some implementations, sufficiently severe anomalies (e.g., significantly exceeding a threshold, etc.) are generated regardless of the number of anomalies.

[0041] In some implementations, at point 522, the anomaly indication is annotated. This annotation provides insight into the cause of the anomaly.

[0042] At point 524, provide the user with an anomaly indication. This can appear in the dashboard, via messaging, or other means. Note that the user does not need to be the requester. The anomaly indication includes one or more of the following: event information (start / stop / detection time), score, and anomaly type.

[0043] In some implementations, at point 526, user feedback is received and merged. For example, user feedback regarding exceptions that are not actually exceptions is received and used to reconfigure the exception generator 411.

[0044] Figure 6 An exemplary provider network (or “service provider system”) environment according to some implementations is illustrated. Provider network 600 may provide resource virtualization to customers via one or more virtualization services 610, which allow customers to purchase, lease, or otherwise obtain instances 612 of virtualized resources (including, but not limited to, compute and storage resources) implemented on devices within one or more provider networks in one or more data centers. A local Internet Protocol (IP) address 616 may be associated with resource instance 612; the local IP address is the internal network address of resource instance 612 on provider network 600. In some implementations, provider network 600 may also provide public IP addresses 614 and / or ranges of public IP addresses (e.g., Internet Protocol version 4 (IPv4) or Internet Protocol version 6 (IPv6) addresses) available to customers from provider 600.

[0045] Typically, provider network 600 may, via virtualization service 610, allow service provider customers (e.g., customers operating one or more client networks 650A to 650C including one or more customer devices 652) to dynamically associate at least some public IP addresses 614 assigned to or allocated to the customer with specific resource instances 612 assigned to the customer. Provider network 600 may also allow customers to remap public IP addresses 614 previously mapped to one virtualized computing resource instance 612 assigned to the customer to another virtualized computing resource instance 612 also assigned to the customer. For example, customers of a service provider (such as an operator of one or more customer networks 650A to 650C) may use virtualized computing resource instances 612 and public IP addresses 614 provided by the service provider to implement customer-specific applications and present those applications on an intermediate network 640 such as the Internet. Then, other network entities 620 on intermediate network 640 can generate traffic to a destination public IP address 614 published by one or more customer networks 650A to 650C; this traffic is routed to the service provider's data center and, at the data center, routed via the network layer to the local IP address 616 of the virtualized computing resource instance 612, which is currently mapped to the destination public IP address 614. Similarly, response traffic from the virtualized computing resource instance 612 can be routed back to the source entity 620 on intermediate network 640 via the network layer.

[0046] As used herein, "local IP address" refers to an internal or "private" network address, such as that of a resource instance within a provider network. Local IP addresses may be within the address block reserved by the Internet Engineering Task Force (IETF) Request for Comments (RFC) 1918 and / or have the address format specified by IETF RFC 4193, and may be variable within a provider network. Network traffic originating outside the provider network is not directly routed to local IP addresses; instead, traffic uses public IP addresses that map to the local IP address of the resource instance. Provider networks may include network devices or equipment that provide Network Address Translation (NAT) or similar functionality to perform mappings from public IP addresses to local IP addresses, and vice versa.

[0047] A public IP address is a variable network address on the Internet assigned to a resource instance by a service provider or by a customer. Traffic routed to a public IP address is translated via 1:1 NAT and forwarded to the corresponding local IP address of the resource instance.

[0048] Some public IP addresses may be assigned to specific resource instances by the provider's network infrastructure; these public IP addresses may be referred to as standard public IP addresses, or simply standard IP addresses. In some implementations, the mapping of standard IP addresses to the local IP addresses of resource instances is the default startup configuration for all resource instance types.

[0049] At least some public IP addresses may be assigned to or obtained by customers of Provider Network 600; customers can then assign their assigned public IP addresses to specific resource instances assigned to them. These public IP addresses may be referred to as customer public IP addresses, or simply customer IP addresses. Instead of being assigned to resource instances by Provider Network 600 as in the case of standard IP addresses, customer IP addresses may be assigned to resource instances by the customer, for example, via an API provided by the service provider. Unlike standard IP addresses, customer IP addresses are assigned to customer accounts and may be remapped to other resource instances by the respective customer as needed or desired. Customer IP addresses are associated with customer accounts, not with specific resource instances, and the customer controls the IP address until the customer chooses to release it. Unlike regular static IP addresses, customer IP addresses allow customers to mask resource instance or availability zone failures by remapping their public IP addresses to any resource instance associated with their customer account. For example, customer IP addresses enable customers to resolve customer resource instance or software issues by remapping their customer IP addresses to alternative resource instances.

[0050] Figure 7 This is a block diagram of an exemplary provider network providing storage services and hardware virtualization services to customers according to some implementation schemes. Hardware virtualization service 720 provides customers with multiple computing resources 724 (e.g., computing instances 725 such as VMs). For example, computing resources 724 may be leased or rented to customers of provider network 700 (e.g., customers implementing customer network 750). Each computing resource 724 may have one or more local IP addresses. Provider network 700 may be configured to route packets from the local IP addresses of computing resources 724 to public internet destinations, and from public internet sources to the local IP addresses of computing resources 724.

[0051] Provider network 700 may provide, for example, a client network 750 coupled to intermediate network 740 via local network 756 to implement virtual computing systems 792 via hardware virtualization service 720 coupled to intermediate network 740 and provider network 700. In some embodiments, hardware virtualization service 720 may provide one or more APIs 702 (e.g., web service interfaces) via which client network 750 may access functionality provided by hardware virtualization service 720, for example, via console 794 (e.g., web-based applications, standalone applications, mobile applications, etc.). In some embodiments, each virtual computing system 792 at provider network 700 and client network 750 may correspond to computing resources 724 that are leased, rented, or otherwise provided to client network 750.

[0052] Customers can access the functionality of storage service 710 from an instance of virtual computing system 792 and / or another customer device 790 (e.g., via console 794) via one or more APIs 702, to access data from and store data in storage resources 718A to 718N of virtual data storage areas 716 (e.g., folders or "buckets," virtualized volumes, databases, etc.) provided by provider network 700. In some embodiments, a virtualized data storage gateway (not shown) may be provided at customer network 750, which may locally cache at least some data (e.g., frequently accessed or critical data) and may communicate with storage service 710 via one or more communication channels to upload new or modified data from the local cache, thereby maintaining the primary storage area (virtualized data storage area 716) for data maintenance. In some embodiments, users can install and access virtual data storage area 716 volumes via storage service 710, which acts as a storage virtualization service, via virtual computing system 792 and / or on another customer device 790, and these volumes may appear to the user as local (virtualized) storage devices 798.

[0053] Although Figure 7 Not shown, but one or more virtualization services can also be accessed from resource instances within provider network 700 via one or more APIs 702. For example, a customer, equipment service provider, or other entity can access virtualization services from within a corresponding virtual network on provider network 700 via API 702 to request the allocation of one or more resource instances within the virtual network or another virtual network.

[0054] Explanatory System

[0055] In some implementations, a system that implements some or all of the techniques described herein may include a general-purpose computer system (such as...) Figure 8The illustrated computer system 800 includes or is configured to access one or more computer-accessible media. In the illustrated embodiment, the computer system 800 includes one or more processors 810 coupled to system memory 820 via an input / output (I / O) interface 830. The computer system 800 also includes a network interface 840 coupled to the I / O interface 830. Although Figure 8 The computer system 800 is shown as a single computing device, but in various embodiments, the computer system 800 may include a single computing device or any number of computing devices configured to work together as a single computer system 800.

[0056] In various embodiments, computer system 800 may be a single-processor system including one processor 810 or a multiprocessor system including several processors 810 (e.g., two, four, eight, or another suitable number). Processor 810 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 810 may be a general-purpose or embedded processor implementing any of a variety of instruction set architectures (ISAs), such as x86, ARM, PowerPC, SPARC, or MIPS ISA or any other suitable ISA. In a multiprocessor system, each of processors 810 may typically (but not necessarily) implement the same ISA.

[0057] System memory 820 may store instructions and data accessible by one or more processors 810. In various embodiments, system memory 820 may be implemented using any suitable memory technology, such as random access memory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory. In the illustrated embodiment, program instructions and data (such as the methods, techniques, and data described above) implementing one or more desired functions are shown stored in system memory 820 as anomaly detection service code 825 (e.g., executable to fully or partially implement anomaly detection service 110) and data 826.

[0058] In one embodiment, I / O interface 830 may be configured to coordinate I / O traffic between processor 810, system memory 820, and any peripheral devices (including network interface 840 or other peripheral interfaces) within a device. In some embodiments, I / O interface 830 may perform any necessary protocol, timing, or other data transformations to convert data signals from one component (e.g., system memory 820) into a format suitable for use by another component (e.g., processor 810). In some embodiments, I / O interface 830 may include devices that support attachment via various types of peripheral buses (e.g., variations of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard). In some embodiments, the functionality of I / O interface 830 may be split into two or more separate components, such as a northbridge and a southbridge. Furthermore, in some embodiments, some or all of the functionality of I / O interface 830 (such as an interface to system memory 820) may be directly incorporated into processor 810.

[0059] Network interface 840 can be configured to allow communication between computer system 800 and other devices 860 attached to one or more networks 850 (e.g., such as...). Figure 1 The network interface 840 can exchange data with other computer systems or devices shown. In various implementations, for example, the network interface 840 may support communication via any suitable wired or wireless general-purpose data network (e.g., Ethernet network type). Additionally, the network interface 840 may support communication via telecommunications / telephone networks (such as analog voice networks or digital fiber optic communication networks), via storage area networks (SANs) (such as Fibre Channel SANs), or via any other suitable type of network and / or protocol for I / O.

[0060] In some embodiments, the computer system 800 includes one or more offload cards 870A or 870B (including one or more processors 875 and may include one or more network interfaces 840), which are connected using I / O interfaces 830 (e.g., a version of the Peripheral Component Interconnect Fast (PCI-E) standard or another interconnect bus such as Fast Path Interconnect (QPI) or Hyper Path Interconnect (UPI). For example, in some embodiments, the computer system 800 may act as a host electronic device hosting computing resources (such as compute instances) (e.g., operating as part of a hardware virtualization service), and the one or more offload cards 870A or 870B act as a virtualization manager that manages the compute instances running on the host electronic device. As an example, in some embodiments, the one or more offload cards 870A or 870B may perform compute instance management operations, such as pausing and / or unpausing compute instances, starting and / or terminating compute instances, performing memory transfer / copy operations, etc. In some implementations, these management operations may be performed by one or more offload cards 870A or 870B in cooperation with a hypervisor (e.g., upon request from the hypervisor) executed by other processors 810A to 810N of the computer system 800. However, in some implementations, the virtualization manager implemented by one or more offload cards 870A or 870B may accommodate requests from other entities (e.g., from the computing instance itself) and may not cooperate with (or serve) any single hypervisor.

[0061] In some embodiments, system memory 820 may be an embodiment of a computer-accessible medium configured to store program instructions and data as described above. However, in other embodiments, program instructions and / or data may be received, transmitted, or stored on different types of computer-accessible media. Generally, computer-accessible media may include non-transitory storage media or memory media, such as magnetic or optical media, for example, a disk or DVD / CD coupled to computer system 800 via I / O interface 830. Non-transitory computer-accessible storage media may also include any volatile or non-volatile media, such as RAM (e.g., SDRAM, Double Data Rate (DDR) SDRAM, SRAM, etc.), read-only memory (ROM), etc., which may be included as system memory 820 or another type of memory in some embodiments of computer system 800. Furthermore, computer-accessible media may include transmission media or signals, such as electrical signals, electromagnetic signals, or digital signals, conveyed via communication media (such as networks and / or wireless links, such as those implemented via network interface 840).

[0062] The various implementations discussed or proposed herein can be implemented in a wide variety of operating environments, in some cases of which may include one or more user computers, computing devices, or processing devices that can be used to operate any of a number of applications. User or client devices may include: any of many general-purpose personal computers, such as desktop or laptop computers running standard operating systems; and cellular, wireless, and handheld devices running mobile software and capable of supporting many networking and messaging protocols. Such systems may also include a number of workstations running a variety of commercially available operating systems and any of other known applications for purposes such as development and database management. These devices may also include other electronic devices, such as virtual terminals, thin clients, gaming systems, and / or other devices capable of communicating via a network.

[0063] Most implementations utilize at least one network familiar to those skilled in the art to support communication using any of a variety of widely available protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), File Transfer Protocol (FTP), Universal Plug and Play (UPnP), Network File System (NFS), Public Internet File System (CIFS), Extensible Messaging and Presence Protocol (XMPP), AppleTalk, etc. The one or more networks may include, for example, a Local Area Network (LAN), a Wide Area Network (WAN), a Virtual Private Network (VPN), the Internet, an intranet, an extranet, a Public Switched Telephone Network (PSTN), an infrared network, a wireless network, and any combination thereof.

[0064] In implementations utilizing a web server, the web server can run any of a variety of server or middleware applications, including HTTP servers, File Transfer Protocol (FTP) servers, Common Gateway Interface (CGI) servers, data servers, Java servers, traffic application servers, etc. The one or more servers may also be able to respond to requests from user devices (such as requests implemented in any programming language such as...). One or more web applications may execute programs or scripts written in C, C#, or C++, or any scripting language such as Perl, Python, PHP, or TCL, or combinations thereof. One or more servers may also include database servers, including but not limited to commercially available database servers from Oracle(R), Microsoft(R), Sybase(R), IBM(R), etc. Database servers may be relational or non-relational (e.g., "NoSQL"), distributed or non-distributed, etc.

[0065] The environment disclosed herein may include a variety of data storage areas and other memories and storage media as discussed above. These may reside in a variety of locations, such as on storage media local to (and / or residing in) one or more computers, or on storage media located remotely from any or all of the computers across a network. In a particular set of embodiments, information may reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, any necessary files for performing functions belonging to a computer, server, or other network device may be stored locally or remotely, depending on the circumstances. Where the system includes computerized devices, each such device may include hardware elements electrically coupled via a bus, including, for example, at least one central processing unit (“CPU”), at least one input device (e.g., mouse, keyboard, controller, touchscreen, or keypad) and / or at least one output device (e.g., display device, printer, or speaker). Such a system may also include one or more storage devices, such as hard disk drives, optical storage devices, and solid-state storage devices such as random access memory (RAM) or read-only memory (ROM), as well as removable media devices, memory cards, flash memory cards, etc.

[0066] Such devices may also include computer-readable storage medium readers, communication devices (e.g., modems, network interface cards (wireless or wired), infrared communication devices, etc.), and working memory, as described above. A computer-readable storage medium reader may be connected to or configured to receive computer-readable storage media, which represents remote, local, fixed, and / or removable storage devices, as well as storage media for temporarily and / or more permanently containing, storing, transmitting, and retrieving computer-readable information. Systems and various devices will also typically include multiple software applications, modules, services, or other elements residing within at least one working memory device, including operating systems and applications such as client applications or web browsers. It should be understood that alternative embodiments may have numerous variations different from those described above. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software (including portable software such as applets), or both. Furthermore, connectivity to other computing devices, such as network input / output devices, may be employed.

[0067] Storage media and computer-readable media used to contain code or portions of code may include any suitable media known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing and / or transmitting information (such as computer-readable instructions, data structures, program modules or other data), including RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc memory (CD-ROM), digital versatile optical disc (DVD) or other optical storage devices, magnetic cartridges, magnetic tapes, disk storage devices or other magnetic storage devices, or any other media that can be used to store desired information and is accessible by system devices. Based on the disclosure and teachings provided herein, those skilled in the art will understand other ways and / or methods for implementing the various embodiments.

[0068] In the above description, various implementation schemes are described. Specific configurations and details are set forth for illustrative purposes to provide a thorough understanding of the implementation schemes. However, it will be apparent to those skilled in the art that the described implementation schemes can be practiced without specific details. Furthermore, well-known features may have been omitted or simplified to avoid obscuring the described implementation schemes.

[0069] In this document, bracketed text and boxes with dashed borders (e.g., large dashes, small dashes, dot dashes, and dots) are used to illustrate optional operations for adding additional features to some embodiments. However, this notation should not be construed as implying that these are the only options or optional operations, and / or that in some embodiments, boxes with solid borders are not optional.

[0070] In various embodiments, reference numerals with suffix letters may be used to indicate one or more instances of the referenced entity, and when multiple instances exist, each instance need not be identical, but may instead share some general characteristics or act according to convention. Furthermore, unless explicitly stated to the contrary, the use of a particular suffix does not imply the existence of a specific number of entities. Therefore, in various embodiments, two entities using the same or different suffix letters may or may not have the same number of instances.

[0071] References to "an implementation," "implementation," "exemplary implementation," etc., indicate that the described implementation may include a particular feature, structure, or characteristic, but each implementation need not necessarily include that particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same implementation. Additionally, when a particular feature, structure, or characteristic is described in connection with an implementation, it should be understood that, whether explicitly described or not, implementing such a feature, structure, or characteristic in conjunction with other implementations is also within the knowledge of those skilled in the art.

[0072] Furthermore, in the various embodiments described above, unless otherwise specifically indicated, disjunctive language such as the phrase “at least one of A, B, or C” is intended to mean A, B, or C, or any combination thereof (e.g., A, B, and / or C). Thus, disjunctive language is generally not intended, nor should it be construed, as implying that a given embodiment requires at least one A, at least one B, or at least one C to each be present.

[0073] Unless otherwise expressly stated, articles such as “a” or “an” should generally be understood to include one or more of the described items. Therefore, phrases such as “device configured to…” are intended to include one or more of the stated devices. Such one or more of the stated devices may also be collectively configured to perform the stated narrative. For example, “processor configured to perform narratives A, B, and C” may include a first processor configured to perform narrative A, which works in conjunction with a second processor configured to perform narratives B and C.

[0074] At least some examples of the disclosed technology can be described in accordance with the following terms:

[0075] 1. A computer-implemented method, comprising:

[0076] Receive requests to use one or more machine learning models to detect anomalies in time series data using the anomaly detection service;

[0077] Configure the anomaly detection service as follows:

[0078] Determine the time series metrics to be evaluated.

[0079] The configuration for the anomaly detection service is generated based on one or more of the following: the request, the time-series data, the type of anomaly to be detected, and domain knowledge metadata, wherein the configuration identifies at least one specific machine learning model among the one or more machine learning models, and

[0080] Use the generated configuration to configure the anomaly detection service;

[0081] The configured anomaly detection service can be used to evaluate the time series data for anomalies in the following ways:

[0082] Ingest the time series data,

[0083] Observe potential anomalous behavior using at least one specific machine learning model identified from the one or more machine learning models.

[0084] Aggregate the observed potential anomalous behaviors with other observed potential anomalous behaviors, and

[0085] When the observed potential anomalous behavior of the aggregate quantity exceeds a threshold, an anomaly indication is generated; and

[0086] Provide the user with the exception.

[0087] 2. The computer-implemented method as described in Clause 1, wherein the type of anomaly to be detected is one of a threshold-based anomaly, a missing data anomaly, and a change-point anomaly.

[0088] 3. The computer-implemented method as described in any one of Clauses 1-2, wherein the time-series data is ingested from a metrics service of a provider network.

[0089] 4. A computer-implemented method, comprising:

[0090] Receive requests to use one or more detectors to detect anomalies in time series data using the anomaly detection service;

[0091] Configure the anomaly detection service as follows:

[0092] The configuration for the anomaly detection service is generated based at least in part on one or more of the following: the request, the time-series data, and metadata, wherein the configuration identifies at least one specific detector among the one or more detectors, and

[0093] Use the generated configuration to configure the anomaly detection service;

[0094] The configured anomaly detection service can be used to evaluate the time series data for anomalies in the following ways:

[0095] Ingest the time series data,

[0096] Observe potential anomalous behavior using at least one specific detector identified from the one or more detectors, and

[0097] Generate exception indications; and

[0098] Provide the user with the aforementioned error indication.

[0099] 5. The computer-implemented method as described in Clause 4, wherein the time-series data is ingested from a metrics service of a provider network.

[0100] 6. The computer-implemented method as described in any one of Clauses 4-5, wherein the type of anomaly to be detected is one of threshold-based anomalies, missing data anomalies, and change-point anomalies.

[0101] 7. A computer-implemented method as described in any one of Clauses 4-6, wherein at least one particular detector identified in one or more detectors is a machine learning model, a rule-based detector, or a statistics-based detector.

[0102] 8. The computer-implemented method as described in any one of Clauses 4-7, further comprising:

[0103] Aggregate the observed potential anomalous behaviors with other observed potential anomalous behaviors, and

[0104] Anomaly indications are only generated when the potential anomalous behavior observed by the aggregate exceeds a threshold.

[0105] 9. The computer-implemented method as described in any one of Clauses 4-8, further comprising:

[0106] The exception is annotated to provide insights into the reasons for its generation.

[0107] 10. The computer-implemented method as described in any one of clauses 4-9, further comprising:

[0108] An alert is issued in response to the aforementioned anomaly.

[0109] 11. A computer-implemented method as described in any one of Clauses 4-10, wherein the request includes one or more of the following: an indication of the time series data source to be evaluated, the location of the detected anomaly, an indication of which users are allowed to provide feedback, an initial configuration, an indication of the type of anomaly to be detected, an indication of when to present the anomaly to the user, an indication of who should receive the anomaly indication or an indication of the existence of the anomaly, a namespace including indicators, an indication of the evaluation period, statistical data or percentiles to be used during the evaluation, data points to be alarmed, and an indication of the handling of missing data.

[0110] 12. The computer-implemented method as described in any one of clauses 4-11, further comprising:

[0111] Configure the anomaly detection service as follows:

[0112] Determine the time series metrics to be evaluated.

[0113] The configuration for the anomaly detection service is generated based on one or more of the following: the request, the time-series data, the type of anomaly to be detected, and domain knowledge metadata, wherein the configuration identifies at least one specific detector among the one or more detectors, and

[0114] Use the generated configuration to configure the anomaly detection service.

[0115] 13. The computer-implemented method as described in any one of Clauses 4-12, further comprising:

[0116] Receive feedback and reconfigure the anomaly detection service based on the received feedback.

[0117] 14. The computer-implemented method as described in Clause 13, wherein the anomaly indication includes at least one of the following: a score indicating the degree of difference between the potential anomalous behavior and the expected behavior, event information including start and stop times, and the type of the detected anomaly.

[0118] 15. A system comprising:

[0119] One or more first electronic devices, the one or more first electronic devices being used to provide time-series data for analysis of anomalies;

[0120] One or more second electronic devices, the one or more second electronic devices being configured to implement an anomaly detection service in a multi-tenant provider network, the anomaly detection service including instructions that, when executed, cause the anomaly detection service to:

[0121] Receive requests to use one or more detectors to detect anomalies in time series data using the anomaly detection service;

[0122] Configure the anomaly detection service as follows:

[0123] The configuration for the anomaly detection service is generated based at least in part on one or more of the following: the request, the time-series data, and metadata, wherein the configuration identifies at least one specific detector among the one or more detectors, and

[0124] Use the generated configuration to configure the anomaly detection service;

[0125] The configured anomaly detection service can be used to evaluate the time series data for anomalies in the following ways:

[0126] Ingest the time series data,

[0127] Observe potential anomalous behavior using at least one specific detector identified from the one or more detectors, and

[0128] Generate an exception indication; and

[0129] Provide the user with the aforementioned error indication.

[0130] 16. The system as described in Clause 15, wherein the time series data is ingested from a metrics service of a provider network.

[0131] 17. The system of any one of Clauses 15-16, wherein the type of anomaly to be detected is one of threshold-based anomalies, missing data anomalies, and change-point anomalies.

[0132] 18. The system of any one of Clauses 15-17, wherein at least one particular detector identified in the one or more detectors is a machine learning model, a rule-based detector, or a statistics-based detector.

[0133] 19. The system as described in any one of Clauses 15-18, wherein the anomaly detection service further issues an alert for the anomaly.

[0134] 20. The system as described in any one of Clauses 15-19, wherein the request includes one or more of the following: an indication of the time series data source to be evaluated, the location of the detected anomaly, an indication of which users are allowed to provide feedback, initial configuration, an indication of the type of anomaly to be detected, an indication of when to present the anomaly to the user, an indication of who should receive the anomaly indication or an indication of the existence of the anomaly, including a namespace for the metrics, an indication of the evaluation period, statistical data or percentiles to be used during the evaluation, data points to be alarmed, and an indication of the handling of missing data.

[0135] Therefore, the specification and drawings are to be considered illustrative rather than restrictive. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of this disclosure as set forth in the appended claims.

Claims

1. A computer-implemented method, comprising: Receive requests to use one or more detectors to detect anomalies in time series data using the anomaly detection service; Configure the anomaly detection service as follows: The configuration for the anomaly detection service is generated based at least in part on one or more of the following: the request, the time-series data, the type of anomaly to be detected, and domain knowledge metadata, wherein the configuration identifies at least one specific detector among the one or more detectors, and Use the generated configuration to configure the anomaly detection service; The configured anomaly detection service can be used to evaluate the time series data for anomalies in the following ways: Ingest the time series data, Observe potential anomalous behavior using at least one specific detector identified from the one or more detectors, and Generate an exception indication; as well as Provide the user with the aforementioned error indication.

2. The computer-implemented method of claim 1, wherein the time-series data is ingested from a metrics service of a provider network.

3. The computer-implemented method as described in claim 1 or 2, wherein the type of anomaly to be detected is one of a threshold-based anomaly, a missing data anomaly, and a change-point anomaly.

4. The computer-implemented method of claim 1 or 2, wherein the identified at least one particular detector among the one or more detectors is a machine learning model, a rule-based detector, or a statistics-based detector.

5. The computer-implemented method as described in claim 1 or 2, further comprising: Aggregate the observed potential anomalous behaviors with other observed potential anomalous behaviors, and Anomaly indications are only generated when the potential anomalous behavior observed by the aggregate exceeds a threshold.

6. The computer-implemented method as described in claim 1 or 2, further comprising: The exception is annotated to provide insights into the reasons for its generation.

7. The computer-implemented method as described in claim 1 or 2, further comprising: An alert is issued in response to the aforementioned anomaly.

8. The computer-implemented method of claim 1 or 2, wherein the request includes one or more of the following: an indication of the time series data source to be evaluated, the location of the detected anomaly, an indication of which users are allowed to provide feedback, an initial configuration, an indication of the type of anomaly to be detected, an indication of when to present the anomaly to the user, an indication of who should receive the anomaly indication or an indication of the existence of the anomaly, a namespace including indicators, an indication of the evaluation period, statistical data or percentiles to be used during the evaluation, data points to be alerted, and an indication of the handling of missing data.

9. The computer-implemented method as described in claim 1 or 2, further comprising: Receive feedback and reconfigure the anomaly detection service based on the received feedback.

10. The computer-implemented method of claim 9, wherein the anomaly indication includes at least one of the following: a score indicating the degree of difference between the potential anomalous behavior and the expected behavior, event information including start and stop times, and the type of the detected anomaly.

11. A computer system comprising: One or more first electronic devices, the one or more first electronic devices being used to provide time-series data for analysis of anomalies; One or more second electronic devices, the one or more second electronic devices being configured to implement an anomaly detection service in a multi-tenant provider network, the anomaly detection service including instructions that, when executed, cause the anomaly detection service to: Receive requests to use one or more detectors to detect anomalies in time series data using the anomaly detection service; Configure the anomaly detection service as follows: The configuration for the anomaly detection service is generated based at least in part on one or more of the following: the request, the time-series data, the type of anomaly to be detected, and domain knowledge metadata, wherein the configuration identifies at least one specific detector among the one or more detectors, and Use the generated configuration to configure the anomaly detection service; The configured anomaly detection service can be used to evaluate the time series data for anomalies in the following ways: Ingest the time series data, Observe potential anomalous behavior using at least one specific detector identified from the one or more detectors, and Generate an exception indication; and Provide the user with the aforementioned error indication.

12. The computer system of claim 11, wherein the type of anomaly to be detected is one of a threshold-based anomaly, a missing data anomaly, and a change-point anomaly.

13. The computer system of claim 11 or 12, wherein at least one of the identified detectors in the one or more detectors is one of a machine learning model, a rule-based detector, or a statistics-based detector.

14. The computer system of claim 11 or 12, wherein the request includes one or more of the following: an indication of the time series data source to be evaluated, the location of the detected anomaly, an indication of which users are allowed to provide feedback, an initial configuration, an indication of the type of anomaly to be detected, an indication of when to present the anomaly to the user, an indication of who should receive the anomaly indication or an indication of the existence of the anomaly, a namespace including indicators, an indication of the evaluation period, statistical data or percentiles to be used during the evaluation, data points to be alerted, and an indication of the handling of missing data.

Images