Method, system, and computer readable storage medium for generating predictions about a network

Abstract

Embodiments of the present application relate to machine learning pipelines for predictions about a network. The present disclosure describes techniques that include using automatically trained machine learning systems to generate predictions. In one example, the present disclosure describes a method that includes, based on a request for a prediction: training each respective machine learning (ML) model of a plurality of ML models to generate a respective training phase prediction of a plurality of training phase predictions; automatically determining a selected ML model of the plurality of ML models based on evaluation metrics for the plurality of MLs; and applying the selected ML model to generate the prediction based on data collected from a network that includes a plurality of network devices.

Description

Technical Field

[0001] This disclosure relates to the analysis of computer networks. Background Technology

[0002] Virtualized data centers are becoming a core foundation of modern information technology (IT) infrastructure. In particular, modern data centers have made extensive use of virtualization environments, in which virtual machines, such as virtual machines or containers, are deployed on the underlying computing platform of physical computing devices and run on it.

[0003] Virtualization within large data centers offers several advantages, including efficient utilization of computing resources and simplified network configuration. Therefore, enterprise IT staff often prefer virtualized computing clusters in data centers not only for the efficiency and higher return on investment (ROI) they provide, but also for their management advantages. However, virtualization can present some challenges when analyzing, assessing, and / or troubleshooting network malfunctions. Summary of the Invention

[0004] This disclosure describes techniques including collecting data about a network and using the collected data to train and utilize machine learning (ML) models to generate predictions about the network. Predictions can provide insights into one or more aspects of the network, such as predicted business level, CPU utilization, business anomalies, etc. As described herein, a network analysis system for a network can train each of a predetermined plurality of ML models to generate corresponding training-stage predictions in a plurality of training-stage predictions based on data. The network analysis system can determine the selected ML model from the predetermined plurality of ML models based on evaluation metrics. Furthermore, the network analysis system can apply the selected ML model to generate predictions based on data. The techniques described herein can provide one or more technical advantages. For example, the techniques described herein can reduce the complexity of network analysis systems and simplify the process for obtaining predictions.

[0005] In some examples, the present invention describes operations performed by a network analysis system or other network system according to one or more aspects of this disclosure. In one particular example, the present disclosure describes a method comprising, based on a request for prediction: training each of the predetermined plurality of machine learning (ML) models by a computing system to generate a corresponding training-stage prediction among a plurality of training-stage predictions; automatically determining a selected ML model among the plurality of ML models by the computing system based on evaluation metrics for the plurality of ML models; and applying the selected ML model by the computing system to generate a prediction based on data collected from a network comprising a plurality of network devices.

[0006] In another example, this disclosure describes a system comprising: a data repository configured to store data collected from a network including a plurality of network devices; and processing circuitry configured to: based on a request for prediction, train each of a predetermined plurality of machine learning (ML) models to generate a corresponding training-stage prediction among a plurality of training-stage predictions; automatically determine a selected ML model from the predetermined plurality of ML models based on an evaluation metric for the predetermined plurality of ML models; and apply the selected ML model to generate a prediction based on the data collected from the network including the plurality of network devices.

[0007] In another example, this disclosure describes a non-transitory computer-readable storage medium including instructions that, when executed, configure the processing circuitry of a computing system to perform operations including: training each of a predetermined plurality of machine learning (ML) models to generate a corresponding training-stage prediction among a plurality of training-stage predictions based on a request for a prediction; automatically determining a selected ML model among the predetermined plurality of ML models based on an evaluation metric for the predetermined plurality of ML models; and applying the selected ML model to generate a prediction based on data collected from a network including a plurality of network devices. Attached Figure Description

[0008] Figure 1A This is a conceptual diagram illustrating an example network of a machine learning (ML) system for analyzing networks, according to one or more aspects of this disclosure;

[0009] Figure 1B This is a conceptual diagram illustrating example components of a system for analyzing networks according to one or more aspects of this disclosure;

[0010] Figure 2 This is a block diagram illustrating an example network for analyzing networks according to one or more aspects of this disclosure;

[0011] Figure 3 This is a conceptual diagram illustrating example components of a machine learning (ML) system according to one or more aspects of this disclosure;

[0012] Figure 4 This is a conceptual diagram illustrating an example training workflow according to one or more aspects of this disclosure;

[0013] Figure 5 This is a conceptual diagram illustrating an example production workflow according to one or more aspects of this disclosure;

[0014] Figure 6 It is a conceptual diagram illustrating an example model object according to one or more aspects of this disclosure;

[0015] Figure 7 This is a conceptual diagram illustrating example component objects according to one or more aspects of this disclosure; and

[0016] Figure 8 This is a flowchart illustrating the operation of a network analysis system according to one or more aspects of this disclosure. Detailed Implementation

[0017] Data centers utilizing virtualized environments (where virtual machines or containers are deployed on and run on the underlying computing platform of physical computing devices) can offer efficiency, cost, and organizational advantages. However, gaining meaningful insights into application, node, and link workloads can still be crucial for managing any data center architecture. Collecting business samples from network devices can help provide such insights. In the various examples described in this paper, business samples are collected and then processed by analytical algorithms to generate various types of forecasts. Forecasts can include data indicating the utilization of specific communication links or devices within the network, the volume of specific flows within the network, etc. Forecasts can relate to the future or current state of the network.

[0018] In addition, network administrators may want to obtain predictions about the network. For example, a network administrator might want to know how much data will be transmitted through a specific communication link within the network at a specific time or period in the future. In another example, a network administrator might want to know how much data will be transmitted through a specific virtual network at a specific time or period in the future. In yet another example, a network administrator might want to know whether a specific node or set of nodes (or link or set of links) will exceed a resource utilization threshold.

[0019] You might want to use machine learning (ML) models to generate such predictions. ML models can be trained to generate predictions based on data collected from the network, such as streaming data. Different ML models may be needed to generate different types of predictions. Furthermore, different types of ML models may be better suited to generating specific types of predictions than others. Therefore, network administrators may configure the network's ML system to use ML models on an ad-hoc basis to obtain different predictions. Adding ML models ad-hoc can increase the complexity and storage requirements of the ML system. For example, it may be necessary to install various software packages on the ML system to support various ML models. The number and size of these packages may be unknown in advance. In addition, the lack of uniformity in the ML systems used to train the ML models can increase the complexity of the ML system. Moreover, from the perspective of network administrators or clients, it may be difficult to determine in advance which type of ML model is best suited to generate specific types of network insights. Determining the type of ML model and training it to generate specific network insights can be a challenging and time-consuming process.

[0020] The technology disclosed herein can address one or more of the challenges described above. For example, according to examples of this disclosure, a computing system, such as a flow controller, can collect data for a network with multiple network devices. For instance, the computing system can collect streaming data from the network. Streaming data may include underlying streaming data, overlay streaming data, and / or other types of streaming data. The computing system can store the data in a database. Furthermore, based on a prediction request received by the computing system, the ML system can train each of a predetermined plurality of ML models to generate corresponding training-stage predictions in a plurality of training-stage predictions. The network analysis system can automatically determine the selected ML model among the plurality of ML models based on evaluation metrics for the plurality of ML models. Additionally, the network analysis system can apply the selected ML model to generate predictions based on data collected from the network.

[0021] By training multiple ML models from a pre-defined set of ML models, the complexity of a network analysis system can be reduced, for example, because it may not be necessary to add ML models on an ad-hoc basis. Furthermore, by automatically selecting ML models, administrators can more easily choose the appropriate ML model to generate predictions. In some examples, a network analysis system may be able to automatically train ML models, select ML models, and provide predictions based solely on a request for a prediction.

[0022] Figure 1A This is a conceptual diagram illustrating an example network including an ML system 138 for analyzing networks, according to one or more aspects of this disclosure. Figure 1A An example implementation of network system 100 and data center 101 is shown, which host one or more computing networks, computing domains or projects, and / or cloud-based computing networks (generally referred to herein as cloud computing clusters). Cloud-based computing clusters can coexist in a common overall computing environment, such as a single data center, or they can be distributed across an environment, such as between different data centers. Cloud-based computing clusters can be, for example, different cloud environments, such as OpenStack cloud environments, Kubernetes cloud environments, or various combinations of other computing clusters, domains, networks, etc. In other cases, other implementations of network system 100 and data center 101 may be suitable. Such implementations may include... Figure 1A The examples include a subset of components, and / or may include Figure 1A Additional components not shown.

[0023] exist Figure 1A In the example, data center 101 provides an operating environment for applications and services to customers 104 coupled to data center 101 through service provider network 106. Figure 1AIt can be shown as a distribution in Figure 1A Among multiple devices in the example, in other cases, it is attributed to Figure 1A The features and technologies of one or more devices may be executed internally by native components of one or more such devices. Similarly, one or more such devices may include certain components and execute various technologies described herein as attributable to one or more other devices. Furthermore, certain operations, technologies, features, and / or functions may be combined. Figure 1A The description may be made or performed by a specific component, device, and / or module. In other examples, such operations, techniques, features, and / or functions may be performed by other components, devices, or modules. Therefore, some operations, techniques, features, and / or functions attributable to one or more components, devices, or modules may also be attributable to other components, devices, and / or modules, even if not specifically described in this way herein.

[0024] Data center 101 hosts infrastructure equipment such as network and storage systems, redundant power supplies, and environmental controls. Service provider network 106 can be coupled to one or more networks managed by other providers, and thus can form part of a large-scale public network infrastructure such as the Internet.

[0025] In some examples, data center 101 can represent one of many geographically distributed network data centers. For example... Figure 1A As shown in the example, data center 101 is a facility that provides network services to client 104. Client 104 can be a collective entity, such as a business, government, or individual. For example, a network data center can host network services for several businesses and end users. Other exemplary services may include data storage, virtual private networks, traffic engineering, file services, data mining, scientific or supercomputing, etc. In some examples, data center 101 is a standalone network server, network peer, etc.

[0026] exist Figure 1AIn the example, data center 101 includes a set of storage systems, application servers, compute nodes, or other devices, including network devices 110A to 110N (collectively referred to as "network devices 110," meaning any number of network devices). Devices 110 may be interconnected via a high-speed switching fabric 121 provided by one or more layers of physical network switches and routers. In some examples, devices 110 may be included within fabric 121, but are shown separately for ease of illustration. Network devices 110 can be any of many different types of network devices (core switches, spine network devices, branch network devices, edge network devices, or other network devices), but in some examples, one or more devices 110 may serve as physical compute nodes in the data center. For example, one or more of devices 110 may provide an operating environment for executing one or more customer-specific virtual machines or other virtualized instances (such as containers). In such an example, one or more of devices 110 may alternatively be referred to as host compute devices or more simply as hosts. Network devices 110 can thus execute one or more virtualized instances, such as virtual machines, containers, or other virtual execution environments for running one or more services (such as Virtual Network Functions (VNFs)).

[0027] Typically, each network device 110 can be any type of device that can operate on a network and generate data accessible by telemetry or other means (e.g., streaming data, such as sFlow data), and can include any type of computing device, sensor, camera, node, monitoring device, or other device. Furthermore, some or all of the network devices 110 can represent components of another device that can generate data that can be collected by telemetry or other means. For example, some or all of the network devices 110 can represent physical or virtual network devices such as switches, routers, hubs, gateways, and security devices (such as firewalls, intrusion detection and / or intrusion prevention devices).

[0028] Although not specifically shown, switching fabric 121 may include top-of-rack (TOR) switches of the distribution layer coupled to chassis switches, and data center 101 may include one or more non-edge switches, routers, hubs, gateways, security devices (such as firewalls, intrusion detection and / or intrusion prevention devices), servers, computer terminals, laptops, printers, databases, wireless mobile devices (such as cellular phones or personal digital assistants), wireless access points, bridges, cable modems, application accelerators, or other network devices. Switching fabric 121 may perform Layer 3 routing to route network traffic between data center 101 and customer 104 via service provider network 106. Gateway 108 is used to forward and receive packets between switching fabric 121 and service provider network 106.

[0029] According to one or more examples of this disclosure, a software-defined networking (“SDN”) controller 132 provides a logical, and in some cases physically, centralized controller to facilitate the operation of one or more virtual networks within a data center 101. In some examples, the SDN controller 132 operates in response to configuration input received from an orchestration engine 130 via a northbound API 131, which in turn may operate in response to configuration input received from an administrator 128 who interacts with and / or operates the user interface device 129.

[0030] User interface device 129 can be implemented as any suitable device for presenting output and / or accepting user input. For example, user interface device 129 may include a display. User interface device 129 may be a computing system, such as a mobile or non-mobile computing device operated by a user and / or administrator 128. According to one or more aspects of this disclosure, user interface device 129 may, for example, represent a workstation, laptop or notebook computer, desktop computer, tablet computer, or any other computing device that can be operated by a user and / or present a user interface. In some examples, user interface device 129 may be physically separate from controller 201 and / or located in a different location from controller 201. In such examples, user interface device 129 may communicate with controller 201 via a network or other communication means. In other examples, user interface device 129 may be a local peripheral device of controller 201 or may be integrated into controller 201.

[0031] In some examples, orchestration engine 130 manages the functionality of data center 101, such as compute, storage, networking, and application resources. For instance, orchestration engine 130 can create virtual networks for tenants within or across data centers. Orchestration engine 130 can attach virtual machines (VMs) to tenants' virtual networks. Orchestration engine 130 can connect tenants' virtual networks to external networks, such as the Internet or a VPN. Orchestration engine 130 can implement security policies across a set of VMs or at the boundary of a tenant's network. Orchestration engine 130 can deploy network services (e.g., load balancers) within tenants' virtual networks.

[0032] In some examples, SDN controller 132 manages network and networking services such as load balancing and security, and can allocate resources from device 110, used as a host device, to various applications via southbound API 133. That is, southbound API 133 represents a set of communication protocols used by SDN controller 132 to ensure that the actual state of the network equals the desired state specified by orchestration engine 130. For example, SDN controller 132 can fulfill high-level requests from orchestration engine 130 by configuring physical switches, such as TOR switches, chassis switches, and switching fabric 121; physical routers; physical service nodes, such as firewalls and load balancers; and virtual services in VMs, such as virtual firewalls. SDN controller 132 maintains routing, network, and configuration information within a state database.

[0033] Policy controller 140 interacts with one or more of devices 110 (and / or other devices) to collect data across data center 101 and / or network system 100. The collected data may include streaming data. Such streaming data may include underlying streaming data and overlay streaming data. In some examples, underlying streaming data may be collected by sampling streaming data collected at Layer 2 of the OSI model. Overlay streaming data may be data derived from overlay services established across one or more virtual networks within network system 100 (e.g., data samples). Overlay streaming data may, for example, include information identifying the source and destination virtual networks. Other types of collected data may include data relating to the utilization of computing resources (such as processors, memory, storage space, link bandwidth, etc.), packet throughput information of physical or virtual machines, power consumption information, etc.

[0034] According to one or more aspects of this disclosure, Figure 1A The policy controller 140 can configure each of the devices 110 to collect data. For example, in reference to Figure 1A In the described example, policy controller 140 outputs a signal to each device 110. Each device 110 receives the signal and interprets it as a command to collect data. Subsequently, each device 110 transmits data to policy controller 140. Policy controller 140 receives the data, prepares it for use in response to analysis queries, and stores the data. Figure 1A In the example, other network devices (including network devices within switching structure 121 (not specifically shown)) may also be configured to collect data, such as underlying streaming data, overlay streaming data, or other types of data.

[0035] Policy controller 140 can receive requests for information about network system 100. For example, in the described example, user interface device 129 detects input and outputs information about that input to policy controller 140. Policy controller 140 determines that the information corresponds to a request for information about network system 100, such as a request from a user of user interface device 129 or from another source. Policy controller 140 can use ML system 138 to process the request. Policy controller 140 can generate a response to the request and output information about that response to user interface device 129.

[0036] exist Figure 1A In some examples, the ML system 138 is implemented in a separate computing system from the policy controller 140. For instance, in one example, the ML system 138 may be implemented in a computing system at the residence of a user served by data center 101, and the policy controller 140 may be implemented in a cloud-based computing system. In other examples, the ML system 138 and the policy controller 140 may be implemented on the same computing system. Furthermore, in some examples, the ML system 138 may be part of the policy controller 140.

[0037] According to one or more techniques of the present invention, in response to a request for prediction, ML system 138 can train each of a predetermined plurality of ML models to generate a corresponding training-stage prediction among a plurality of training-stage predictions. Additionally, ML system 138 can automatically determine a selected ML model from the predetermined plurality of ML models based on evaluation metrics for the predetermined plurality of ML models. ML system 138 can apply the selected ML model to generate a prediction based on data collected from a network (e.g., network system 100) including multiple network devices (e.g., device 110). In some examples, ML system 138 can provide the prediction to policy controller 140. Policy controller 140 can process the prediction to generate a response to the request.

[0038] Figure 1B This is a conceptual diagram illustrating example components of a system for analyzing networks according to one or more aspects of this disclosure. Figure 1B Including combination Figure 1A The description contains many of the same elements. Figure 1B The elements shown can correspond to in Figure 1A The figures are identified by the same reference numerals. Figure 1A The elements shown. Typically, elements with similar numbering can be combined according to... Figure 1AThe provided elements are implemented in a manner consistent with the description of the corresponding elements, although in some examples such elements may involve alternative implementations with more, fewer, and / or different capabilities and attributes.

[0039] Figure 1B Example components of policy controller 140 are shown. Policy controller 140 is shown as including load balancer 141, collector(s) modules 142, queues and event repository 143, topology and metric source 144, data repository 145, and API 146. This disclosure may also refer to data repository 145 as a time-series database and API 146 as a decision endpoint. Typically, policy controller 140 and its components are designed and / or configured to ensure high availability and the ability to handle large volumes of data, such as streaming data. In some examples, multiple instances of components of policy controller 140 may be orchestrated (e.g., via orchestration engine 130) to execute on different physical servers to ensure that no component of policy controller 140 has a single point of failure. In some examples, policy controller 140 or its components may scale independently and horizontally to enable efficient and / or effective processing of desired traffic volumes, such as streaming data.

[0040] and Figure 1A The same as in the middle, Figure 1B The policy controller 140 configures each device 110 to collect data, such as streaming data or other types of data. For example, the policy controller 140 may output a signal to each device 110 to configure each device 110 to collect data. One or more of the devices 110 may then collect data and report such data to the policy controller 140.

[0041] exist Figure 1B In this configuration, the load balancer 141 of the policy controller 140 receives data from the device 110. The load balancer 141 can distribute the received data across multiple collector modules 142 to ensure the collector modules' active / active failover strategy. In some examples, multiple load balancers 141 may be required to ensure high availability and scalability.

[0042] Multiple collector modules 142 collect data from load balancer 141. Multiple collector modules 142 send the data upstream to queues and event repositories 143. In some examples where the data includes streaming data, multiple collector modules 142 can address, process, and / or accommodate unified data from sFlows, NetFlow v9, IPFIX, jFlow, Contrail Flow, and other formats. Multiple collector modules 142 can be able to parse internal headers from sFlow packets and other data stream packets. Multiple collector modules 142 can be able to handle message overflows, enriching stream records with topology information (e.g., AppFormix topology information). Multiple collector modules 142 can also be able to convert data to binary format before writing or sending it to queues and event repositories 143. The underlying streaming data of the "sFlow" type (which refers to "sampled stream") is a standard for packet derivation at Layer 2 of the OSI model. sFlow provides a method for deriving truncated packets and interface counters for network monitoring purposes.

[0043] The queue and event repository 143 processes the collected data. For example, the queue and event repository 143 may receive data from one or more collector modules 142, store the data, and make the data available for ingestion in the data repository 145. In some examples, this allows for the separation of the task of receiving and storing large amounts of data from the tasks of indexing and preparing the data for analytical queries. In some examples, the queue and event repository 143 may also enable independent users to directly consume the data. In some examples, the queue and event repository 143 can be used to detect anomalies and generate alerts in real time. In some examples where the data includes streaming data, the streaming data can be parsed by reading encapsulated packets, including VXLAN, UDP-based MPLS, and GRE-based MPLS. For example, the queue and event repository 143 parses the source IP, destination IP, source port, destination port, and protocol from the internal (underlying) packets. Some types of streaming data (including sFlow data) only include a portion of the sampled network traffic (e.g., the first 128 bytes), so in some cases, the streaming data may not include all internal fields. In such examples, such data can be marked as missing.

[0044] Topology and metrics source 144 can enrich or augment the data using topology information and / or metrics information. For example, topology and metrics source 144 can provide network topology metadata, which may include identified nodes or network devices, configuration information, configurations, established links, and other information about such nodes and / or network devices. In some examples, topology and metrics source 144 may use AppFormix topology data, or it may be an executing AppFormix module. Information received from topology and metrics source 144 can be used to enrich the data collected by collector(s) ...

[0045] Data repository 145 can be configured to store data received from queues and event repositories 143, as well as topology and metric sources 144, in an indexed format, enabling fast aggregation queries and fast random access data retrieval. In some examples, data repository 145 can achieve fault tolerance and high availability through data sharding and replication.

[0046] API 146 can handle requests sent by one or more user interface devices 129. For example, in some examples, API 146 can receive requests from user interface device 129 via HTTP POST requests. In some examples, API 146 can implement a representative state transfer (REST) ​​API.

[0047] According to one or more techniques disclosed herein, API 146 may receive requests for forecasts. Forecasts may include one or more predictions about the future condition of network system 100, may include data about the current or past condition of network system 100, or may include other types of data derived at least in part from data collected by collector(s) ...

[0048] Based on a request, ML system 138 can train each of a plurality of ML models to generate corresponding training-stage predictions in a plurality of training-stage predictions (e.g., based on data stored in data repository 145 or provided training data). ML system 138 can automatically determine the selected ML model among the plurality of ML models based on evaluation metrics used for the plurality of ML models. After selecting an ML model, ML system 138 can apply the selected ML model to generate predictions based on the data stored in data repository 145.

[0049] API 146 can return a response to UI device 129 based on a prediction. In some examples, API 146 can process the prediction to determine whether to generate an alert. For example, API 146 can determine whether to generate an alert based on a comparison of one or more values ​​in the prediction with corresponding thresholds. API 146 can provide the alert to UI device 129, for example, in the form of a pop-up notification, email message, instant message, text or graphical message in the user interface, or in other forms. Example types of alerts can include network traffic anomaly alerts. In some examples, orchestration engine 130, SDN controller 132, or other systems can perform automated decision-making in response to the prediction.

[0050] By implementing the techniques of this disclosure, the process of generating predictions can be formalized. For example, instead of generating a set of ad hoc ML models on a case-by-case basis, ML system 138 can select ML models from a predefined set of ML models. By using a predefined set of ML models, the storage requirements of ML system 138 can be simplified because it may not be necessary to store, support, and maintain the various software packages required to support ad hoc individual ML models. In some examples, ML system 138 implements standardized training and production workflows for requests. The workflows are described in more detail elsewhere in this disclosure. Using such a standardized training workflow simplifies the software infrastructure that would otherwise be required to support the use of Ad hoc ML models. Simplifying the software infrastructure reduces the data storage requirements of ML system 138. The techniques of this disclosure also make it easier to generate predictions for users. Creating ad hoc models for each new case requires significant research and understanding to create accurate models for predictions. The techniques of this disclosure can train and select accurate ML models for predictions using a variety of major, widely known algorithms, which can save time and effort for predicting any unknown metrics.

[0051] Figure 2 This is a block diagram illustrating an example network for analyzing networks according to one or more aspects of this disclosure. Figure 2 The network system 200 can be described as Figure 1A or Figure 1B Example or alternative implementation of network system 100. Figure 2 One or more aspects of it can be described in this paper in the context of Figure 1.

[0052] Despite such as Figure 1A , Figure 1B and Figure 2The data center shown can be operated by any entity, but some data centers can be operated by a service provider whose business model involves providing computing power to its customers. Therefore, a data center typically contains a large number of compute nodes or host devices. To operate efficiently, these hosts have the ability to connect to each other and to the outside world, and this capability is provided through physical network devices that can be interconnected in a leaf-spine topology. The collection of these physical devices (e.g., network devices and hosts) forms the underlying network.

[0053] Each host device in such a data center typically runs multiple virtual machines, known as workloads. Clients in the data center usually have access to these workloads and can use them to install applications and perform other operations. Workloads running on different host devices but accessible to a specific client are organized into a virtual network. Each client typically has at least one virtual network. These virtual networks are also known as overlay networks. In some cases, clients in the data center may encounter connectivity issues between two applications running on different workloads. Troubleshooting such issues can often be complex due to the deployment of workloads in large, multi-tenant data centers.

[0054] exist Figure 2 In the example, network 205 connects policy controller 140, host device 210A, host device 210B, and host device 210N. Policy controller 140 may correspond to... Figure 1A and Figure 1B The illustrated policy controller 140 is an example or alternative implementation. Host devices 210A, 210B to 210N may be collectively referred to as "host device 210" to mean any number of host devices 210.

[0055] Each host device 210 can be Figure 1A and Figure 1B Example of device 110, but in Figure 2 In the example, instead of network devices, each host device 210 is implemented as a server or host device that serves as a computing node in the virtualized data center. Therefore, in Figure 2 In the example, each host device 210 executes multiple virtual computing instances, such as virtual machines 228.

[0056] Also connected to network 205 is user interface device 129, such as Figure 1A and Figure 1BAs shown, user interface device 129 can be operated by administrator 128. In some examples, user interface device 129 can present one or more user interfaces on a display device associated with user interface device 129, some of which may have a form similar to user interface 262.

[0057] Figure 2 The underlying stream data 204 and overlay stream data 206 flowing within network system 200 are also shown. Specifically, the underlying stream data 204 is shown leaving the spine device 202A and flowing towards the policy controller 140. Similarly, the overlay stream data 206 is shown leaving the host device 210A and flowing over network 205. In some examples, as described herein, the overlay stream data 206 is transmitted to the policy controller 140 via network 205. For simplicity, Figure 2 A single instance of underlying flow data 204 and a single instance of overlay flow data 206 are shown. However, it should be understood that each of the spine device 202 and leaf device 203 can generate underlying flow data 204 and transmit it to policy controller 140, and in some examples, each host device 210 (and / or other device) can generate underlying flow data 204 and transmit such data to policy controller 140 via network 205. Furthermore, it should be understood that each host device 210 (and / or other device) can generate overlay flow data 206 and transmit such data to policy controller 140 via network 205.

[0058] Network 205 can correspond to Figure 1A and Figure 1B The network 205 may correspond to any one of the switching structure 121 and / or the service provider network 106, or alternatively, a combination of the switching structure 121, the service provider network 106, and / or another network. Figure 1A and Figure 1B Some components include gateway 108, SDN controller 132, and orchestration engine 130.

[0059] Network 205 shows spine devices 202A and 202B (collectively referred to as "spine devices 202," meaning any number of spine devices 202), and leaf devices 203A, 203B, and 203C (collectively referred to as "leaf devices 203," and also meaning any number of leaf devices 203). Although network 205 is shown as having spine devices 202 and leaf devices 203, other types of network devices may be included in network 205, including core switches, edge network devices, top-of-rack devices, and other network devices.

[0060] Typically, network 205 can be the Internet, or it may include or represent any public or private communications network or other network. For example, network 205 can be cellular, ZigBee, Bluetooth, Near Field Communication (NFC), satellite, enterprise, service provider, and / or other types of networks capable of data transfer between computing systems, servers, and computing devices. One or more client devices, server devices, or other devices may use any suitable communication technology to transmit and receive data, commands, control signals, and / or other information on network 205. Network 205 may include one or more network hubs, network switches, network routers, satellite antennas, or any other network devices. Such devices or components may be operatively coupled to each other to provide information exchange between computers, devices, or other components (e.g., between one or more client devices or systems and one or more server devices or systems). Figure 2 Each device or system shown can be operatively coupled to network 205 using one or more network links. The link that couples such a device or system to network 205 can be Ethernet, Asynchronous Transfer Mode (ATM), or other types of network connection, and such connection can be wireless and / or wired. Figure 2 One or more of the devices or systems on the network 205 shown may be in a remote location relative to one or more other devices or systems shown.

[0061] Policy controller 140 can be implemented as any suitable computing system, such as one or more server computers, workstations, mainframes, devices, cloud computing systems, and / or other computing systems capable of performing the operations and / or functions described in one or more aspects of this disclosure. In some examples, policy controller 140 represents a cloud computing system, server farm, and / or server cluster (or a portion thereof) that provides services to client devices and other devices or systems. In other examples, policy controller 140 may represent, or be implemented through, one or more virtualized computing instances (e.g., virtual machines, containers) of a data center, cloud computing system, server farm, and / or server cluster.

[0062] exist Figure 2In some examples, policy controller 140 may include power supply 241, one or more processors 243, one or more communication units 245, one or more input devices 246, and one or more output devices 247. Storage device 250 may include one or more collector modules 252, user interface module 254, API 146, data repository 259, and ML-related database 260. In some examples, ML-related database 260 is separate from policy controller 140. For example, ML-related database 260 may be included in ML system 138. Furthermore, in some examples, ML-related database 260 may be a metadata repository.

[0063] One or more of the devices, modules, storage areas, or other components of the policy controller 140 can be interconnected to enable inter-component communication (physical, communicative, and / or operational). In some examples, such connectivity may be provided via a communication channel (e.g., communication channel 242), a system bus, a network connection, an inter-process communication data structure, or any other method for transmitting data.

[0064] Power supply 241 can provide power to one or more components of policy controller 140. Power supply 241 may receive power from a main alternating current (AC) power source in a data center, building, home, or other location. In other examples, power supply 241 may be a battery or a device that supplies direct current (DC). In yet another example, policy controller 140 and / or power supply 241 may receive power from another power source. One or more of the devices or components shown within policy controller 140 may be connected to power supply 241 and / or may receive power from power supply 241. Power supply 241 may have intelligent power management or consumption capabilities, and such features may be controlled, accessed, or regulated by one or more modules of policy controller 140 and / or by one or more processors 243 to intelligently consume, distribute, supply, or otherwise manage power.

[0065] One or more processors 243 of the policy controller 140 may implement functions associated with the policy controller 140 or with one or more modules shown and / or described herein and / or execute instructions associated with the policy controller 140 or with one or more modules shown and / or described herein. The one or more processors 243 may be processing circuitry that performs operations according to one or more aspects of this disclosure, may be part of such processing circuitry, and / or may include such processing circuitry. Examples of processors 243 include microprocessors, application processors, display controllers, auxiliary processors, one or more sensor hubs, and any other hardware configured to function as a processor, processing unit, or processing device. The policy controller 140 may use one or more processors 243 to perform operations according to one or more aspects of this disclosure using software, hardware, firmware, or a mixture of hardware, software, and firmware residing at and / or executing at the policy controller 140.

[0066] One or more communication units 245 of the policy controller 140 can communicate with devices external to the policy controller 140 by transmitting and / or receiving data, and can in some respects function as input and output devices. In some examples, the communication unit 245 can communicate with other devices via a network. In other examples, the communication unit 245 can transmit and / or receive radio signals on a radio network such as a cellular radio network. Examples of communication units 245 include network interface cards (e.g., such as Ethernet cards), optical transceivers, radio frequency transceivers, GPS receivers, or any other type of device capable of transmitting and / or receiving information. Other examples of communication units 245 may include those capable of transmitting and / or receiving data via… GPS, NFC, ZigBee, and cellular networks (e.g., 3G, 4G, 5G), as well as those found in mobile devices and Universal Serial Bus (USB) controllers, etc. A device for wireless communication. Such communication may follow, implement, or comply with appropriate protocols, including Transmission Control Protocol / Internet Protocol (TCP / IP), Ethernet, Bluetooth, NFC, or other technologies or protocols.

[0067] One or more input devices 246 may represent any input device of the policy controller 140 not otherwise described herein. One or more input devices 246 may generate, receive, and / or process input from any type of device capable of detecting input from a person or machine. For example, one or more input devices 246 may generate, receive, and / or process input in the form of electrical, physical, audio, image, and / or visual input (e.g., peripherals, keyboards, microphones, cameras, etc.).

[0068] One or more output devices 247 may represent any output device of the policy controller 140 not otherwise described herein. One or more output devices 247 may generate, receive, and / or process input from any type of device capable of detecting input from a person or machine. For example, one or more output devices 247 may generate, receive, and / or process outputs in the form of electrical and / or physical outputs (e.g., peripheral devices, actuators, etc.).

[0069] One or more storage devices 250 in the policy controller 140 may store information for processing during operation of the policy controller 140. Storage devices 250 may store program instructions and / or data associated with one or more modules described according to one or more aspects of this disclosure. One or more processors 243 and one or more storage devices 250 may provide an operating environment or platform for such modules, which may be implemented as software, but in some examples may include any combination of hardware, firmware, and software. One or more processors 243 may execute instructions, and one or more storage devices 250 may store instructions and / or data of one or more modules. The combination of processors 243 and storage devices 250 may retrieve, store, and / or execute instructions and / or data of one or more applications, modules, or software. Processors 243 and / or storage devices 250 may also be operatively coupled to one or more other software and / or hardware components, including but not limited to one or more components of the policy controller 140 and / or one or more devices or systems shown as connected to the policy controller 140.

[0070] In some examples, one or more storage devices 250 are implemented as temporary memory, which may indicate that the primary purpose of one or more storage devices is not long-term storage. The storage device 250 of the policy controller 140 can be configured to act as volatile memory for short-term storage of information, and therefore the stored content is not retained if it is deactivated. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art. In some examples, the storage device 250 also includes one or more computer-readable storage media. The storage device 250 can be configured to store a larger amount of information compared to volatile memory. The storage device 250 can also be configured to act as non-volatile storage space for long-term storage of information, and retain information after an activation / deactivation cycle. Examples of non-volatile memory include magnetic hard disks, optical disks, flash memory, or electrically programmable memory (EPROM) or electrically erasable programmable memory (EEPROM).

[0071] Multiple collector modules 252 can perform functions related to receiving both the underlying streaming data 204 and the overlay streaming data 206, and perform load balancing as needed to ensure high availability, throughput, and scalability for collecting such streaming data. Collector modules 252 can process the data to prepare it for storage in data repository 259. In some examples, multiple collector modules 252 can store the data in data repository 259. Data repository 259 can be equivalent to data repository 145. Figure 1B ).

[0072] User interface module 254 can perform functions related to generating user interfaces for presenting the results of analytical queries performed by API 146. In some examples, user interface module 254 can generate enough information to generate a set of user interfaces and cause communication unit 215 to output such information on network 205 for use by user interface device 129 to present one or more user interfaces at a display device associated with user interface device 129.

[0073] API 146 can execute analytical queries involving data stored in data repository 259, which is derived from a collection of underlying streaming data 204, overlay streaming data 206, and / or other types of data. In some examples, API 146 can receive requests in the form of information derived from an HTTP POST request, and in response, ML system 138 can be used to obtain predictions in response to the request. In some examples, the request includes a query or another data structure specifying the data to be included in the prediction. Furthermore, in some examples, API 146 can retrieve topology information related to device 110 and perform analyses, including data deduplication, overlay underlying correlation, business path identification, and heatmap business calculations.

[0074] Data repository 259 can represent any suitable data structure or storage medium used to store information related to data stream information, including the storage of data derived from underlying stream data 204 and overlay stream data 206. Data repository 259 can be responsible for storing data in an indexed format to enable fast data retrieval and query execution. The information stored in data repository 259 can be searchable and / or categorized, allowing one or more modules within policy controller 140 to provide input requesting information from data repository 259 and, in response to that input, receive information stored within data repository 259. Data repository 259 can be primarily maintained by collector modules(s) 252. Data repository 259 can be implemented using multiple hardware devices and can achieve fault tolerance and high availability through data sharding and replication. In some examples, data repository 259 can be implemented using the open-source ClickHouse column-oriented database management system.

[0075] Each host device 210 represents a physical computing device or compute node that provides an execution environment for virtual hosts, virtual machines, containers, and / or other virtualized computing resources. In some examples, each host device 210 may be a component of a cloud computing system, server farm, and / or server cluster (or a portion thereof) that provides services to client devices and other devices or systems.

[0076] Certain aspects of host device 210 are described herein with respect to host device 210A. Other host devices 210 (e.g., host devices 210B to 210N) may be described similarly and may also include the same, similar, or corresponding components, devices, modules, functions, and / or other features. Therefore, the description of host device 210A herein may be applied accordingly to one or more other host devices 210 (e.g., host devices 210B to 210N).

[0077] exist Figure 2In the example, host device 210A includes underlying physical computing hardware, including power supply 211, one or more processors 213, one or more communication units 215, one or more input devices 216, one or more output devices 217, and one or more storage devices 220. Storage device 220 may include hypervisor 221, which includes kernel module 222, virtual router module 224, and agent module 226. Virtual machines 228A to 228N (collectively referred to as "virtual machines 228" and representing any number of virtual machines 228) execute on or are controlled by hypervisor 221. Similarly, virtual router agent 229 may execute on or under the control of hypervisor 221. One or more devices, modules, storage areas, or other components of host device 210 may be interconnected to enable inter-component communication (physical, communicative, and / or operational). In some examples, such connectivity may be provided via a communication channel (e.g., communication channel 212), system bus, network connection, inter-process communication data structure, or any other method for transmitting data.

[0078] Power supply 211 can provide power to one or more components of host device 210. Processor 213 can implement functions associated with host device 210 and / or execute instructions associated with host device 210. Communication unit 215 can communicate with other devices or systems on behalf of host device 210. One or more input devices 216 and output devices 217 can represent any other input and / or output devices associated with host device 210. Storage device 220 can store information for processing during operation of host device 210A. Each such component can be implemented in a manner similar to that described herein in conjunction with policy controller 140, etc.

[0079] Hypervisor 221 can be used as a module or system for instantiating, creating, and / or executing one or more virtual machines 228 on the underlying host hardware device. In some cases, hypervisor 221 may be referred to as a virtual machine manager (VMM). Hypervisor 221 can execute in an execution environment provided by storage device 220 and processor 213 or on top of an operating system kernel (e.g., kernel module 222). In some examples, hypervisor 221 is an operating system-level component that executes on a hardware platform (e.g., host 210) to provide a virtualization operating environment and orchestration controller for virtual machines 228 and / or other types of virtual computing instances. In other examples, hypervisor 221 may be a software and / or firmware layer that provides a lightweight kernel and operates to provide a virtualization operating environment and orchestration controller for virtual machines 228 and / or other types of virtual computing instances. Hypervisor 221 may incorporate the functionality of kernel module 222 (e.g., as a "Type 1 hypervisor"), such as Figure 2 As shown. In other examples, hypervisor 221 may execute on the kernel (e.g., as a "type 2 hypervisor").

[0080] Virtual router module 224 can execute multiple routing instances for the corresponding virtual network in data center 101 and can route packets to the appropriate virtual machine executing in the operating environment provided by device 110. Virtual router module 224 can also be responsible for collecting overlay flow data, such as Contrail Flow data used when using an infrastructure employing Contrail SDN. Therefore, each host device 210 can include a virtual router. For example, packets received by virtual router module 224 of host device 210A from the underlying physical network structure can include an external header to allow the physical network structure to tunnel the payload or "internal packet" to the physical network address of the network interface of host device 210A. The external header can include not only the physical network address of the server's network interface but also a virtual network identifier, such as a VxLAN label or Multiprotocol Label Switching (MPLS) label identifying one of the virtual networks, and the corresponding routing instance executed by the virtual router. Internal packets include an internal header with a destination network address conforming to the virtual network addressing space of the virtual network identified by the virtual network identifier.

[0081] Agent module 226 may execute as part of hypervisor 221, or it may execute in kernel space or as part of kernel module 222. Agent module 226 may monitor some or all performance metrics associated with host device 210A and may implement policies received from policy controller 140. Agent module 226 may configure virtual router module 224 to transmit overlay stream data to policy controller 140.

[0082] Virtual machines 228A through 228N (collectively referred to as "virtual machines 228," meaning any number of virtual machines 228) can represent example instances of virtual machines 228. Host device 210A can partition the virtual and / or physical address space provided by storage device 220 into user space for running user processes. Host device 210A can also partition the virtual and / or physical address space provided by storage device 220 into kernel space, which is protected and may be inaccessible to user processes.

[0083] Typically, each virtual machine 228 can be any type of software application, and a virtual address can be assigned to each virtual machine 228 for use in a corresponding virtual network, where each virtual network can be a different virtual subnet provided by the virtual router module 224. Each virtual machine 228 can be assigned its own virtual Layer 3 (L3) IP address, for example, for sending and receiving communication, but the IP address of the physical server running the virtual machine on it is unknown. In this way, the "virtual address" is an address used for the application, which is different from the address used for the underlying physical computer system (e.g., ...). Figure 2 The logical address of host device 210A in the example.

[0084] Each virtual machine 228 may represent a tenant virtual machine that runs client applications, such as web servers, database servers, enterprise applications, or hosts virtualization services used to create service chains. In some cases, any one or more of the host device 210 or another computing device directly (i.e., not as virtual machines) hosts client applications. Although one or more aspects of this disclosure have been described with respect to virtual machines or virtual hosts, the techniques described herein with respect to one or more aspects of this disclosure for such virtual machines or virtual hosts can also be applied to containers, applications, processes, or other execution units (virtualized or non-virtualized) executing on the host device 210.

[0085] exist Figure 2In the example, virtual router agent 229 is included within host device 210A and can communicate with SDN controller 132 and virtual router module 224 to control the overlay of virtual networks and coordinate the routing of data packets within host device 210A. Typically, virtual router agent 229 communicates with SDN controller 132, which generates commands to control the routing of packets through data center 101. Virtual router agent 229 can execute in user space and acts as a proxy for control plane messages between virtual machine 228A and SDN controller 132. For example, virtual machine 228A can request to send a message via virtual router agent 229 using its virtual address, and virtual router agent 229 can then send the message and request to receive a response to the message for the virtual address of virtual machine 228A that initiated the first message. In some cases, virtual machine 228A can invoke procedures or function calls presented by the application programming interface of virtual router agent 229, and in such examples, virtual router agent 229 also handles message encapsulation, including addressing.

[0086] Policy controller 140 can store data such as underlying stream data 204 and overlay stream data 206 in data repository 259. For example, in Figure 2 In this process, multiple collector modules 252 output information to a data repository 259. The data repository 259 can store data in an indexed format, enabling fast aggregation queries and fast random access data retrieval. In some examples, the data repository 259 can achieve fault tolerance and high availability by sharding and replicating data across multiple storage devices, which can be located across multiple physical hosts.

[0087] Policy controller 140 can receive requests. For example, user interface device 129 detects input and outputs a signal derived from that input via network 205. Communication unit 215 of policy controller 140 detects the signal and outputs information about the signal to API 146. API 146 determines that the signal corresponds to a request from a user of user interface device 129, the request being for information about network system 200 within a given time window. For example, a user of user interface device 129 (e.g., administrator 128) might want to obtain a forecast of the expected level of network traffic sent by a specific host device 210 during a specific future time period. For example, if a specific host device hosts a video streaming service, given past traffic patterns on network 205, the user might want to know how much network traffic the specific host device might send on an upcoming Sunday evening.

[0088] In some examples, policy controller 140's API 146 receives requests from systems such as software systems or devices without human intervention. For example, an application or other type of program running on user interface device 129, the policy controller 140's device, or another device can automatically send requests. In such examples, policy controller 140 may receive the same requests on a periodic or event-driven basis. In some examples, API 146 may receive requests from orchestration engine 130 (e.g., for virtual machine or container analytics), SDN controller 132 (e.g., for switch infrastructure or connectivity analytics and management), or other systems.

[0089] API 146 can use ML system 138 to process requests for predictions. To process the request, ML system 138 can determine whether an ML model has already been selected for the request. For example, ML system 138 can store previously received requests, previously processed queries, or other types of data in an ML-related database 260. In this example, ML system 138 can compare the request with the database of previously received requests to determine whether an ML model has already been selected for the request.

[0090] If an ML model has already been selected for the request, ML system 138 can apply the selected ML model to generate predictions based on the data stored in data repository 259. In some examples, if ML system 138 has already selected an ML model for the request, ML system 138 can initiate a production workflow. In such examples, the production workflow is the process of generating predictions using the ML model previously selected for the request. This disclosure relates, for example, to... Figure 5 The production workflow is described in more detail elsewhere in this disclosure.

[0091] On the other hand, if the ML system 138 has not previously selected an ML model for the request, the ML system 138 initiates a training workflow. The training workflow is a process in which the ML system 138 trains multiple ML models to generate training-phase predictions based on data stored in the data repository 259. Since the collector modules 252 continue to collect data and continue to store data in the data repository 259, the data used by the multiple ML models to generate training-phase predictions during the training workflow may differ from the data used by the selected ML model to generate predictions during the production workflow. Furthermore, when executing the training workflow, the ML system 138 automatically determines the selected ML model from the predetermined multiple ML models based on evaluation metrics for those models. This disclosure relates, for example, to... Figure 4 The production workflow is described in more detail elsewhere in this disclosure.

[0092] API 146 can be configured to use ML system 138 to handle various types of requests. Example request types may include, but are not limited to, best model prediction requests, statistical model prediction requests, deep learning model prediction requests, and fine-tuning requests. When ML system 138 processes a best model prediction request, it can return a prediction generated by the ML model selected by ML system 138 as the best ML model for the request. The selected ML model for handling the best model prediction request may not be limited to a specific type of model, such as a statistical ML model or a deep learning ML model. When ML system 138 processes a statistical model request, it can return a prediction generated by a statistical ML model selected by ML system 138 from a variety of predefined types of statistical ML models. Similarly, when ML system 138 processes a deep learning model request, it can return a prediction generated by a deep learning ML model selected by ML system 138 from a variety of predefined types of deep learning ML models.

[0093] When ML system 138 processes a fine-tuning request, it can perform a fine-tuning process to improve the already trained ML model. In some examples, API 146 or ML system 138 automatically generates fine-tuning requests or initiates fine-tuning processes. For example, fine-tuning processes can be initiated automatically nightly, hourly, or based on another schedule or event-driven basis. Example fine-tuning processes are described in more detail elsewhere in this disclosure.

[0094] Policy controller 140 can cause a user interface 262 containing prediction-based data to be presented at user interface device 129. For example, API 146 can receive predictions from ML system 138 and output the predictions to user interface module 254. User interface module 254 can use the predictions from API 146 to generate data sufficient to create at least a portion of user interface 262. For example, user interface module 254 can generate a JavaScript Object Notation (JSON) object containing data sufficient to create at least a portion of user interface 262. User interface module 254 causes communication unit 245 to output a signal via network 205 or another network. User interface device 129 detects and processes the signal to generate user interface 262. User interface device 129 presents user interface 262 on a display associated with user interface device 129. Figure 2 The content of the user interface 262 shown in the example may differ.

[0095] The modules and systems shown in Figure 12 (e.g., virtual router module 224, agent module 226, collector(s) 252, user interface module 254, API 146, ML system 138) and / or other modules and systems shown or described elsewhere in this disclosure can perform operations described using software, hardware, firmware, or a mixture of hardware, software, and firmware residing in and / or executing on one or more computing devices. For example, a computing device or a group of computing devices may execute one or more such modules using multiple processors or multiple devices. A computing device may execute one or more such modules as a virtual machine executing on underlying hardware. One or more of these modules may execute as one or more services of an operating system or computing platform. One or more such modules may execute as one or more executable programs at the application layer of a computing platform. In other examples, the functionality provided by the modules may be implemented by dedicated hardware devices.

[0096] Although certain modules, data repositories, components, programs, executables, data items, functional units, and / or other items included in one or more storage devices may be shown separately, one or more of these items may be combined and operated as a single module, component, program, executable, data item, or functional unit. For example, one or more modules or data repositories may be combined or partially combined such that they operate as a single module or provide functionality. Furthermore, one or more modules may interact with each other and / or operate in combination such that, for example, one module serves as a service or extension of another module. Moreover, each module, data repository, component, program, executable, data item, functional unit, or other item shown within the storage device may include multiple components, subcomponents, modules, submodules, data repositories, and / or other components or modules or data repositories not shown.

[0097] Furthermore, each module, data repository, component, program, executable file, data item, functional unit, or other item shown within the storage device can be implemented in various ways. For example, each module, data repository, component, program, executable file, data item, functional unit, or other item shown within the storage device can be implemented as a downloadable or pre-installed application or "app". In other examples, each module, data repository, component, program, executable file, data item, functional unit, or other item shown within the storage device can be implemented as part of an operating system that executes on a computing device.

[0098] Figure 3 This is a conceptual diagram illustrating example components of an ML system 138 according to one or more aspects of this disclosure. Figure 3In one example, ML system 138 includes a data processing unit 300, a training workflow unit 302, a production workflow unit 304, and a model monitoring unit 306. In other examples, ML system 138 may include more, fewer, or different units. Figure 3 Each component can be implemented using software that runs on a computing environment such as one or more servers, processors, or virtualized execution units (e.g., containers or virtual machines).

[0099] Data processing unit 300 is configured to process data from data repository 259 for use in one or more ML models. For example, data processing unit 300 can perform dimensionality reduction on the data. For instance, the input data may include a matrix of size N×M, where N is the number of entries and M is the dimension of each entry. In this example, if the required input size for one or more ML models is K, assuming M>K, then data processing unit 300 can use a dimensionality reduction algorithm such as an autoencoder to reshape the input data from N×M to N×K.

[0100] In some examples, data processing unit 300 may perform data preprocessing. Examples of data preprocessing types may include filling in non-numeric (NaN) values, normalizing or scaling the data, noise removal (e.g., using an autoencoder or other tools), zero centering, min / max normalization, principal component analysis (PCA), or other types of processing to prepare the data for use by one or more ML models. Principal component analysis can be used to decorrelate data for linear regression. For example, PCA can be used to whiten data by setting the covariance matrix to an identity matrix. In some examples, data processing unit 300 may perform data augmentation to generate additional data based on data from data repository 259. For example, data processing unit 300 may use one or more generative adversarial networks (GANs) to perform data augmentation. In some examples, the data generated by data processing unit 300 is shareable (or otherwise reusable) between different ML models. Making data reusable between different ML models can reduce the storage requirements of ML system 138.

[0101] Training workflow unit 302 is configured to execute training an ML model to generate a training workflow that produces predictions for a specific request received by ML system 138. ML system 138 can utilize training workflow unit 302 during the initial setup of the ML model for the request. In some examples, ML system 138 can use training workflow unit 302 to perform a fine-tuning process that improves a selected ML model for a specific request. In some examples, training workflow unit 302 can implement a Kubernetes-based machine learning pipeline.

[0102] Training workflow unit 302 can train various types of ML models. For example, in some examples, training workflow unit 302 is configured to train a baseline ML model. The baseline ML model can be another type of ML model besides deep learning ML models and statistical ML models. The baseline ML model can be able to generate predictions based on a limited amount of data. For example, the baseline ML model can be able to generate predictions based on less than one hour of data (e.g., hourly predictions). Example types of baseline ML models can include exponentially weighted moving average (EWMA) models, hidden Markov models, etc.

[0103] In some examples, training workflow unit 302 is configured to train a statistical ML model. Example types of statistical models include the Holt-Winters model, the Autoregressive Integrated Moving Average (ARIMA) model, the Seasonal ARIMA model, the Vector Autoregressive (VAR) model, the Facebook PROPHET model, and others. In some examples, statistical ML models can be more useful than basic ML models when more data is available for forecasting. For example, when more than 24 hours of data is available, a statistical ML model can be used to generate hourly forecasts.

[0104] In some examples, training workflow unit 302 is configured to train a deep learning ML model. Deep learning ML models may require more data than basic ML models or statistical ML models, but can be capable of providing more complex types of predictions. Example types of deep learning ML models can include Long Short-Term Memory (LSTM) models, bidirectional LSTM models, recurrent neural networks, or other types of neural networks including multiple layers. In other examples, ML system 138 can use neural network models other than deep learning ML models.

[0105] ML models can be grouped into regression-based ML models, classification-based ML models, and unsupervised learning models. Each of these groups can have baseline, statistical, and deep learning ML. In some examples, for regression-based ML models, the training workflow unit 302 can use a Hodrick-Prescott filter to perform initial levels of ML model selection. Specifically, the Hodrick-Prescott filter decomposes time series data (y_t) into a trend component and a periodic component c_t: y_t = tou_t (trend) + c_t (period). Time series data is the data used by the ML model to generate predictions. By splitting the time series data into trend and periodic components, the training workflow unit 302 can determine whether the time series data has more periodic characteristics or more trend characteristics. If the training workflow unit 302 determines that the time series data has more periodic characteristics than trend characteristics, then the training workflow unit 302 can eliminate ML models that perform better on time series data with trend characteristics. Conversely, if the training workflow unit 302 determines that the time series data has more trend characteristics than periodic characteristics, then the training workflow unit 302 can eliminate ML models that perform better on time series data with periodic characteristics. For example, EWMA models and Holts-Winter models perform better on time series data with periodic characteristics. ARIMA models, VAR models, etc., may perform better on time series data with trend characteristics.

[0106] By performing this initial level of ML model selection, training workflow unit 302 can avoid training every regression-based ML model, potentially saving time and computational resources. In some examples, training workflow unit 302 can filter regression-based ML models based on the amount of data available. For example, if less than a threshold amount of data is available in data repository 259 (e.g., 24-48 hours), training workflow unit 302 can train only the baseline regression-based ML model. Otherwise, if more than the threshold amount of data is available in data repository 259, training workflow unit 302 can additionally or alternatively train other types of regression-based ML models, such as statistical models or low-capacity deep learning ML models.

[0107] Example types of regression-based baseline ML models can include Hidden Markov Models (HMMs) and seasonal trend decomposition methods. Example types of regression-based statistical ML models can include Error Trend Seasonal (ETS) models (including exponential smoothing models, trend method models, and ETS decomposition), EWMA models (including simple moving averages and EWMA), Holt Winters models, ARIMA models, SARIMA models, vector autoregressive models, seasonal trend autoregressive (STAR) models, and Facebook PROPHET models. Example types of regression-based deep learning ML models can include LSTM architectures (including single-layer LSTM, deep LSTM, and bidirectional LSTM), RNNs, and gated recurrent units (GRUs). Example types of classification-based baseline ML models can include logistic regression models and K-nearest neighbor models. Example types of classification-based statistical ML models can include support vector machines and boosting ensemble algorithms (e.g., XGBoost). Example types of classification-based deep learning ML models can include LSTM architectures, RNN architectures, GRU architectures, and artificial neural network architectures. Example types of unsupervised ML models can include K-means clustering models, Gaussian clustering models, and density-based spatial clustering.

[0108] In some examples, training workflow unit 302 initiates a training workflow as part of the initial setup of the ML pipeline. An ML pipeline is the process of generating predictions for requests. Therefore, when the ML system 138 receives a new request, the ML system 138 executes the initial setup process of the ML pipeline to generate predictions for the new request.

[0109] In some examples, training workflow unit 302 begins training the workflow in response to a new instance of a previously received request. For instance, certain predictions for certain types of requests can be optimally generated using a basic ML model or a statistical ML model. For example, predicting a request could include predicting the traffic volume on communication links in network 205 for the next hour based on traffic volume on communication links over the past hour. In this example, a basic ML model might be sufficient to generate the prediction. Therefore, in this example, in response to a new instance of a request, training workflow unit 302 could retrain one or more basic ML models on data from the past hour to generate predictions for the next hour. Similarly, training workflow unit 302 could, for example, retrain one or more statistical ML models in response to daily / nightly instances of the same request.

[0110] In some examples, training workflow unit 302 can determine a subset of available ML models for training. In other words, training workflow unit 302 can be configured to train X ML models, but it can also determine a subset of Y ML models, where Y is less than X. Training workflow unit 302 can train each corresponding ML model in the requested subset of available ML models for each request, but not train the remaining ML models not in the requested subset. Training workflow unit 302 can determine the subset in one of several ways. For example, training workflow unit 302 can apply a Hodrick-Prescott filter to select a baseline ML model and a statistical ML model for training and / or fine-tuning.

[0111] Training workflow unit 302 can execute model evaluation and model selection processes. During model evaluation, training workflow unit 302 can train an ML model and generate evaluation metrics for the ML model. During model selection, training workflow unit 302 can use the evaluation metrics to determine the selected ML model. This disclosure relates to... Figure 4 It provides example details about the model evaluation and model selection processes.

[0112] In addition, regarding Figure 3 For example, production workflow unit 304 processes production workflows. Production workflow unit 304 can generate production workflows to generate predictions using the selected ML model for the request. This disclosure relates to... Figure 5 Example details about production workflows are provided. In some examples, production workflow unit 304 can implement a Kubernetes-based machine learning pipeline.

[0113] In addition, Figure 3 In the example, model monitoring unit 306 monitors the predictions generated by the selected ML model in response to the request. For example, model monitoring unit 306 can monitor drift in the selected ML model and can automatically trigger a training workflow that retrains the selected ML model using subsequently collected data. In other words, model monitoring unit 306 can initiate a fine-tuning process for the selected ML model. Therefore, after the selected ML model generates predictions, model monitoring unit 306 can continuously or repeatedly monitor the performance of the selected ML model.

[0114] Therefore, in some examples, the predictions generated by the selected ML model may include predictions about network traffic flows in network 205. Furthermore, in such examples, model monitoring unit 306 may generate ground truth data about network traffic flows in network 205. The ground truth data corresponds to the actual values ​​being predicted in the forecast. Model monitoring unit 306 may generate ground truth data based on data stored in data repository 259 (e.g., streaming data). Additionally, model monitoring unit 306 may determine whether to retrain the selected ML model based on a comparison of the forecast and ground truth data. Based on the determination that the selected ML model needs to be retrained, model monitoring unit 306 may automatically retrain the selected ML model. The retraining workflow may include updating the selected ML model using additional training data. For example, in an example where the selected ML model is a deep learning ML model, training workflow unit 302 may generate new input-expected-output pairs from new data stored in data repository 259 and further train the deep learning ML model using the new input-expected-output pairs.

[0115] In some examples, to determine whether to retrain the selected ML model, the model monitoring unit 306 can determine an error value based on a comparison between a specific prediction and the corresponding real data. In some examples, the error value indicates the sum of the differences between the values ​​in the real data and the corresponding values ​​in the prediction. The model monitoring unit 306 can determine to retrain the selected ML model based on the error value exceeding a confidence interval threshold.

[0116] Figure 4 This is a conceptual diagram illustrating an example training workflow unit 302 according to one or more aspects of this disclosure. Figure 4 In one example, training workflow unit 302 includes model training unit 400, parameter storage device 402, model evaluation unit 404, model and weight update unit 406, and model selection unit 408. In other examples, training workflow unit 302 may include more, fewer, or other units.

[0117] The model training unit 400 can perform the process for training an ML model. More specifically, the model training unit 400 can train an ML model to generate predictions of the type requested in the request. Thus, in some examples, the model training unit 400 can train each of a predetermined plurality of ML models to generate corresponding training-stage predictions in a plurality of training-stage predictions based on data stored in the data repository 259.

[0118] As part of training the ML model, the model training unit 400 can retrieve the parameters of the ML model from the parameter storage device 402. Furthermore, the model training unit 400 can update the parameters of the ML model as part of the training process. The model training unit 400 can store the updated parameters of the ML model back to the parameter storage device 402. Example types of ML model parameters may include weights and biases of neurons used in a deep learning ML model. Example types of parameters for a Support Vector Machine (SVM) model may include data representing one or more hyperplanes that separate one or more classes. Example types of parameters for a regression-based ML model may include coefficients representing a regression function.

[0119] In some examples, model training unit 400 can update the hyperparameters of certain types of ML models. For example, in an example where model training unit 400 is training a deep learning ML model, it can update hyperparameters such as learning rate, minimum batch size, topology parameters, etc. In an example where model training unit 400 is training an SVM model, it can update regularization constants and kernel hyperparameters. When training a statistical ML model, model training unit 400 can use grid search or other techniques to update hyperparameters. When training a deep learning ML model, model training unit 400 can perform cross-validation in stages. In the first stage, model training unit 400 can train the deep learning ML model in a short time to determine approximate values ​​for feasible hyperparameters. In the second stage, model training unit 400 can train the deep learning ML model over a longer period to fine-tune the hyperparameters.

[0120] In some examples, the ML system 138 may be configured to apply multiple ML models, but the model training unit 400 may select a subset of the multiple ML models (e.g., based on the Hodrick-Prescott filter, as described elsewhere in this disclosure). In some examples, the multiple ML models may include ML models that have been previously trained and / or fine-tuned.

[0121] Model training unit 400 can train different types of ML models in various ways. For example, in one example where model training unit 400 is training a deep learning ML model, the request may include a query and input data criteria. The query may describe the prediction requested. The input data criteria may specify the data used to generate the prediction (e.g., streaming data). Furthermore, in this example, model training unit 400 may generate multiple input-expected-output pairs. Each input-expected-output pair includes an input dataset and an expected output dataset. The input dataset may include data stored in data repository 259 and satisfy the input data criteria. The expected output dataset includes data in response to the query. In other words, model training unit 400 may run queries on data from past time periods to generate expected output datasets. In this example, model training unit 400 may train one or more ML models based on input-expected-output pairs. For example, model training unit 400 may use a supervised learning process to train one or more ML models based on input-output pairs. In this way, model training unit 400 may be able to train one or more ML models without explicitly receiving the input-expected-output dataset from the user seeking a prediction. This can help automate the prediction acquisition process.

[0122] exist Figure 4 In the example, model training unit 400 can use model evaluation unit 404 as part of the process of training the ML model. Model evaluation unit 404 can be responsible for actually using the ML model to generate predictions. For example, in an example where the ML model is a deep learning ML model, model evaluation unit 404 can perform a forward pass through the deep learning ML model to generate output data (e.g., predictions during training). In this example, model evaluation unit 404 can compare the output data with the expected output data (e.g., using an error function) to generate error values. Model evaluation unit 404 can use the error values ​​to perform a backpropagation process. The backpropagation process can update the weights of the deep learning ML model. Model and weight update unit 406 can store the updated weights back to parameter storage device 402.

[0123] In some examples, model evaluation unit 404 can determine confidence intervals for predictions generated by one or more evaluated ML models. For example, in some examples, model evaluation unit 404 applies ML models to a validation dataset that includes validation input data associated with the expected predictions. For each ML model, model evaluation unit 404 can calculate the standard deviation of the predictions generated by the ML model for the validation dataset. Model evaluation unit 404 can then use the standard deviation to determine the confidence intervals for the ML models. In some examples, confidence intervals can be used for the selection of ML models.

[0124] In some examples, the model training unit 400 can generate augmented data and use it as validation data when training a specific type of ML model. The model training unit 400 can use one or more generative adversarial networks to generate augmented data based on data stored in the data repository 259.

[0125] The model selection unit 408 can automatically determine the selected ML model from a predetermined number of ML models based on the evaluation metrics of the ML model trained for the request. The model selection unit 408 can use various evaluation metrics of the ML model to determine the selected ML model. For example, for a regression-based ML model, the model selection unit 408 can use evaluation metrics such as the root mean square error (RMSE), mean squared error (MSE), mean absolute error (MAE), the corresponding ML model's Akaike information standard (AIC), AICc, Mallows Cp, confidence interval regression, the corresponding ML model's Bayesian information standard (BIC), or other types of data. For classification-based ML models (e.g., statistical models, deep learning ML models, etc.), evaluation metrics may include accuracy, specificity, recall, F1 score (Receiver Operating Characteristic (ROC) / Area Under the Curve (AUC), etc. For ML models trained using unsupervised learning, the model selection unit 408 may use the Silhouette score as an evaluation metric. In some examples, the model selection unit 408 may compare the evaluation metrics of ML models and select the ML model with the lowest (or highest) evaluation metric. In some examples, the model selection unit 408 may use multiple evaluation metrics to determine the selected ML model. For example, the model selection unit 408 may use a combination of metrics. A weighted evaluation method is used to determine the selected ML model. Metrics can be weighted to assign more or less emphasis to specific metrics. In this example, for each ML model, model selection unit 408 can determine the sum of the weighted evaluation metrics for the ML models and select the ML model with the lowest (or highest) sum. After model selection unit 408 determines the selected ML model, it can store the parameters of the selected ML model (e.g., weights, bias values, coefficients, etc.) in parameter storage device 402 for later use. In some examples, model selection unit 408 may include one or more previously trained ML models used for the request from among the multiple ML models selected.

[0126] Figure 5 This is a conceptual diagram illustrating an example production workflow unit 304 according to one or more aspects of this disclosure. Figure 5In the example, the production workflow unit 304 includes a model training unit 400, a parameter storage device 402, a model evaluation unit 404, a model selection unit 408, a model prediction unit 500, and a model service unit 502.

[0127] The model prediction unit 500 can obtain an indication of the selected ML model for the request from the model selection unit 408. Additionally, the model prediction unit 500 can obtain parameters for the selected ML model from the parameter storage device 402. The model prediction unit 500 can apply the selected ML model to generate (e.g., based on data in data storage 259) a prediction for the request. In some examples, the request itself can specify which data in data storage 259 should be used as input to the ML model for training and / or generating production workflow predictions. For example, the request can specify a query for the data to be used as input. In some examples, dedicated training data can be provided to the ML system 138 to train the ML model to generate predictions based on the request. A flag can be used to indicate to the ML system 138 whether dedicated training data has been provided or whether the request itself specifies data in data storage 259 as input for training the ML model and / or generating production workflow predictions.

[0128] Model service unit 502 can also process predictions generated by applying a selected ML model. For example, model service unit 502 can process predictions to provide confidence intervals and forecasts.

[0129] In some examples, whenever the ML system 138 receives a specific request, the production workflow unit 304 utilizes the model training unit 400, the model evaluation unit 404, the model selection unit 408, and the model service unit 502. In some such examples, training, evaluating, and selecting the ML model as part of the production workflow for the request (and therefore as each time the ML system 138 receives a request) can be specified by the user or program that generated the request. Training, evaluating, and selecting the ML model as part of the production process is particularly useful when the selected ML model changes over time as more input data becomes available, or when the information in the input data tends to change in some way to make the selection of a different ML model advantageous.

[0130] In other examples, each time the ML system 138 receives a specific request, the production workflow unit 304 does not use the model training unit 400, the model evaluation unit 404, or the model selection unit 408. For example, if the same selected ML model is useful for the request multiple times (e.g., on the same type of input data but with different time ranges), then each time the ML system 138 receives an instance of the request, the production workflow unit 304 does not use the model training unit 400, the model evaluation unit 404, or the model selection unit 408. Perhaps particularly common is that when the selected ML model is a deep learning model, the production workflow unit 304 does not train, evaluate, or select the ML model each time the ML system 138 receives an instance of the request.

[0131] Figure 6 This is a conceptual diagram illustrating an example model object 600 according to one or more aspects of this disclosure. In some examples of this disclosure, ML system 138 can use model objects such as model object 600 to manage ML models in ML workflows such as training workflows and production workflows. Model object 600 can be an object-oriented programming software object. ML system 138 can use model object 600 for the persistence, efficient storage, and retrieval of ML models.

[0132] exist Figure 6 In the example, model object 600 includes model identifier 602, model state data 604, data retrieval unit 606, data converter 608, and trainer 610. Model identifier 602 identifies the ML model. For example, model identifier 602 may be an identifier that globally identifies the ML model in ML system 138. In some examples, model identifier 602 may be used as a reference to identify model object 600 in internal and external API calls to the ML pipeline (e.g., training workflow or production workflow).

[0133] Model state data 604 can indicate whether the ML model is in a new state (e.g., untrained), trained, evaluated, deployed, or obsolete state. When the ML model is in a trained state, it can be at least partially trained to generate predictions for a request. When the ML model is in an evaluated state, the ML model may have been considered for selection but was not selected. When the ML model is in a deployed state, it may have been selected as the ML model for the request. When the ML model is in an obsolete state, the ML system 138 no longer uses the ML model, and will not consider using it for subsequent selections or as the selected ML model. Model object 600 can be bound to a training workflow or a production workflow. The training workflow can transition model object 600 between new, trained, evaluated, and deployed states. When model state data 604 indicates that the ML model is in a deployed state, the production workflow can use the ML model associated with model object 600.

[0134] ML system 138 can use the data retriever 606 of model object 600 to retrieve input data, such as streaming data, used by the ML model. In some examples, data retriever 606 is a software object that includes functionality for retrieving input data. For example, as described below... Figure 7 The data retrieval unit 606 can be implemented as a component object. In some examples, the data retrieval unit 606 specifies a query for retrieving input data.

[0135] The ML system 138 can use a data converter 608 to transform the input data before providing it to the ML model. For example, the data processing unit 300 of the ML system 138 ( Figure 3 The input data can be preprocessed using the data retrieval unit 606. In some examples, the data converter 608 is a software object that includes the functionality to transform the input data. For example, as described below... Figure 7 The data converter 608 can be implemented as a component object. In some examples, the data converter 608 specifies instructions on how to transform the input data.

[0136] The ML system 138 can use the trainer 610 of the model object 600 to perform the training process of the ML model. For example, the trainer 610 can be or can indicate one or more programs, scripts, parameters or other data for training the ML model.

[0137] Figure 7 This is a conceptual diagram illustrating an example component object 700 according to one or more aspects of this disclosure. Component objects, such as component object 700, can be reusable, predefined, containerized operations / tasks that form the basic units of a workflow. Figure 7In the example, component object 700 includes driver 702, executor 704, and publisher 706. Component object 700 may use metadata repository 708. In other examples, component objects may include more, fewer, or different units.

[0138] Driver 702 consumes input, such as metadata about the driving component. This metadata can be stored in metadata repository 710. Actor 704 can execute actions or tasks of the component. Publisher 706 writes artifacts generated by the component back to metadata repository 710 for use in object operations. For example, when component object 700 is a data retriever (e.g., data retriever 606), driver 702 can obtain metadata specifying the actions that executor 704 will perform to retrieve input data for the ML object. In this example, publisher 706 can write the input data retrieved by executor 704 back to metadata repository 710 (e.g., for use by ML system 138 when applying an ML model).

[0139] In the example where component object 700 is a data converter (e.g., data converter 608), driver 702 can obtain metadata specifying the actions used to transform the input data. Actor 704 can then apply the actions. In this example, publisher 706 can write the transformed input data back to metadata repository 710 (e.g., for use by ML system 138 when applying an ML model).

[0140] Component objects can communicate with each other through metadata repository 710. Metadata repository 710 can implement one or more APIs to log and retrieve metadata from a storage backend (e.g., data repository 259, parameter storage device 402, etc.). In some examples, metadata includes artifacts generated by components / steps of a workflow. Furthermore, in some examples, metadata includes information about the execution of these components / steps. In some examples, metadata includes information about the pipeline and associated lineage information. For example, ML system 138 can generate metadata about workflows running on an ML model. This includes, but is not limited to, a specific identifier for the run, component-level artifacts generated during the run, and associated lineage information from pipelined workflows that have previously run on the model to achieve better debuggability and logging. Metadata repository 710 can also contain artifacts generated by component objects (such as component object 700). Artifacts can be abstract entities with an ArtifactType registered on the fly in metadata repository 710 via a client API. ArtifactType indicates the properties of instances of such a type.

[0141] Figure 8 This is a flowchart illustrating the operation of a policy controller 140 according to one or more aspects of this disclosure. Figure 8 In the example, policy controller 140 (e.g., collector module 252 of a network analysis system) can collect data such as streaming data or other types of data for network 205 (800). Network 205 includes multiple network devices (e.g., spine device 202, leaf device 203, host device 210, etc.). Policy controller 140 can store the data in a database such as data storage 259 (802).

[0142] In addition, Figure 8 In the example, policy controller 140 can receive requests for predictions (804). For example, policy controller 140 can receive requests from user interface device 129 or another source.

[0143] Based on a prediction request received by policy controller 140, ML system 138 can train each of a predetermined plurality of ML models to generate (e.g., based on data collected from network 205 or provided training data) a corresponding training-stage prediction (806) among a plurality of training-stage predictions. Examples of training ML models are provided elsewhere in this disclosure. In some examples, the predetermined plurality of ML models is a first plurality of ML models, and ML system 138 can be configured to apply a second plurality of ML models, which includes the first plurality of ML models and one or more additional ML models. However, instead of training each of the second plurality of ML models, ML system 138 can select the first plurality of ML models from the second plurality of ML models. For example, ML system 138 can use a Hodrick-Prescott filter to determine whether the input data has a predominant periodic or trend characteristic, and select an ML model from the second plurality of ML models that is suitable for the periodic or trend characteristic of the input data.

[0144] Furthermore, the ML system 138 can automatically determine the selected ML model from a predetermined plurality of ML models based on evaluation metrics for those models (808). In some examples, as part of selecting the ML model, for each corresponding ML model among the predetermined plurality of ML models, the ML system 138 can determine a corresponding set of one or more evaluation metrics for that ML model based on predictions generated by that ML model. Additionally, the ML system 138 can determine a score for that ML model based on the set of one or more evaluation metrics. For example, to determine a score for a corresponding ML model, for each corresponding ML model among the predetermined plurality of ML models, the ML system 138 can determine the score for that ML model as a weighted average of the ML metrics for that ML model. The ML system 138 can compare the scores for the ML models to determine the selected prediction. In some examples, the ML system 138 can normalize the evaluation metrics such that they are on a common scale, for example (0 to 10, 0 to 100, etc.). In some examples, the evaluation metrics for the corresponding ML model include one or more of the following: root mean square error (RMSE), mean squared error (MSE), mean absolute error (MAE), Akaike information criterion (AIC), or Bayesian information criterion (BIC). Certain types of evaluation metrics may be more suitable for selecting specific types of ML models. For example, root mean square error, AIC, and BIC may be very suitable for selection in regression-based ML models.

[0145] ML system 138 can apply a selected ML model to generate a prediction (810) based on data collected from network 205. For example, in an example where the selected ML model is a deep learning ML model, ML system 138 can perform a forward pass through the deep learning ML model to generate a prediction. ML system 138 or policy controller 140 can perform various operations related to the prediction. For example, in some examples, ML system 138 or policy controller 140 can determine whether to generate an alert based on the prediction. In this example, ML system 138 or policy controller 140 can generate an alert based on the determination to generate an alert.

[0146] For the processes, apparatuses, and other examples or illustrations described herein (including in any flowchart or diagram), certain operations, actions, steps, or events included in any techniques described herein may be performed in a different order, may be added, combined, or omitted entirely (e.g., not all described actions or events are necessary for implementing the technique). Furthermore, in some examples, operations, actions, steps, or events may be performed simultaneously rather than sequentially, for example, through multithreaded processing, interrupt handling, or multiple processors. Even if not explicitly identified as automatically executed, certain other operations, actions, steps, or events may be automatically executed. Moreover, certain operations, actions, steps, or events described as automatically executed may alternatively not be automatically executed, but rather, in some examples, such operations, actions, steps, or events may be performed in response to input or another event.

[0147] For ease of illustration, only a limited number of devices (e.g., user interface device 129, spine device 202, leaf device 203, host device 210, policy controller 140, and other devices) are shown in the figures and / or other descriptions referenced herein. However, the techniques according to one or more aspects of this disclosure can be implemented with many more such systems, components, devices, modules, and / or items, and collective references to such systems, components, devices, modules, and / or items can refer to any number of such systems, components, devices, modules, and / or items.

[0148] The accompanying drawings included herein each illustrate at least one example implementation of an aspect of this disclosure. However, the scope of this disclosure is not limited to such implementations. Therefore, other examples or alternative implementations of the systems, methods, or techniques described herein, besides those shown in the drawings, may be suitable in other circumstances. Such implementations may include a subset of the devices and / or components included in the drawings, and / or may include additional devices and / or components not shown in the drawings.

[0149] The detailed descriptions presented above are intended as descriptions of various configurations, not as representations of the only configuration in which the concepts described herein can be practiced. Specific details are included to provide a full understanding of the various concepts. However, these concepts can be practiced without these specific details. In some cases, well-known structures and components are shown in block diagrams in the accompanying drawings to avoid obscuring these concepts.

[0150] Therefore, although one or more implementations of various systems, devices, and / or components may be described with reference to specific figures, such systems, devices, and / or components can be implemented in a variety of different ways. For example, one or more devices shown as separate devices in the figures herein may alternatively be implemented as a single device; for example, one or more components shown as separate components may alternatively be implemented as a single component. Moreover, in some examples, one or more devices shown as a single device in the figures herein may alternatively be implemented as multiple devices; one or more components shown as a single component may alternatively be implemented as multiple components. Each of such multiple devices and / or components may be directly coupled via wired or wireless communication, and / or remotely coupled via one or more networks. Furthermore, one or more devices or components that may be shown in the various figures herein may alternatively be implemented as part of another device or component not shown in such figures. In this and other ways, some of the functions described herein can be performed by distributed processing of two or more devices or components.

[0151] Furthermore, certain operations, techniques, features, and / or functions may be described herein as being performed by a specific component, device, and / or module. In other examples, such operations, techniques, features, and / or functions may be performed by different components, devices, or modules. Therefore, in other examples, some operations, techniques, features, and / or functions that may be described herein as attributable to one or more components, devices, or modules may be attributable to other components, devices, and / or modules, even if not specifically described in this way herein.

[0152] While specific advantages have been identified in conjunction with the description of some examples, various other examples may include some, all, or none of the listed advantages. Other advantages in terms of technology or other aspects may become apparent to those skilled in the art from this disclosure. Furthermore, although specific examples have been disclosed herein, various aspects of this disclosure can be implemented using any number of techniques, whether currently known or not, and therefore, this disclosure is not limited to the examples specifically described and / or shown herein.

[0153] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on and / or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium (which includes any medium that facilitates the transfer of a computer program from one place to another (e.g., according to a communication protocol)). In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures to implement the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0154] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, the definition of medium includes coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) if instructions are sent from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave). However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient tangible storage media. The disks and optical discs used include compact discs (CDs), optical discs, digital versatile optical discs (DVDs), and Blu-ray discs, where disks typically copy data magnetically, while optical discs copy data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0155] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the terms "processor" or "processing circuit" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the described technique. Additionally, in some examples, the described functionality may be provided within dedicated hardware and / or software modules. Similarly, the technique can be fully implemented in one or more circuit or logic elements.

[0156] The techniques disclosed herein can be implemented in various devices or apparatuses, including integrated circuits (ICs) or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but they do not necessarily need to be implemented by different hardware units. Rather, as described above, various units can be combined within hardware units or provided as a collection of interoperable hardware units (including one or more processors described above) combined with suitable software and / or firmware.

Claims

1. A method for generating predictions about a network, the method comprising: Based on requests for predictions regarding the amount of data that will flow through communication links, devices, or virtual networks at a specific time or during a specific time period: In response to the request for the prediction, a computing system initiates a training workflow in which the computing system trains each of a predetermined plurality of machine learning (ML) models to generate a corresponding training-stage prediction for the request for the prediction, wherein the training includes using each ML model to generate a corresponding training-stage prediction with respect to the amount of data. Each of the predetermined multiple ML models is a different ML model that generates different types of predictions among multiple types of ML models, including baseline ML models, statistical ML models, and deep learning ML models; The computing system determines one or more evaluation metrics for each of the predetermined plurality of ML models, based on the corresponding training phase predictions generated by the ML models during the training period. The computing system selects a chosen ML model from the predetermined plurality of ML models based on one or more evaluation metrics for each of the predetermined plurality of ML models; as well as The computing system applies the selected ML model to streaming data collected from the network during a production workflow for processing the request for the prediction, to generate a prediction of the amount of data that will flow through the communication link, the device, or the virtual network during the specific time or the specific time period, wherein the network includes multiple network devices.

2. The method of claim 1, wherein selecting the selected ML model comprises: For each of the predetermined plurality of ML models: A score for the ML model is determined based on one or more corresponding evaluation metrics. as well as The scores for the ML models are compared to select the chosen ML model.

3. The method according to claim 1, wherein the one or more evaluation metrics include one or more of the following: root mean square error, mean square error, mean absolute error, Akaike Information Criterion (AIC), or Bayesian Information Criterion (BIC).

4. The method of claim 2, wherein determining the score for the ML model comprises: The calculation system determines the score for the ML model as a weighted average of one or more evaluation metrics for the ML model.

5. The method according to any one of claims 1 to 4, wherein: The predetermined plurality of ML models is the first plurality of ML models. The computing system is configured to apply a second plurality of ML models, including the first plurality of ML models and one or more additional ML models. The method further includes the computing system selecting the first plurality of ML models from the second plurality of ML models.

6. The method according to any one of claims 1 to 4, wherein the method further comprises: The computing system generates real data about the network; The computing system determines whether to retrain the selected ML model based on a comparison between the predictions generated by the selected ML model and the real data. as well as Based on the determination of retraining the selected ML model, the selected ML model is automatically retrained.

7. The method according to claim 6, wherein: The real data includes real data corresponding to the predictions generated by the selected ML model, and Determining whether to retrain the selected ML model includes: The calculation system determines the error value based on a comparison between the prediction and the actual data corresponding to the prediction; and The calculation system determines to retrain the selected ML model based on the error value exceeding a confidence interval threshold.

8. The method according to any one of claims 1 to 4, wherein the method further comprises: The computing system determines whether to generate an alert based on the predictions generated by the selected ML model; as well as The alarm is generated by the computing system based on the determination that generated the alarm.

9. The method according to any one of claims 1 to 4, wherein: The request includes querying and indicating input data standards, and Training each of the predetermined plurality of ML models includes: The computing system generates multiple input-output pairs, wherein: Each input-output pair includes an input dataset and a expected output dataset. The input dataset includes data stored in the database that meets the input data criteria. The expected output dataset includes data in response to the query; and The computing system trains the predetermined multiple ML models based on the input-output pairs.

10. The method according to any one of claims 1 to 4, wherein the data collected from the network comprises streaming data, and wherein the streaming data comprises at least one of: underlying streaming data or overlay streaming data.

11. A system for generating predictions about a network, the system comprising: A data repository is configured to store data collected from a network that includes multiple network devices. as well as The processing circuit is configured as follows: Based on requests for predictions regarding the amount of data that will flow through communication links, devices, or virtual networks at a specific time or during a specific time period: In response to the request for the prediction, a training workflow is initiated to train each of a predetermined plurality of machine learning (ML) models to generate a corresponding training-stage prediction for the request for the prediction, wherein the training includes using each ML model to generate a corresponding training-stage prediction with respect to the amount of data. Each of the predetermined multiple ML models is a different ML model that generates different types of predictions among multiple types of ML models, including baseline ML models, statistical ML models, and deep learning ML models; For each of the predetermined plurality of ML models, one or more corresponding evaluation metrics are determined based on the corresponding training phase predictions generated by the ML model during the training period. The selected ML model is selected from the predetermined plurality of ML models based on one or more evaluation metrics for each of the predetermined plurality of ML models; as well as During a production workflow for processing the request for the prediction, the selected ML model is applied to streaming data collected from the network to generate a prediction of the amount of data that will flow through the communication link, the device, or the virtual network during the specific time or the specific time period, wherein the network includes multiple network devices.

12. The system of claim 11, wherein the processing circuitry is configured such that, as part of selecting the selected ML model, the processing circuitry: For each of the predetermined plurality of ML models: A score for the ML model is determined based on one or more corresponding evaluation metrics; and The scores for the ML models are compared to select the chosen ML model.

13. The system of claim 11, wherein the one or more evaluation metrics include one or more of the following: root mean square error, mean square error, mean absolute error, Akaike Information Criterion (AIC), or Bayesian Information Criterion (BIC).

14. The system of claim 12, wherein the processing circuitry is configured such that, as part of determining the score for the ML model, the processing circuitry determines the score for the ML model as a weighted average of the one or more evaluation metrics for the ML model.

15. The system according to any one of claims 11 to 14, wherein: The predetermined plurality of ML models is the first plurality of ML models. The processing circuitry is configured to apply a second plurality of ML models, including the first plurality of ML models and one or more additional ML models. The processing circuit is also configured to select the first plurality of ML models from the second plurality of ML models.

16. The system according to any one of claims 11 to 14, wherein the processing circuitry is further configured to: Generate real data about the network; Whether to retrain the selected ML model is determined based on a comparison between the predictions generated by the selected ML model and the real data; and Based on the determination of retraining the selected ML model, the selected ML model is automatically retrained.

17. The system according to claim 16, wherein: The real data includes real data corresponding to the predictions generated by the selected ML model, and The processing circuit is configured such that, as part of determining whether to retrain the selected ML model, the processing circuit: The error value is determined based on a comparison between the prediction and the actual data corresponding to the prediction; as well as The determination to retrain the selected ML model is made based on the error value exceeding the confidence interval threshold.

18. The system according to any one of claims 11 to 14, wherein the processing circuitry is further configured to: Whether to generate an alert is determined based on the prediction generated by the selected ML model; and The alarm is generated based on the determination that generated the alarm.

19. The system according to any one of claims 11 to 14, wherein: The request includes querying and indicating input data standards, and The processing circuit is configured such that, as part of training each of the predetermined plurality of ML models, the processing circuit: Generate multiple input / output pairs, where: Each input-output pair includes an input dataset and a expected output dataset. The input dataset includes data stored in the database that meets the input data criteria. The expected output dataset includes data in response to the query; and The predetermined multiple ML models are trained based on the input-output pairs.

20. A computer-readable storage medium encoded with instructions for causing one or more programmable processors to perform the method according to any one of claims 1 to 10.

Images