Performance regulation in network systems

Abstract

A container-based orchestration system includes a master node and a plurality of worker nodes. The master node can receive, from each agent executing on a corresponding worker node, node characteristics associated with the worker node. The master node can determine, for each worker node, one or more parameters corresponding to the node characteristics associated with the corresponding worker node and a node profile of the worker node, and provide the parameters to the agent executing on the corresponding worker node. The agent configures the worker node according to the parameters. In response to receiving a request to deploy a container group pod to a worker node, the master node can select the worker node to receive the pod based on the node characteristics and pod characteristics. The agent can configure the selected worker node to execute a workload of the pod according to the one or more parameters.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Patent Application No. 17 / 455,867, filed November 19, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to computing systems, and more specifically to the parameter configuration of network nodes in network systems. Background Technology

[0004] In a typical cloud data center environment, there exists a large collection of interconnected servers that provide computing and / or storage capacity to run various applications. For example, a data center may include facilities for hosting applications and services for subscribers (i.e., the data center's customers). For instance, a data center may host all infrastructure equipment, such as network and storage systems, redundant power supplies, and environmental controls. In a typical data center, storage systems and application server clusters are interconnected via a high-speed switch architecture provided by one or more physical network switches and router layers. More complex data centers provide globally distributed infrastructure through subscriber support facilities located in various physical hosting facilities.

[0005] Virtualized data centers are becoming a core foundation of modern information technology (IT) infrastructure. Specifically, modern data centers have widely utilized virtualized environments, in which virtual machines (also referred to in this paper as virtual execution elements, such as virtual machines or containers) are deployed and run on the underlying computing platform of physical computing devices.

[0006] Virtualization within a data center, or any environment comprising one or more servers, offers several advantages. One advantage is that virtualization can significantly improve efficiency. With the advent of multi-core processor architectures where each physical central processing unit (CPU) has a large number of cores, the underlying physical computing devices (i.e., servers) have become increasingly powerful, making virtualization easier and more efficient. A second advantage is that virtualization provides significant control over the computing infrastructure. As physical computing resources become alternative resources, such as in cloud-based computing environments, the provisioning and management of computing infrastructure becomes easier. Therefore, in addition to the efficiency and increased return on investment (ROI) offered by virtualization, enterprise IT personnel often prefer virtualized computing clusters in data centers due to their management advantages.

[0007] Containerization is a virtualization solution based on operating system-level virtualization. Containers are lightweight, portable execution elements used to isolate applications from each other and from the host machine. Because containers are not tightly coupled to the host hardware computing environment, applications can be bound to container images and executed as a single, lightweight wrapper on any host or virtual machine that supports the underlying container architecture. In this way, containers solve the problem of how to make software work across different computing environments. Containers offer the promise of consistent operation from one computing environment to another, whether virtual or physical.

[0008] Due to the inherently lightweight nature of containers, a single host can typically support far more container instances than a traditional virtual machine (VM). Short-lived containers are generally more efficient to create and move than VMs, and they can also be managed as logically related groups of components (sometimes referred to as "pods" in some orchestration platforms, such as Kubernetes). Network systems that manage the deployment and infrastructure for application execution can use pods as part of orchestration, for example, to automate the deployment, scaling, and operation of applications across host clusters, and to provide compute infrastructure that can include container-centric compute infrastructure. Network systems can also use pods for network management, for example, to create virtual networks within the network infrastructure, enabling grouped applications to run between virtual execution environments such as containers or VMs, and between applications running on traditional (e.g., physical) environments. Summary of the Invention

[0009] Generally, this disclosure describes a performance controller for a container-based orchestration system that implements techniques for determining performance parameters based on node configuration and dynamically adjusting these parameters as conditions and workloads change within the network system. In large-scale deployment environments such as data centers, multi-node clusters may exist, where nodes within the cluster may have different configurations and varying available resources. Furthermore, nodes may be used by different applications with different runtime characteristics and requirements. For example, some applications may require significant memory, while others require fast response times. Various parameters affect aspects of the performance of nodes and the applications running on them. The values ​​of these parameters may vary depending on the available resources (real and virtual) on the node and the workload hosted by the node. The process of setting or adjusting parameters on a node to attempt to achieve performance characteristics or for a specific type of workload, application, or environment can be referred to as “tuning” the node. Similarly, parameters that can be adjusted to tune a node to achieve performance characteristics or for a specific type of workload, application, or environment can be referred to as “tuning parameters.”

[0010] As an example, a node providing virtual routing services can provide these services as a user-mode application running in the node's user-space memory, as a kernel-mode virtual router (vrouter) running in kernel-space memory, or as a vrouter integrated with a smart network interface card (NIC). A node hosting a virtual router running in user-space memory may have different tuning parameter values ​​than a node hosting a kernel-mode virtual router running in kernel-space memory. In some cases, tuning a node for a specific performance characteristic can lead to a degradation in another performance characteristic. For example, tuning a node for power saving might improve the node's performance in terms of power usage, but might reduce the node's performance in terms of CPU processing.

[0011] Existing systems do not provide the ability to dynamically and automatically reconfigure nodes or adjust tuning parameters to maximize performance under current workload conditions. If tuning is done, it is typically done manually, node by node (e.g., by an administrator), and once done, the tuning parameters usually remain static and do not change. Because tuning the numerous nodes that may exist in a data center is both tedious and error-prone, it is often not done at all.

[0012] This disclosure describes techniques for dynamically and automatically reconfiguring nodes and / or adjusting parameters to improve or maximize performance under current workload conditions. A performance controller on the master node of a node cluster can communicate with performance agents on the worker nodes of the cluster. The performance agents running on the nodes can both report current node configuration data to the performance controller and receive node tuning parameters, which may be based on the current node configuration and the current workload being performed on the node.

[0013] The techniques disclosed herein can be incorporated into practical applications that offer technological advantages over existing systems. For example, as noted above, existing systems do not dynamically and automatically reconfigure nodes or adjust tuning parameters to maximize performance under current workload conditions. A practical application of the techniques disclosed herein is a performance controller and performance agent that can set various parameters on worker nodes to tune worker nodes based on a node profile specifying the desired characteristics of the worker nodes. The technical advantage of the techniques disclosed herein is that a system including a performance controller implementing this technique can maximize node performance in a cluster using feedback received from a performance agent executing on the worker nodes of the cluster. The performance controller can use feedback received from the node's performance agent to set values ​​for various parameters to tune the node for maximum performance given the characteristics specified in the assigned node profile. The performance controller can provide tuning parameter values ​​to the performance agent, and the performance agent can enforce the tuning parameters on the node. Furthermore, the tuning parameters can be adjusted over time as conditions within the node or node cluster change. Therefore, the techniques disclosed herein can improve existing systems by facilitating the maximization of performance in clusters with a large number of worker nodes.

[0014] In one example, a method includes: receiving, by a master node of a cluster, multiple node characteristics associated with a corresponding worker node from each agent executing on a corresponding worker node among multiple worker nodes in the cluster, wherein the cluster is a container-based orchestration system cluster, and wherein the master node manages the orchestration of workloads on the worker nodes of the cluster; for each worker node among the multiple worker nodes, determining one or more parameters corresponding to the multiple node characteristics associated with the corresponding worker node and the node profile of the corresponding worker node, and providing one or more parameters to a corresponding agent executing on the worker node through the master node, wherein the corresponding agent configures the worker node according to the one or more parameters; receiving, by the master node, a request to deploy a pod having one or more container group pod characteristics; selecting, by the master node, a selected worker node among the multiple worker nodes based on the multiple node characteristics and the one or more pod characteristics, wherein the pod is to be deployed on the selected worker node; and scheduling the pod on the selected worker node by the master node.

[0015] In another example, a worker node includes multiple processors; memory coupled to the multiple processors; an orchestration agent that deploys pods to worker nodes in response to instructions from a master node, the master node orchestrating workloads on the worker nodes, the master node and worker nodes residing in the same node cluster, wherein the cluster is a container-based orchestration system cluster; and a performance agent executable by the multiple processors, the performance agent being configured to: monitor resource usage on the worker nodes; report resource usage to the master node; receive one or more parameters from the master node; receive an indication from the orchestration agent that a pod has been deployed to a worker node, the worker node being selected by the master node based on resource usage; and configure the worker node to perform the pod's workload based on one or more parameters.

[0016] In another example, a container-based orchestration system includes a master node; and a plurality of worker nodes communicatively coupled to the master node, wherein the master node manages the orchestration of workloads on each of the worker nodes, and wherein the master node includes a memory storing instructions that, when executed by the master node's processing circuitry, cause the master node to: receive a plurality of node characteristics associated with a corresponding worker node from each agent executing on a corresponding worker node in the plurality of worker nodes in the cluster; determine, for each of the worker nodes, one or more parameters corresponding to the plurality of node characteristics associated with the corresponding worker node and its node profile; and provide one or more parameters to a corresponding agent executing on the worker node, wherein the corresponding agent configures the worker node according to the one or more parameters; receive a request to deploy a pod having one or more pod characteristics; select a selected worker node from the plurality of worker nodes to receive the pod based on the plurality of node characteristics and the one or more pod characteristics; and schedule the pod on the selected worker node, wherein the agent of the selected worker node is configured to: receive an indication that the pod has been scheduled to the selected worker node; and configure the selected worker node to execute the pod's workload according to the one or more parameters.

[0017] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, the drawings, and the claims. Attached Figure Description

[0018] Figure 1 This is a block diagram illustrating an example network according to one or more aspects of this disclosure.

[0019] Figure 2 This is a block diagram illustrating aspects of the master node and worker nodes in a cluster of a network system according to one or more aspects of this disclosure.

[0020] Figure 3 This is a block diagram of an example computing device that includes performance agents for adjusting performance parameters and configuring computing devices according to the techniques described in this disclosure.

[0021] Figure 4 This is a block diagram of an example computing device operation that includes a performance controller for adjusting the performance parameters of worker nodes.

[0022] Figure 5 This is a block diagram illustrating an example cluster adapted according to the techniques described in this article.

[0023] Figure 6 This is a flowchart illustrating example operations according to one or more techniques of the present invention.

[0024] Figure 7 The example pod application specification is shown. Detailed Implementation

[0025] Figure 1 This is a conceptual diagram illustrating an example network that includes a performance controller for adjusting the performance parameters of nodes in the network. Figure 1 The illustration depicts a network system 100 and a data center 101 hosting one or more computing networks, computing domains, or projects, and / or, as commonly referred to herein, a cloud-based computing cluster. Cloud-based computing clusters can coexist in a common overall computing environment, such as a single data center, or be distributed across various environments, such as across different data centers. Cloud-based computing clusters can, for example, be different cloud environments, such as OpenStack cloud environments, Kubernetes cloud environments, or various combinations of other computing clusters, domains, networks, etc. Other implementations of network system 100 and data center 101 may be suitable in other circumstances. Such implementations may include... Figure 1 The examples include a subset of components and / or may include Figure 1 Additional components not shown.

[0026] exist Figure 1 In the example, data center 101 provides an operating environment for applications and services to customer 104 coupled to data center 101 via service provider network 106. Although combined Figure 1 The functions and operations described in the network system 100 can be illustrated as distributed across Figure 1 Among the multiple devices, but in other examples, it belongs to Figure 1The features and technologies of one or more devices can be executed internally by native components of one or more such devices. Similarly, one or more such devices can include certain components and execute various technologies that may otherwise be attributed to one or more other devices in the description herein. Furthermore, they can be combined... Figure 1 Describes or otherwise describes certain operations, techniques, features, and / or functions performed by a particular component, device, and / or module. In other examples, such operations, techniques, features, and / or functions may be performed by other components, devices, or modules. Thus, some operations, techniques, features, and / or functions attributable to one or more components, devices, or modules may be attributable to other components, devices, and / or modules, even if not specifically described in this way herein.

[0027] Data center 101 hosts infrastructure equipment such as network and storage systems, redundant power supplies, and environmental controls. Service provider network 106 can be coupled to one or more networks managed by other providers, thus forming part of a large-scale public network infrastructure, such as the Internet.

[0028] In some examples, data center 101 can represent one of many geographically distributed network data centers. For example... Figure 1 As illustrated in the example, data center 101 is a facility that provides network services to client 104. Client 104 can be a collective entity, such as a business or government, or an individual. For example, a network data center may host web services for multiple businesses and end users. Other example services may include data storage devices, virtual private networks, business engineering, file services, data mining, scientific or supercomputing, etc. In some examples, data center 101 is a single network server, a network peer, or something else.

[0029] exist Figure 1 In this example, data center 101 includes a set of storage systems, application servers, compute nodes, or other devices, including network device 107. Network device 107 may include master node 109 and worker nodes 110A to 110N (collectively referred to as “worker nodes 110”, meaning any number of worker nodes). Device 107 may be interconnected via a high-speed switch architecture 121 provided by one or more physical network switches and router layers. In some examples, device 107 may be included within architecture 121, but is shown separately for ease of illustration.

[0030] Network device 107 can be any of several different types of network devices (core switch, spine network device, leaf network device, edge network device, or other network devices), but in some examples, one or more devices 107 can serve as physical compute nodes or storage nodes in a data center. For example, one or more devices 107 can provide an operating environment for executing one or more customer-specific applications or services. Alternatively or additionally, one or more devices 107 can provide an operating environment for one or more virtual machines or other virtualized instances such as containers. In some examples, one or more devices 107 can be alternatively referred to as host compute devices, or more simply as hosts. Thus, network device 107 can execute one or more virtualized instances, such as virtual machines, containers, or other virtual execution environments, for running one or more services, such as virtualized network functions (VNFs).

[0031] Generally, each network device 107 in network device 107 can be any type of device that can operate on a network and generate data accessible by telemetry or other means (e.g., connectivity data, traffic data, sFlow data). This device can include any type of computing device, sensor, camera, node, monitoring device, or other device. Furthermore, some or all of the network devices in network device 107 can represent components of another device that can generate data collectable by telemetry or other means. For example, some or all of the network devices in network device 107 can represent physical or virtual network devices such as switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection and / or intrusion prevention devices.

[0032] Although not specifically shown, switch architecture 121 may include top-of-rack (TOR) switches of the distribution layer coupled to the chassis switches, and data center 101 may include one or more non-edge switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection and / or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices. Switch architecture 121 may perform Layer 3 routing to route network traffic between data center 101 and customer 104 through service provider network 106. Gateway 108 is used to forward and receive packets between switch architecture 121 and service provider network 106.

[0033] Any network device in network device 107 can be configured with virtual execution elements using the resources of the virtualization server to provide isolation between one or more processes (applications) executing on the server. "Hypervisor-based," "hardware-level," or "platform" virtualization refers to the creation of virtual machines, each containing a guest operating system for executing one or more processes. Generally, virtual machines provide a virtualization / guest operating system for executing applications in an isolated virtual environment. Because virtual machines are virtualized from the host server's physical hardware, the executing application is isolated from both the host hardware and other virtual machines. Each virtual machine can be configured with one or more virtual network interfaces for communication on the corresponding virtual network.

[0034] A virtual network is a logical structure implemented on top of a physical network. Virtual networks can be used as an alternative to VLAN-based isolation and to provide multi-tenancy in virtualized data centers (e.g., Data Center 101). Each tenant or application can have one or more virtual networks. Each virtual network can be isolated from all other virtual networks unless explicitly permitted by security policies.

[0035] Virtual networks can use the Data Center 101 Edge Router ( Figure 1 (Not shown) Connects to and extends across Physical Multiprotocol Label Switching (MPLS) Layer 3 Virtual Private Networks (L3VPNs) and Ethernet Virtual Private Networks (EVPNs). Virtual networks can also be used to implement Network Functions Virtualization (NFV) and service chaining.

[0036] Virtual networks can be implemented using various mechanisms. For example, each virtual network can be implemented as a Virtual Local Area Network (VLAN), a Virtual Private Network (VPN), etc. Virtual networks can also be implemented using two networks: a physical underlying network consisting of a switch architecture 121 and a virtual overlay network. The role of the physical underlying network is to provide an "IP architecture," offering unicast Internet Protocol (IP) connectivity from any physical device (server, storage device, router, or switch) to any other physical device. The underlying network can provide unified low-latency, non-blocking, high-bandwidth connectivity from any point in the network to any other point in the network.

[0037] As further described below regarding the virtual router (vrouter) 111A, the virtual routers operating in network device 107 create a virtual overlay network on top of the physical underlying network using a mesh of dynamic “tunnels” between them. For example, these overlay tunnels can be MPLS, VXLAN, or NVGRE tunnels over GRE / UDP tunnels. The virtual router can also support Segmented Routing-MPLS (SR-MPLS) and IPv6 layers. The underlying physical routers and switches may not contain any per-tenant state of virtual machines or other virtual enforcement elements such as media access control (MAC) addresses, IP addresses, or policies. For example, the forwarding tables of the underlying physical routers and switches may only contain the IP prefix or MAC address of network device 107. (The gateway router or switch connecting the virtual network to the physical network is an exception and may contain tenant MAC or IP addresses.)

[0038] “Container-based” or “OS-based” virtualization refers to the virtualization of an operating system to run multiple isolated systems on a single machine (virtual or physical). These isolated systems represent containers, such as those provided by the Kubernetes Container Runtime Interface – Open Container Initiative (CRI-O), containerd, the open-source Docker container application, or CoreOS Rkt (“Rocket”). Like virtual machines, each container is virtualized and may remain isolated from the host machine and other containers. However, unlike virtual machines, each container can omit its separate operating system, providing only an application suite and application-specific libraries. Generally, containers are executed by the host machine (e.g., one of the network devices in network device 107) as isolated user-space instances and can share the operating system and common libraries with other containers running on the host machine. Therefore, containers can require fewer processing power, storage, and network resources compared to virtual machines. A group of one or more containers can be configured to share one or more virtual network interfaces for communication on corresponding virtual networks. In some examples, containers are managed by their host kernel to allow limiting resources (CPU, memory, block I / O, network, etc.) and determining the priority of those resources without launching any virtual machines. In some examples, containers can be deployed based on Linux Containers (LXC), an operating system-level virtualization approach that uses a single Linux kernel to run multiple isolated Linux systems (containers) on a controlling host.

[0039] Network device 107 hosts virtual network endpoints for one or more virtual networks, which are connected via switch architecture 121. Figure 1Operations are carried out on the physical network represented in the diagram. Although the description primarily concerns data center-based switching networks, other physical networks, such as service provider network 106, may reside under one or more virtual networks.

[0040] In some examples, network system 100 includes orchestrator 130 and SDN controller 132. Virtual execution elements can be deployed to a virtualized environment using a cluster-based framework, where the cluster master node (e.g., master node 109) manages the deployment and operation of containers to one or more cluster worker nodes (e.g., worker nodes 110A to 110N). The terms "master node" and "worker node" as used herein encompass different orchestration platform terms used to distinguish between the main management element of a cluster and similar devices that are the main virtual execution elements of a cluster's managed devices.

[0041] Orchestrator 130 and SDN controller 132 may execute on separate computing devices or on the same computing device. Each of orchestrator 130 and SDN controller 132 may be a distributed application executing on one or more computing devices. Orchestrator 130 and SDN controller 132 may implement corresponding master nodes for one or more clusters, each cluster having one or more worker nodes 110 implemented by corresponding network devices 107. Generally, SDN controller 132 controls the network configuration of the data center 101 architecture to establish one or more virtual networks, for example, for packet communication between virtual network endpoints. SDN controller 132 provides a logically and, in some cases, physically centralized controller for facilitating the operation of one or more virtual networks within data center 101. In some examples, SDN controller 132 may operate in response to configuration input received from orchestrator 130 and / or administrator / operator. Additional information regarding the SDN controller 132 in conjunction with other devices or other software-defined network operations of data center 101 can be found in International Application No. PCT / US2013 / 044378, filed June 5, 2013, entitled "PHYSICAL PATH DETERMINATION FOR VIRTUAL NETWORK PACKET FLOWS", U.S. Patent No. 9,571,394, granted February 14, 2017, entitled "Tunneled Packet Aggregation for Virtual Networks", and U.S. Patent No. 10,728,145, granted July 28, 2020, entitled "MULTIPLE VIRTUAL NETWORK INTERFACESUPPORT FOR VIRTUAL EXECUTION ELEMENTS", each of which is incorporated herein by reference as fully set forth herein.

[0042] Generally, orchestrator 130 controls the deployment, scaling, and operation of virtual execution elements across clusters of network devices 107 and provides computing infrastructure, which may include container-centric computing infrastructure. In some examples, orchestrator 130 manages the functionality of data center 101, such as compute, storage, networking, and application resources. For example, orchestrator 130 can create virtual networks for tenants within or across data centers 101. Orchestrator 130 can attach virtual machines (VMs) to tenants' virtual networks. Orchestrator 130 can connect tenants' virtual networks to external networks, such as the Internet or a VPN. Orchestrator 130 can implement security policies across a set of VMs or at the boundary of tenant networks. Orchestrator 130 can deploy network services (e.g., load balancers) within tenants' virtual networks. Orchestrator 130, and in some cases SDN controller 132, can implement corresponding cluster masters for one or more Kubernetes clusters. As an example, Kubernetes is a container management platform that provides portability across public and private clouds, where each cloud can provide virtualization infrastructure for the container management platform. Kubernetes is available from the Cloud Native Computing Foundation in San Francisco, California. Each network device 107 in network device 107 can host one or more virtual execution elements, each virtual execution element having at least one virtual network endpoint for configuring one or more virtual networks in a physical network. A virtual network endpoint of a virtual network can represent one or more virtual execution elements that share the virtual network interface of the virtual network. For example, a virtual network endpoint can be a virtual machine, a group of one or more containers (e.g., a pod), or another virtual execution element, such as a Layer 3 endpoint of a virtual network. The term "virtual execution element" includes virtual machines, containers, and other virtualized computing resources that provide at least a partially independent execution environment for an application. The term "virtual execution element" can also encompass a pod of one or more containers. Figure 1 As shown, worker node 110A hosts a virtual network endpoint as a pod 122A with one or more containers. However, given the hardware resource limitations of worker node 110, worker node 110 can execute as many virtual execution elements as possible. Each virtual network endpoint can use one or more virtual network interfaces to perform packet I / O or otherwise process packets. For example, a virtual network endpoint can use a virtual hardware component (e.g., a VNF) enabled by NIC 113A to perform packet I / O and use switch architecture 121 to receive / transmit packets on one or more communication links. Other examples of virtual network interfaces are described below.

[0043] Each worker node 110 includes at least one NIC 113, and each NIC 113 includes at least one interface for exchanging packets with switch fabric 121 via a communication link. For example, worker node 110A includes NIC 113A. Any of the NICs 113 can provide one or more virtual hardware components, such as a virtual router 111 for virtualizing input / output (I / O). The virtual hardware component for I / O can be a virtualization (“physical function”) of the physical NIC 113. For example, the physical function of a network interface card (or “network adapter”) is virtualized to present one or more virtual network interfaces as “virtual functions” for use by a corresponding endpoint executed on network device 110. In this way, virtual network endpoints can share the same physical hardware resources, and virtual functions are examples of virtual hardware components. The virtual hardware component associated with NIC 113A can be associated with a Layer 2 destination address, which can be assigned by NIC 113A or the software process responsible for configuring NIC 113A.

[0044] Virtual router 111 terminates the virtual network overlay tunnel and determines the virtual network of the received packet based on the packet's tunnel encapsulation header, and forwards the packet to the appropriate destination virtual network endpoint. For worker node 110A, for example, for each packet outbound from a virtual network endpoint hosted by worker node 110A (e.g., pod 122A), virtual router 111A appends a tunnel encapsulation header to generate an encapsulated packet, or "tunnel" packet, that indicates the packet's virtual network, and virtual router 111A outputs the encapsulated packet via the virtual network overlay tunnel to a physical destination computing device, such as another network device in network device 107. As used herein, virtual router 111 can perform the following operations of a tunnel endpoint: encapsulate internal packets originating from a virtual network endpoint to generate a tunnel packet and decapsulate the tunnel packet to obtain internal packets for routing to other virtual network endpoints.

[0045] Network system 100 implements an automation platform for the automated deployment, scaling, and operation of virtual execution elements across network device 107 to provide virtualized infrastructure for executing application workloads and services. In some examples, the platform may be a container orchestration platform that provides container-centric infrastructure for automating the deployment, scaling, and operation of containers. In the context of virtualized computing infrastructure, "orchestration" generally refers to the provisioning, scheduling, and management of virtual execution elements and / or applications and services executed on those virtual execution elements by host servers available to the orchestration platform. Specifically, container orchestration permits container coordination and refers to the deployment, management, scaling, and configuration of containers to managed servers, for example, through a container orchestration platform. Example instances of orchestration platforms include Kubernetes, Docker Swarm, Mesos / Marathon, OpenShift, OpenStack, VMware, and Amazon ECS.

[0046] In one example, pod 122A is a Kubernetes pod and an example of a virtual network endpoint. A pod is a group of one or more logically related containers. Figure 1 (Not shown in the image), the container's shared storage, and options regarding how the container runs. When instantiated for execution, a pod can alternatively be referred to as a "pod replica." Each container in pod 122A is an example of a virtual execution element. The containers of a pod reside together on a single server, are jointly scheduled, and run in a shared context. The shared context of a pod can be a set of Linux namespaces, cgroups, and other small aspects of isolation. Within the pod's context, individual applications may have other sub-isolation applied. A pod such as pod 122A can represent various workloads executed by network system 100.

[0047] Worker node 110A includes a container platform 119A for running containerized applications, such as container 122A. Container platform 119A receives requests from orchestrator 130 to acquire and host containers in worker node 110A. Container platform 119A acquires and executes containers.

[0048] Container platform 119A includes a network module 117A that configures virtual network interfaces for virtual network endpoints. Container platform 119A uses network module 117A to manage the network of pods, including pod 122A. For example, network module 117A creates virtual network interfaces to connect pods to virtual router 111A and enables containers of these pods to communicate with other virtual network endpoints on the virtual network via the virtual network interfaces. For example, network module 117A can insert the virtual network interface of a virtual network into the network namespace of a container in pod 122A and configure (or request configuration) the virtual network interface of the virtual network in virtual router 111A such that virtual router 111A is configured to send packets received from the virtual network via the virtual network interface to the container of pod 122A, and to send packets received from the container of pod 122A on the virtual network via the virtual network interface. Network module 117A can assign network addresses (e.g., virtual IP addresses for the virtual network) and can set routes for the virtual network interfaces. In Kubernetes, by default, all pods can communicate with all other pods without using Network Address Translation (NAT).

[0049] As part of the process of creating pod 122A, orchestrator 130 sends a message to SDN controller 132 requesting SDN controller 132 to create appropriate virtual network interfaces for multiple virtual networks (indicated in configuration data). Orchestrator 130 may store, send, or otherwise notify SDN controller 132 of virtual network configuration objects for the multiple virtual networks specified for pod 122A. For example, orchestrator 130 may obtain a list of pods including annotations indicating the interface types for the virtual networks used by pod 122A and deploy pod 122A to a host computing device. In this example, orchestrator 130 may store pod configuration data for pod 122A (e.g., virtual network configuration objects for the multiple virtual networks specified for pod 122A). Pod configuration data may include the interface types of the pod's virtual networks. Pod configuration data may determine the interface types specified in the request to configure pod 122A. SDN controller 132 may configure any virtual networks that have not yet been configured in network system 100.

[0050] SDN controller 132 processes pod creation requests to generate interface configuration data for multiple virtual network interfaces 126 of pod 122A for communication via virtual networks. The interface configuration data may include a container or pod unique identifier and a list or other data structure specifying network configuration data for each virtual network interface 126. The network configuration data for the virtual network interfaces may include a network name, an assigned virtual network address, a MAC address, and / or a domain name server value. Multiple virtual network interfaces 126 may correspond to multiple virtual networks.

[0051] The network configuration data for pod 122A may include data describing multiple different virtual network interfaces 126. For example, orchestrator 130 may obtain a pod manifest that includes annotations indicating the interface type of the virtual network for pod 122A and deploy container 122A to a host computing device. In this example, orchestrator 130 may store pod configuration data for pod 122A (e.g., network configuration data for pod 122A for virtual network interface 126A). The pod configuration data may include the interface type of the pod's virtual network. The pod configuration data may determine the interface type specified in the request to configure pod 122A.

[0052] The performance controller 133 of the master node 107 can interact with the performance agent 135 of the worker node 110. The performance controller 133 and the performance agent 135 can implement one or more of the techniques described herein to adjust or regulate parameters of the worker node 110's operation to improve or maximize the performance of the worker node 110 in various objectives, such as low latency, maximum throughput, power savings, etc. For example, the performance controller 133 and the performance agent 135 can dynamically and automatically reconfigure the worker node 110 and / or adjust the parameters of the worker node 110 to improve or maximize performance under current workload conditions. The performance agent 135, executing on the worker node 110, can both report current node configuration data to the performance controller 133 and receive node tuning parameters from the performance controller 133, which can be based on the current node configuration and the current workload executing on the node. The following section discusses... Figure 2 Further details on the operation of performance controller 133 and performance agent 135 are provided.

[0053] Figure 2 This is a block diagram illustrating aspects of the master node and worker nodes in a cluster of a network system according to one or more aspects of this disclosure. Figure 2In the example shown, cluster 201 includes master node 202 and worker nodes 204A to 204N (collectively referred to as "worker nodes 204"). Worker nodes can also be referred to as "compute nodes". Master node 202 can be... Figure 1 The example implementation of master node 107, and worker node 204 can be Figure 1 Example implementation of worker node 110.

[0054] In some aspects, the master node 202 includes a controller manager 210, a performance controller 212, an API server 214, a default scheduler 215, a custom scheduler 216, and a database 218. The performance controller 212 can be... Figure 1 An example implementation of performance controller 133. Master node 202 can manage worker nodes 204A to 204N in cluster 201. In some respects, master node 202 and worker nodes 204A to 204N can be Kubernetes nodes.

[0055] Database 218 serves as backup storage for cluster data in cluster 101. This cluster data may include cluster state and configuration data. In some aspects, database 218 may be implemented as a key-value store. Database 218 may be a central database or a distributed database. In some aspects, database 218 may be a Kubernetes etcd database. Database 218 may store node profiles 220 and pod specifications 222. In some aspects, node profile 220 may be a Kubernetes configmap. Node profile 220 may include parameters describing one or more worker nodes 204A to 204N. In some aspects, one or more of these parameters may be tuning parameters for worker nodes. Examples of these parameters include HugePages, CPU pinning, Non-Uniform Memory Access (NUMA) node concatenation, pod scheduling, CPU sharing limits, Network Interface Card (NIC) configuration, CPU regulators, Operating System (OS) startup parameters, Virtual Router (vrouter) configuration, CPU yielding parameters, etc.

[0056] Oversized pages (also known as superpages or large pages) indicate that the worker node supports setting the memory page size to a larger size than the default size (typically 4KB). Using a larger page size than the default can make memory management more efficient, at the cost of potentially wasting memory.

[0057] CPU pinning determines how a pod or application is assigned to CPUs. CPU pinning can be used to permanently assign a fixed set of CPUs to an application or pod. It can be advantageous to explicitly assign different processors to different components to compete for the same processor. For example, from a performance perspective, pinning a DPDK vrouter to a different processor than the one used by an application such as a VNF using the vrouter can be advantageous. In this example, the DPDK vrouter and the process using the DPDK vrouter do not need to compete for the same set of processors.

[0058] NUMA node concatenation specifies how pods are concatenated to NUMA nodes in a multiprocessor system. In a NUMA implementation, each core in a multiprocessor system can access local memory (e.g., memory on its NUMA node) or memory on other NUMA nodes. Accessing memory on a NUMA node, rather than its own memory, is slower than accessing local memory. In some respects, performance can be improved by concatenating pod applications or other processes to the same NUMA node. For example, in the case of DPDK vrouter, applications using DPDK vrouter execute faster if they are concatenated to the same NUMA node as DPDK vrouter.

[0059] CPU sharing limits specify how CPU should be shared between applications in a pod; for example, how much shared CPU time should be provided to a process.

[0060] Network interface card (NIC) configuration can indicate whether the NIC supports DPDK mode, frame size, etc. For example, "jumbo frame" indicates that the NIC supports Ethernet frames larger than 1500 bytes. Additionally, depending on the NIC type, certain tasks can be offloaded from other processors to the NIC. For example, the NIC may be able to perform tasks such as Generic Receive Offload (GRO), Generic Segment Offload (GSO), hashing, checksum calculation, multi-queue load balancing, etc. The vrouter can be tuned to use these NICs to assist in offloading.

[0061] A CPU controller can limit the CPU frequency to achieve power saving goals.

[0062] Operating system (OS) boot parameters can include parameters such as setting the massive page size, enabling / disabling hyper-threading, and CPU isolation parameters.

[0063] The virtual router configuration can indicate whether the virtual router is configured as a kernel-mode virtual router or a Data Plane Development Kit (DPDK) virtual router.

[0064] CPU yield parameters control the conditions under which a process yields the CPU to other processes.

[0065] Different node profiles can have different parameter settings for the above parameters, depending on whether the profile is for maximum packets per second (PPS), minimum latency, power saving, maximum throughput, or a balance of the above. For example, a low-latency profile might tune nodes to disable hyper-threading, discover physical NIC-to-NUMA integration, perform CPU pinning based on configuration and CPU availability and information about physical NIC-to-NUMA integration, configure vrouter to adjust queue / buffer size and burst processing for low latency, use a large buffer size for a throughput profile, and set CPU regulator parameters for performance.

[0066] As another example, power-saving profiles can allow the scheduler to avoid scheduling and running DPDK-based application workloads on nodes with power-saving profiles to avoid continuous polling of CPUs. The CPU governor can be set to power-saving, which allows the CPU to run at a lower frequency, thus saving power.

[0067] The profile assigned to a node instructs the node to be tuned using the parameters indicated in the profile. For example, a PPS profile with parameters designed to adjust worker nodes to provide maximum PPS can be assigned to a node that is to provide maximum PPS. Similarly, a minimum latency profile with parameters designed to adjust worker nodes to obtain minimum latency can be assigned to a node that is to provide minimum latency. Node profiles can be combined or modified to create new profiles.

[0068] The cluster can include nodes with mixed profiles, and nodes can be categorized based on their profiles. Workload scheduling (based on workload type, such as DPDK or non-DPDK pods) can be tuned based on the profile.

[0069] Pod specification 222 describes aspects of pods that can be deployed to worker nodes 204A through 204N. The pod specification can describe how the system should run the containers within the pod. As an example, the pod specification can instruct the pod to be scheduled by client scheduler 216. Pod specification 222 can instruct whether the pod application should run on a node that supports massive pages. Pod specification 222 can instruct the CPU allocation for applications within the pod. Furthermore, the pod specification can instruct the type of vrouter the pod application is configured for, for example, kernel mode or DPDK mode.

[0070] In some respects, a pod can be matched with a worker node by detecting its type using a custom field for the interface type in "cni-args". For example, cni-args can indicate whether the pod is a DPDK pod (virtio) or a non-DPDK pod (veth by default). Figure 7 A sample DPDK pod specification using the YAML non-markup language (YAML) is shown, which also specifies that oversized pages should be available in worker nodes.

[0071] The API server 214 of master node 202 exposes APIs for interacting with nodes in cluster 201. API server 214 can receive requests from end users, node 204, and external components. Such requests may include requests to deploy, delete, or move pods, query and change the state of objects within the system, communication requests between objects, requests to add or delete resources, etc. API server 214 validates and configures object data. API server 214 may implement a Representation State Transfer (REST) ​​interface to handle REST operations and provide a front-end to the shared state of the corresponding cluster stored in database 218. API server 214 can authenticate and authorize requests. API server 214 communicates with other components of cluster 201 to instantiate virtual execution elements in the cluster. API server 214 may represent a Kubernetes API server.

[0072] Controller Manager 210 manages the controller processes for various components of the Kubernetes platform. A controller process is a process that monitors the state of nodes 204 in cluster 201. In some aspects, a controller process monitors the control loop of the current state of cluster 201 by receiving notifications from API server 214 and, if necessary, performing actions to change the current state of cluster 201 to a desired state. Various controller processes may exist and may monitor and / or control different aspects of cluster 201. Controller Manager 210 may represent the Kubernetes controller manager for the Kubernetes cluster.

[0073] Performance controller 212 implements the techniques described herein to adjust node configuration parameters to maximize node performance based on node profile 220. For example, if a node has a profile indicating that the node should be improved or optimized to obtain maximum PPS, performance controller 212 can adjust the node configuration to maximize PPS. Similarly, if a node has a profile indicating that the node should be improved or optimized to save power, performance controller 212 can adjust the node configuration to save power. Further details regarding the techniques used by performance controller 212 to adjust the configuration of node 204 are provided below.

[0074] The default scheduler 215 manages the scheduling of pods such as pod 224 on node 204. As described above, a pod can include one or more containers, where each container is a self-contained application, including any libraries and configuration settings required to run the application. The default scheduler 215 monitors newly created or requested pods and can select one of the nodes in 204 to receive the pod and run any application within the pod's containers.

[0075] A custom scheduler 216 is a custom scheduler that runs in addition to or in place of the default scheduler 215. A pod 224 can specify the scheduler to use when scheduling the pod by adding an indicator identifying the desired scheduler to its associated pod specification 222. If the pod specification 222 does not specify a scheduler, the default scheduler 215 can be used to schedule the pod on worker nodes. If the pod specification 222 does specify a custom scheduler 216, then the custom scheduler 216 is responsible for scheduling the pod. The custom scheduler 216 can use the techniques described herein to select a worker node 204 for scheduling the pod based on the characteristics of the pod and the characteristics of the available worker nodes. For example, if the pod specification specifies low latency for the container applications within the pod, the custom scheduler 216 can select a worker node 204 that has already been tuned for low latency (e.g., a worker node assigned a low latency profile). Similarly, if a pod specification calls for high throughput for the containerized application within the pod, custom scheduler 216 can select worker nodes 204 that have already been tuned for high throughput (e.g., worker nodes with high throughput profiles). Further details regarding this selection process are described below. In some respects, custom scheduler 216 can refer to a Kubernetes custom scheduler.

[0076] Worker node 204 can host applications, such as applications served by one or more containers of pod 224. Custom scheduler 216 can schedule pods with one or more containers on one or more worker nodes 204. Worker nodes 204A through 204N can include orchestration agents 206A through 206N (collectively referred to as "Orchestration Agent 206") and service agents 208A through 208N (collectively referred to as "Service Agent 208"). In various implementations using Kubernetes, orchestration agent 206 can be a kubelet, and service agent 208 can be a kube agent.

[0077] Service broker 208 is a network broker that runs on worker nodes 204A through 204N. In some respects, service broker 208 provides services as defined in the Kubernetes API on each node and can perform TCP, UDP, and SCTP forwarding. Service broker 208 monitors the addition and removal of services and endpoint objects, and it maintains the network configuration of worker node 204 to ensure communication between pods and containers, for example, by using services to ensure this communication. Service broker 208 can represent the kube broker for worker node 204 of a Kubernetes cluster.

[0078] Orchestration agent 206 is an agent running on each of the nodes 204 in cluster 201. Orchestration agent 206 monitors the status of containers running on its corresponding nodes. Orchestration agent 206 is an orchestrator (e.g., Figure 1 The orchestration agent 206 acts as an agent for the orchestration agent 130, receiving container specification data for containers and ensuring that the containers are executed by the compute device 300. The container specification data can be in the form of a manifest file sent from the orchestration agent 130 to the orchestration agent 206 or received indirectly via a command-line interface, HTTP endpoint, or HTTP server. The container specification data can also be a pod specification 222 (e.g., a Pod specification—a YAML (Yellow Markup Language) or JSON object describing a pod) for a pod (e.g., pod 224). Based on the pod specification, the orchestration agent 206 retrieves and instantiates the container image in pod 224 for execution on worker node 204.

[0079] Performance agent 207 provides node configuration information to performance controller 212 and receives parameter settings and actions from performance controller 212. Performance agent 207 and performance controller 212 communicate with each other via communication channel 215 to exchange information about worker node 204 and adjust the performance of worker node 204. In some examples, communication channel 215 can be implemented using the Kubernetes API, Google Remote Procedure Call (gRPC), etc. Figure 2 As shown. In some respects, performance agent 207 can be a standalone module of worker node 204, such as Figure 2 As shown. In some aspects, performance agent 207 can be a plugin (e.g., a kubelet plugin). In some aspects, performance agent 207 can include another agent or component on worker node 204 (e.g., a vrouter agent). Figure 2 (not shown in the text)

[0080] Performance agent 207 can perform both startup and runtime operations on worker node 204. For example, when worker node 204 is initialized (e.g., after worker node startup or reboot), performance agent 207 can communicate information about worker node 204 to performance controller 212. For example, performance agent 207 can collect information about NUMA configuration (if worker node supports it), number and type of processors, NIC type, memory configuration, disk partitions, power profile, and virtual router mode (e.g., kernel mode or DPDK mode). Performance agent 207 can report the collected information to performance controller 212.

[0081] Performance controller 212 can configure worker nodes using information collected by performance agent 207 along with the profiles of the assigned worker nodes. In some aspects, performance controller 212 can divide the CPUs reported by performance agent 207 into two CPU pools: a reserved pool and a non-reserved pool. In the case of CPUs in the reserved pool, CPUs can be isolated or non-isolated based on profile parameters or virtual router mode. In some aspects, CPUs in the reserved pool are pre-reserved for virtual routers (whether DPDK virtual routers or kernel-mode virtual routers). CPUs reserved for virtual routers are not used for pod workload scheduling. Additionally, specific NUMA nodes can be reserved for vrouter applications based on the data path through the NIC. For example, the data path through the NIC may have management traffic. In this case, performance controller 212 can ignore the NIC for tuning purposes, as management traffic typically does not require performance tuning. In another case, the data path may have data plane traffic through a high-performance NIC with offloading capabilities. In this case, performance controller 212 can consider the NIC when tuning node performance. CPUs can be reserved for kernel functions. In some respects, the "isolcpu" Linux kernel parameter can be used to isolate the CPU.

[0082] CPUs in the non-reserved pool can be used for pod workload scheduling. In some respects, Performance Agent 207 can use the Linux "cpuset" configuration parameter to allocate CPUs to workloads. In other respects, in addition to or instead of cpusett, isolcpu can be used to allocate workloads to CPUs. Workloads that do not require dedicated CPUs can allocate CPUs from the non-reserved pool.

[0083] During the runtime of worker node 204, performance agent 207 can monitor resource usage on its corresponding worker node 244 and enforce performance parameters. Performance agent 207 can provide feedback to performance controller 212 regarding the resource usage of its corresponding worker node 244. Performance controller 212 and / or custom scheduler 216 can use this information to schedule pods on the worker node. As an example, performance controller 212 can select a worker node whose profile matches the features provided in the pod specification, has sufficient available CPU to support the CPU binding parameters (if any) in the pod specification, and has available NUMA memory to support the NUMA parameters (if any) provided in the pod specification. Once assigned to a worker node, performance controller 207 on the worker node can distribute the pod's workload to the CPU and NUMA nodes. As an example, for latency and performance profiles, performance controller 207 can distribute the pod's workload to the same NUMA nodes and sockets as the DPDK vrouter pod.

[0084] Performance agent 207 can implement performance throttling on its corresponding worker nodes. For example, performance agent 207 can implement CPU allocation for container applications within a pod. In various Linux implementations, performance agent 207 can use "cgroups" to implement CPU allocation and other resource allocation. Cgroups are a Linux kernel feature that manages the resource usage (e.g., CPU, memory, disk I / O, network, etc.) of a set of processes. Performance agent 207 can use "isolcpu" (isolated CPU) on DPDK-based worker nodes. Isolcpu removes the CPU from the kernel scheduler. Performance agent 207 can then use processor affinity system calls to allocate container applications to the CPU. In the case of DPDK-based worker nodes, performance agent 207 can allocate DPDK-based virtual routers to CPUs different from those of the VNFs that might use the virtual routers, thereby avoiding processor sharing (and potential CPU usage conflicts) between VNFs and the vrouters they use. Therefore, performance agent 207 can allocate and track which applications are using which CPUs. If performance agent 207 detects that a pod or application is not fulfilling (e.g., not conforming to) the assignment specified by performance controller 212, performance agent 207 may send feedback to performance controller. Performance controller 212 and / or custom scheduler 216 may attempt to resolve the conflict by rescheduling the pod to a different worker node.

[0085] In some respects, users can reassign different profiles to worker node 204 after the initial profile assignment. In this case, workloads on worker node 204 can be migrated to different worker nodes with matching profiles for the workloads. After all workloads have been migrated from worker node 204, performance controller 212 can reconfigure worker node 204 using the new profiles. For example, performance controller 212 can communicate new tuning parameters to performance agent 207 on worker node 204. Performance agent 207 can then set the received parameters on the worker node.

[0086] Now, an operational example of cluster 201 will be provided. System administrators can assign profiles to the worker nodes of cluster 201. For example, consider a cluster with 200 nodes. The system administrator can assign the maximum PPS profile to 25 nodes, the minimum latency profile to 50 nodes, the maximum throughput profile to 75 nodes, and the power saving profile to 50 nodes. Additionally, in some aspects, nodes can be configured to run either kernel-mode virtual routers or DPDK-mode virtual routers. Generally, kernel-mode virtual routers are implemented within the operating system's kernel and therefore run in kernel memory space. DPDK-mode virtual routers run in user memory space. Therefore, DPDK-mode virtual routers can offer improved performance compared to kernel-mode virtual routers because less context switching is required when a DPDK-mode vrouter sends or receives data from other user-mode processes, such as container applications on worker nodes.

[0087] When worker node 204 starts, performance agent 207 can execute the node's introspection process and report hardware and software configuration information to performance controller 212. For example, performance agent 207 can report the number and type of CPUs, NIC type, memory configuration, disk partitions, power profile, NUMA configuration, support for huge pages, support for large pages, etc. Performance controller 212 can determine the values ​​of tuning parameters based on feedback received from the worker node and parameters obtained from the profile assigned to the worker node. Performance controller 212 can send the tuning parameters to performance agent 207 on the worker node. Performance agent 207 can configure the worker node based on the parameters. Additionally, in some aspects, performance agent 207 can tune the worker node configuration based on whether the worker node's vrouter is a kernel-mode vrouter or a DPDK vrouter.

[0088] Custom scheduler 216 can schedule pods to receive pods by selecting worker nodes 204A to 204N. For a pod with pod specifications that indicate to custom scheduler 216 which it should use for scheduling, custom scheduler 216 can select a worker node based on matching the pod characteristics with the worker node's profile and current resource usage. In some aspects, custom scheduler 216 can perform a "best match" selection algorithm; for example, custom scheduler 216 selects a worker node whose configuration and available resources are the best match for the pod specifications. For example, a pod specifying a low-latency application can be assigned to a worker node with parameters tuned for low-latency applications and available CPU, memory, and other resources. Further, the pod can specify that it is configured to use a DPDK virtual router. In this case, the scheduler will select a worker node that supports a DPDK virtual router.

[0089] If no optimal matching criterion option is available, pods can be scheduled based on the maximum number of matches (e.g., matches with CPU affinity, NUMA affinity, isolation, NIC, vrouter mode, etc.). The worker node with the highest number of matches can be selected, and the custom scheduler 216 will schedule the pod accordingly.

[0090] Its pod specification does not specify that a custom scheduler 216 pod can be scheduled by the default scheduler. For example, such a pod can be scheduled by the standard Kubernetes scheduler.

[0091] As described above, performance agent 207 can provide performance controller 212 with dynamic feedback on the operation of its corresponding worker node 204. For example, performance agent 207 can monitor CPU, memory, disk usage, and other resource utilization on the worker node and send this information to performance controller 212. Additionally, performance agent 207 can provide performance controller 212 with information about latency, throughput, interrupts, etc. Performance controller 212 and / or custom scheduler 216 can use this information to schedule pods for future deployments to worker node 204.

[0092] In some respects, data collected from performance agent 207 can be made available to users via dashboards. Users can then use this data to manually assign pods to specific nodes, bypassing the scheduler logic. In this scenario, performance agent 207 can update performance controller 212, ensuring that all resources / data are up-to-date.

[0093] Figure 3 This is a block diagram of an example computing device (e.g., a host) that includes performance agents for adjusting performance parameters and configuring computing devices according to the techniques described in this disclosure. Figure 3 300 computing devices. Figure 3 It can represent a real or virtual server, and can represent Figure 1 Any of the network devices 107 and Figure 2 Example instances of master node 202 and worker node 204. In this example, computing device 300 includes a bus 342 that couples hardware components of the computing device 300 hardware environment. Bus 342 couples network interface card (NIC) 330, storage disk 346, and one or more processors 310 (hereinafter referred to as "processor 310"). NIC 330 may support DPDK. In some cases, the front-side bus may couple processor 310 and memory device 344. In some examples, bus 342 may couple memory device 344, processor 310, and NIC 330. Bus 342 may represent a Peripheral Component Interface (PCI) High Speed ​​(PCIe) bus. In some examples, a Direct Memory Access (DMA) controller may control DMA transfers between components coupled to bus 342. In some examples, components coupled to bus 342 control DMA transfers between components coupled to bus 342.

[0094] Processor 310 may include one or more processors, each processor including an independent execution unit to execute instructions conforming to an instruction set architecture, which are stored in a storage medium. The execution unit may be implemented as a separate integrated circuit (IC) or may be combined within one or more multi-core processors (or “multi-core” processors), each of which is implemented using a single IC (i.e., a chip multiprocessor).

[0095] Disk 346 represents a computer-readable storage medium, which includes volatile and / or non-volatile, removable and / or non-removable media implemented in any method or technology for storing information such as processor-readable instructions, data structures, program modules or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by processor 310.

[0096] Main memory 344 includes one or more computer-readable storage media, which may include random access memory (RAM) such as various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM or static RAM (SRAM)), flash memory, or any other form of fixed or removable storage media, which may be used to carry or store desired program code and program data in the form of instructions or data structures and may be accessible by a computer. Main memory 344 provides a physical address space consisting of addressable memory locations.

[0097] Network interface card (NIC) 330 includes one or more interfaces 332 configured to exchange packets using links of the underlying physical network. Interface 332 may include a port interface card with one or more network ports. NIC 330 may also include on-card memory for, for example, storing packet data. Direct memory access transfers between NIC 330 and other devices coupled to bus 342 can read from / write to NIC memory.

[0098] Memory 344, NIC 330, storage disk 346, and processor 310 provide an operating environment for the software stack including the operating system kernel 314 executing in kernel space. Kernel 314 can represent, for example, Linux, Berkeley Software Distribution (BSD), another Unix variant kernel, or a Windows server operating system kernel available from Microsoft Corp. In some instances, the operating system can execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include Kernel-Based Virtual Machine (KVM) for the Linux kernel, Xen, ESXi available from VMware, Windows Hyper-V available from Microsoft, and other open-source and proprietary hypervisors. The term hypervisor can encompass a Virtual Machine Manager (VMM). The operating system including kernel 314 provides an execution environment for one or more processes in user space 345.

[0099] Kernel 314 includes a physical driver 325 for using network interface card 330. Network interface card 330 can be a physical network interface, and one or more virtual network interfaces use this physical network interface to enable communication such as containers 329A to 229B or one or more virtual machines. Figure 3Network interface card 330 (NIC 330) is shared among one or more virtual execution elements (not shown in the diagram). Shared virtual devices, such as virtualization functions, can provide dedicated resources, allowing each virtual execution element in the virtual execution element to access the dedicated resources of NIC 330, thus making NIC 330 appear as a dedicated NIC for each virtual execution element in the virtual execution element.

[0100] Computing device 300 may (e.g., via NIC 330) be coupled to a physical network switch architecture that includes an overlay network extending the switch architecture from physical switches to "virtual" routers, either software-based or coupled to physical servers within the switch architecture. These "virtual" routers include virtual router 320. The virtual routers may be provided by physical servers (e.g., Figure 1 The network device 107 executes processes or threads or components thereof, which are physical servers that dynamically create and manage one or more virtual networks that can be used for communication between virtual network endpoints. In one example, the virtual router implements each virtual network using an overlay network that provides the ability to decouple the virtual address of an endpoint from the physical address (e.g., IP address) of the server on which the endpoint is executing. Each virtual network can use its own addressing and security scheme and can be considered orthogonal to the physical network and its addressing scheme. Various techniques can be used to transmit packets within a physical network and across virtual networks. As used herein, the term "virtual router" can encompass Open vSwitch (OVS), OVS bridges, Linux bridges, Docker bridges, or other devices and / or software residing on a host device and performing switching, bridging, or routing of packets between virtual network endpoints of one or more virtual networks hosted by one or more network devices 107. Figure 3 In the example computing device 300, the virtual router 320 can be executed in user space as a DPDK-based virtual router, but in various implementations, the virtual router 320 can be executed within a hypervisor, host operating system, host application, or virtual machine.

[0101] Virtual router 320 can be executed as a kernel module (e.g., a kernel-mode virtual router) or as a user-space DPDK procedure (e.g., a DPDK-mode virtual router as shown in user space 345 herein). Virtual router agent 316 can also be executed in user space. Figure 3 In the example computing device 300, the virtual router 320 executes in user space as a DPDK-based virtual router, but in various implementations, the virtual router 320 can execute within a hypervisor, host operating system, host application, or virtual machine.

[0102] Virtual Router Proxy 316 uses channels and SDN controller 132 ( Figure 1 A connection is established, and this channel is used for downloading configuration and forwarding information. Virtual router agent 316 programs this forwarding state to the virtual router data (or "forwarding") plane represented by virtual router 320. Virtual router 320 and virtual router agent 316 can be processes executed by processor 310.

[0103] In a kernel-based deployment of a virtual router 320 (not shown), the virtual router 320 is installed as a kernel module within the operating system. The virtual router 320 registers itself with the TCP / IP stack to receive packets from any of its desired operating system interfaces. Interfaces can be bound interfaces, physical interfaces, tap interfaces (for VMs), veth interfaces (for containers), etc. In this mode, the virtual router 320 relies on the operating system to send and receive packets from different interfaces. The virtual router 320 sends packets via the operating system interface. Additionally, NIC queues (physical or virtual) are handled by the operating system. Packet processing may operate in interrupt mode, which generates interrupts and may result in frequent context switching. At high packet rates, the overhead of frequent interrupts and context switching can overwhelm the operating system and lead to performance degradation.

[0104] In virtual router 320 (e.g.) Figure 3 In the DPDK-based deployment shown, virtual router 320 is installed as a user-space application 345 linked to the DPDK library. This can result in faster performance than kernel-based deployments, especially in the presence of high packet rates. Physical interface 332 is used by the DPDK's polling-mode driver (PMD), rather than by the kernel's interrupt-based driver. The registers of physical interface 332 can be exposed to user space 345 for access by the PMD; physical interface 332 bound in this way is no longer managed or visible to the host operating system, and the DPDK-based virtual router 320 manages physical interface 332. This includes packet polling, packet processing, and packet forwarding. In other words, the user packet processing steps are performed by the virtual router 320's DPDK data plane. Compared to the interrupt-mode of kernel-based virtual routers, this "polling-mode" nature makes packet processing / forwarding on the virtual router 320's DPDK data plane more efficient at higher packet rates. Compared to the kernel-mode virtual router 320, there are relatively fewer interrupts and context switching during packet I / O, and in some cases, interrupts and context switching during packet I / O can be completely avoided.

[0105] Virtual router 320 can implement one or more Virtual Router and Forwarding Instances (VRFs) 322A to 222B for a corresponding virtual network in which virtual router 320 operates as a corresponding tunnel endpoint. Generally, each VRF 322 stores forwarding information for the corresponding virtual network and identifies where data packets are to be forwarded and whether packets are to be encapsulated in a tunneling protocol, such as using a tunnel header that may include one or more headers from different layers of the virtual network protocol stack. Each VRF in VRF 322 may include a network forwarding table that stores routing and forwarding information for the virtual network.

[0106] pod 302A to 202B can represent Figure 1 pod 122A and / or Figure 2 Example instance of pod 224. pod 302A includes one or more containers 329A, while pod 302B includes one or more containers 329B.

[0107] A 304 error for a container platform can be described in more detail. Figure 1 An example instance of container platform 119A. Container platform 304 may include container engine 308, orchestration agent 206, service agent 208, and network module 306. Network module 306 may represent Figure 1 Example instance of network module 117A.

[0108] The orchestration agent 206 and service agent 208 may include code that can be executed by the processor 310 as one or more processes. (See above for reference.) Figure 2 Aspects of orchestration agent 206 and service agent 208 are described.

[0109] Container engine 308 includes code executable by processor 310. Container engine 308 can be one or more computer processes. Container engine 308 runs containerized applications in the form of containers 329A to 329B. Container engine 308 can represent CRI-O, containerized Docker, rkt, or other container engines used to manage containers. Generally, container engine 308 receives requests and manages objects such as images, containers, networks, and volumes. An image is a template with instructions for creating containers. A container is an executable instance of an image. Based on instructions from orchestration agent 206, container engine 308 can obtain images and instantiate them into executable containers 329A to 329B in pods 302A to 202B.

[0110] Orchestration agent 206 instantiates network module 306 to configure one or more virtual network interfaces for each pod 302 in pod 302. For example, orchestration agent 206 receives container specification data for pod 302A and instructs container engine 308 to create pod 302A with container 329A based on the container specification data for pod 302A. Orchestration agent 206 also invokes network module 306 to configure virtual network interface 312 for pod 302A for the virtual network corresponding to VRF 322A. In this example, pod 302A and pod 302B are virtual network endpoints of the virtual networks corresponding to VRF 322A and VRF 322B. Any of the virtual network interfaces 312 and 313 can represent Figure 1 Example instance of virtual network interface 126 in the virtual network interface 126 described herein.

[0111] Network module 306 can obtain interface configuration data for configuring the virtual network interface of pod 302. Virtual router proxy 316 operates as a virtual network control plane module, enabling SDN controller 132 ( Figure 1 The virtual network control plane (including network controller 132 and virtual router agents 316 for worker nodes) manages the configuration of virtual networks implemented in the data plane by the virtual routers 320 of worker nodes, unlike the orchestration control plane (including container platform 304, such as orchestrator 130, for worker nodes and multiple master nodes) which manages provisioning, scheduling, and management of virtual execution elements. The virtual router agent 316 communicates interface configuration data for virtual network interfaces to network module 306, enabling orchestration control plane elements (i.e., network module 306) to configure virtual network interfaces based on the configuration state determined by SDN controller 132, thereby bridging the gap between the orchestration control plane and the virtual network control plane. Additionally, this allows network module 306 to obtain interface configuration data for multiple virtual network interfaces for pods and configure multiple virtual network interfaces, thereby reducing the communication and resource overhead inherent in calling a separate network module 306 to configure each virtual network interface.

[0112] Figure 4 This is a block diagram of an example computing device operation that includes a performance controller for adjusting the performance parameters of the working nodes. Figure 4 The computing device 400 can represent the network device 107 ( Figure 1 ) and master node 202 ( Figure 2 One or more of the following.

[0113] In this example, computing device 400 includes a bus 442 that couples hardware components of the computing device 400 hardware environment. Bus 442 couples a network interface card (NIC) 430, a storage disk 446, and one or more processors 410 (hereinafter referred to as "processor 410"). In some cases, a front-side bus may couple processor 410 and memory device 444. In some examples, bus 442 may couple memory device 444, processor 410, and NIC 430. Bus 442 may represent a Peripheral Component Interface (PCI) High Speed ​​(PCIe) bus. In some examples, a Direct Memory Access (DMA) controller may control DMA transfers between components coupled to bus 442. In some examples, components coupled to bus 442 control DMA transfers between components coupled to bus 442.

[0114] Processor 410 may include one or more processors, each processor including an independent execution unit to execute instructions conforming to an instruction set architecture, which are stored in a storage medium. The execution units may be implemented as individual integrated circuits (ICs) or may be combined within one or more multi-core processors (or “multi-core” processors), each of which is implemented using a single IC (i.e., a chip multiprocessor).

[0115] Disk 446 represents a computer-readable storage medium, which includes volatile and / or non-volatile, removable and / or non-removable media implemented in any method or technology for storing information such as processor-readable instructions, data structures, program modules or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by processor 410.

[0116] Main memory 444 includes one or more computer-readable storage media, which may include random access memory (RAM) such as various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM or static RAM (SRAM)), flash memory, or any other form of fixed or removable storage media, which may be used to carry or store desired program code and program data in the form of instructions or data structures and are accessible by a computer. Main memory 444 provides a physical address space consisting of addressable memory locations.

[0117] Network interface card (NIC) 430 includes one or more interfaces 432 configured to exchange packets using links of the underlying physical network. Interface 432 may include a port interface card with one or more network ports. NIC 430 may also include on-card memory for, for example, storing packet data. Direct memory access transfers between NIC 430 and other devices coupled to bus 442 can read from / write to NIC memory.

[0118] Memory 444, NIC 430, storage disk 446, and processor 410 provide an operating environment for the software stack of the operating system kernel 414, which executes in kernel space. Kernel 414 may represent, for example, Linux, Berkeley Software Distribution (BSD), another Unix variant kernel, or a Windows server operating system kernel available from Microsoft Corp. In some instances, the operating system may execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include Kernel-Based Virtual Machine (KVM) for the Linux kernel, Xen, ESXi available from VMware, Windows Hyper-V available from Microsoft, and other open-source and proprietary hypervisors. The term hypervisor may encompass a Virtual Machine Manager (VMM). The operating system including kernel 414 provides an execution environment for one or more processes in user space 445. Kernel 414 includes a physical drive 425 that provides an interface using network interface card 430 for use by components of computing device 400.

[0119] The computing device 400 can be coupled to a physical network switch architecture (e.g., via NIC 430), which includes an overlay network that extends the switch architecture from the physical switches to software or "virtual" routers coupled to the physical servers of the switch architecture. Figure 3 This virtual router 320. The computing device 400 can use one or more dedicated virtual networks to configure the cluster's worker nodes 204. Figure 2 ).

[0120] Computing device 400 may include an API server 214, a default scheduler 215, a custom scheduler 216, and a controller manager 210, each including code executable by the processor as one or more computer processes. (See above for reference.) Figure 2The API server 214, the default scheduler 215, the custom scheduler 216, and the controller manager 210 are described. Generally, the API server 214 can invoke either the default scheduler 215 or the custom scheduler 216 to schedule worker nodes 204. Figure 2 The virtual execution element is located on the pod. If the pod specification specifies that a custom scheduler 216 should be used to schedule the pod, the custom scheduler 216 can select worker nodes to schedule the virtual execution element according to the techniques described herein. For example, the custom scheduler 216 can select worker nodes based on the best match between node profiles and tuning parameters and virtual element characteristics (e.g., pod characteristics in the pod specification). The API server 214 can invoke the orchestration agent 206 of the selected worker node (…). Figure 2 and Figure 3 This allows the container engine 308 of the selected worker node to... Figure 3 The virtual execution element is retrieved from the storage server and created on the worker node. The orchestration agent 206 of the selected worker node can update the state of the virtual execution element to the API server 214, which persists the new state to the database 218. In this way, the computing device 400 instantiates the new virtual execution element in the computing infrastructure 8.

[0121] SDN controller 132 includes code executable by processor 410. SDN controller 132 may include one or more computer processes. SDN controller 132 may be a logically centralized but physically distributed controller responsible for providing management, control, and analysis functions for the virtualized network. In particular, SDN controller 132 may be a logically centralized control plane and management plane of computing infrastructure 8, and orchestrate virtual routers for one or more worker nodes.

[0122] The SDN controller 132 can provide cloud networking for computing architectures running on network infrastructure. Cloud networks may include private clouds of enterprises or service providers, Infrastructure as a Service (IaaS), and Virtual Private Clouds (VPCs) of cloud service providers (CSPs). Private cloud, VPC, and IaaS use cases may involve multi-tenant virtualized data centers, such as those related to… Figure 1 This describes a multi-tenant virtualized data center. In this scenario, multiple tenants in the data center share the same physical resources (physical servers, physical storage devices, physical networks). Each tenant is allocated its own logical resources (virtual machines, containers, or other forms of virtual execution elements; virtual storage devices; virtual networks). These logical resources are isolated from each other unless specifically permitted by security policies. The data center's virtual networks can also interconnect with physical IP VPNs or L2 VPNs.

[0123] The SDN controller 132 can provide network function virtualization (NFV) to networks such as business edge networks, broadband user management edge networks, and mobile edge networks. NFV involves the orchestration and management of network functions, such as virtual machines, containers, or other virtual execution elements rather than physical hardware facilities, including firewalls, intrusion detection or prevention systems (IDS / IPS), deep packet inspection (DPI), caching, wide area network (WAN) optimization, etc.

[0124] SDN controller 132 programs network infrastructure elements to create virtual networks and can create interface configurations for virtual network interfaces of virtual networks. Additional information regarding the operation of SDN controller 132 with other devices in data center 101 or other software-defined networks can be found in International Application No. PCT / US2013 / 044378 and U.S. Patent Application No. 14 / 226,509, which are incorporated above by reference.

[0125] Network controller manager 427 includes code executable by processor 410. Network controller manager 427 can be one or more computer processes. Network controller manager 427 acts as an interface between orchestration-oriented elements (e.g., default scheduler 215, custom scheduler 216, API server 214, controller manager 426, and database 218) and SDN controller 132. Generally, network controller manager 427 monitors the cluster for new objects (e.g., pods and services). Network controller manager 427 can isolate pods in virtual networks and connect pods to services. Network controller manager 427 can execute as a container of the cluster's master node.

[0126] Although the default scheduler 215, custom scheduler 216, API server 214, network controller manager 426, SDN controller 132, network controller manager 427, and database 218 are described and illustrated as being executed by a single computing device 400, they can be distributed among multiple computing devices 400 that make up a computing system or hardware / server cluster. In other words, each of the multiple computing devices 400 can provide a hardware operating environment for one or more instances of any one or more of the default scheduler 215, custom scheduler 216, API server 214, network controller manager 426, SDN controller 132, network controller manager 427, or database 218. The default scheduler 215, custom scheduler 216, API server 214, controller manager 210, and network controller manager 427 can implement example instances of orchestrator 132. Network controller manager 427 can represent a Kubernetes cloud controller manager or a Kube manager.

[0127] Figure 5 This is a block diagram illustrating an example cluster adapted according to the techniques described in this document. Figure 5 In the illustrated example, cluster 500 includes a master node 502 running performance controller 212. Cluster 500 also includes worker nodes 504A and 504B, each with a different hardware configuration. In this example, worker node 504A is a 32-core dual-socket computing system, while worker node 504B is a 40-core dual-socket computing system. Each worker node in worker nodes 504A and 504B has a corresponding performance agent 207A and 207B.

[0128] In this example, performance controller 212 schedules pods requiring four cores and pods requiring six cores on specific nodes to achieve expected CPU utilization. Furthermore, CPU can be allocated to pods, such as assigning pods to the same NUMA node as the vrouterpod for maximum performance. Figure 5 In the example shown, VNFs 508A and 508B of worker node 504A are DPDK VNFs and have therefore been assigned to the same NUMA node as vrouter 506A (e.g., NUMA 0). Pod1-N 510A is not configured for DPDK and can therefore be assigned to a different NUMA node (e.g., NUMA 1). Similarly, VNFs 508C and 508D utilize DPDK and have therefore been assigned to NUMA 0, i.e., the same NUMA node as vrouter 506B. Pod1-N 510B is not configured to use DPDK and is therefore assigned to NUMA 1. Non-DPDK applications or kernels can be assigned from socket 2 and NUMA 2, and interrupts can be tuned to achieve optimal performance for such Pods. As an example of interrupt tuning, performance controller 212 can disable irqbalance to avoid general load balancing of interrupts. Performance controller 212 can also disable irqbalance for specific CPUs. As another example, performance controller 212 can load balance interrupts and / or map interrupts to specific queues for packet processing based on CPU affinity. For example, performance controller 212 can specify that interrupts on the NIC on CPU socket 0 are handled only by the CPU that is also on socket 0.

[0129] Figure 6This is a flowchart illustrating example operations according to one or more techniques disclosed herein. The master node of a container-based orchestration system can receive multiple node characteristics from each agent of multiple worker nodes in a cluster, the multiple node characteristics being associated with a worker node corresponding to the agent (605). The master node can manage the orchestration of workloads on the worker nodes of the cluster. Furthermore, the master node can determine one or more parameters for each of the multiple worker nodes, corresponding to the multiple node characteristics associated with the corresponding worker node and its node profile, and provide one or more parameters to the corresponding agent executing on the worker node (610). The corresponding agent configures the worker node according to the one or more parameters. Next, the master node can receive a request to deploy a pod to a worker node, the pod having one or more pod characteristics (615). Next, the master node can select a chosen worker node from the multiple worker nodes to receive the pod based on the multiple node characteristics and the one or more pod characteristics (620). Next, the master node can schedule the pod on the selected worker node (625).

[0130] In some respects, the technologies disclosed herein have already been discussed in the context of SDN. However, these technologies are not limited to SDN and can be implemented in many different environments, including container-based orchestration systems such as the Kubernetes platform.

[0131] For any processes, apparatuses, and other examples or illustrations included in any flowchart or diagram herein, certain operations, actions, steps, or events in any of the techniques described herein may be performed in a different order, may be added, combined, or omitted entirely (e.g., not all described actions or events are necessary for technical practice). Furthermore, in some examples, operations, actions, steps, or events may be performed simultaneously, for example, through multithreading, interrupt handling, or multiple processors rather than sequentially. Certain other operations, actions, steps, or events may be performed automatically, even if not specifically identified as automatically performed. Additionally, certain operations, actions, steps, or events described as automatically performed may alternatively not be automatically performed, but rather, in some examples, these operations, actions, steps, or events may be performed in response to input or another event.

[0132] The accompanying drawings included herein each illustrate at least one exemplary implementation of one aspect of this disclosure. However, the scope of this disclosure is not limited to such implementations. Thus, other examples or alternative implementations of the systems, methods, or techniques described herein may be appropriate in other instances, in addition to the example or alternative implementations shown in the figures. Such implementations may include a subset of the devices and / or components included in the figures and / or may include additional devices and / or components not shown in the figures.

[0133] The specific embodiments described above are intended as descriptions of various configurations and are not intended to represent the only configuration in which the concepts described herein can be practiced. The specific embodiments include concrete details to provide a full understanding of the various concepts. However, these concepts can be practiced without these concrete details. In some instances, well-known structures and components are shown in block diagram form in the reference figures to avoid obscuring these concepts.

[0134] Therefore, although one or more implementations of various systems, devices, and / or components may be described with reference to specific figures, such systems, devices, and / or components may be implemented in several different ways. For example, the figures herein (e.g., Figure 1 and / or Figure 3 One or more devices illustrated as individual devices in this document may alternatively be implemented as a single device; one or more components illustrated as a single component may alternatively be implemented as a single component. Furthermore, in some examples, one or more devices illustrated as a single device in the accompanying drawings may alternatively be implemented as multiple devices; one or more components illustrated as a single component may alternatively be implemented as multiple components. Each of such multiple devices and / or components may be directly coupled via wired or wireless communication and / or remotely coupled via one or more networks. Additionally, one or more devices or components illustrated in the various accompanying drawings may alternatively be implemented as part of another device or component not shown in these drawings. In this and other ways, some of the functions described herein can be performed by two or more devices or components via distributed processing.

[0135] Furthermore, certain operations, techniques, features, and / or functions may be described herein as being performed by a specific component, device, and / or module. In other examples, such operations, techniques, features, and / or functions may be performed by different components, devices, or modules. Thus, in other examples, some operations, techniques, features, and / or functions that may be described herein as belonging to one or more components, devices, or modules may belong to other components, devices, and / or modules, even if not specifically described in this way herein.

[0136] While specific advantages have been identified in conjunction with descriptions of some examples, various other examples may include, exclude, or encompass all of the listed advantages. Other techniques or other advantages may become apparent to those skilled in the art based on this disclosure. Furthermore, although specific examples have been disclosed herein, aspects of this disclosure can be implemented using any number of techniques, whether currently known or not, and therefore, this disclosure is not limited to the examples specifically described and / or illustrated herein.

[0137] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on and / or transmitted via a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium or a communication medium including any medium that facilitates the transfer of a computer program from one place to another (e.g., according to a communication protocol). In this way, a computer-readable medium may generally correspond to (1) a tangible computer-readable storage medium that is a non-transient tangible computer-readable storage medium; or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0138] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave technologies, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave technologies are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient tangible storage media. The disks and discs used include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while discs use lasers to reproduce data optically. Combinations of the above should also be included within the scope of computer-readable media.

[0139] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose processors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Therefore, the terms "processor" or "processing circuit system" as used herein can each refer to any of the foregoing structures or any other structure suitable for implementing the described techniques. Additionally, in some examples, the described functionality can be provided within dedicated hardware and / or software modules. Furthermore, these techniques can be fully implemented in one or more circuit or logic elements.

[0140] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless handheld devices, mobile computing devices or non-mobile computing devices, wearable computing devices or non-wearable computing devices, integrated circuits (ICs) or IC sets (e.g., chip sets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but they do not necessarily need to be implemented by different hardware units. Rather, as described above, various units may be combined in hardware units or provided by an interoperable hardware unit set including one or more processors as described above, along with suitable software and / or firmware.

Claims

1. A method for orchestration in a container-based system, the method comprising: The master node of the cluster receives, from each agent executing on a corresponding worker node among a plurality of worker nodes in the cluster, a plurality of node characteristics associated with the corresponding worker node, wherein the cluster is a container-based orchestration system cluster, and wherein the master node manages the orchestration of workloads on the worker nodes of the cluster. For each of the plurality of worker nodes, the master node determines one or more parameters corresponding to the plurality of node characteristics associated with the corresponding worker node and the node profile of the corresponding worker node, and the master node provides the one or more parameters to the corresponding agent executing on the worker node, wherein the corresponding agent configures the worker node according to the one or more parameters; The master node receives requests to deploy pods with one or more container group pod characteristics; The master node selects a selected worker node from the plurality of worker nodes based on the plurality of node characteristics and the one or more pod characteristics, wherein the pod is to be deployed on the selected worker node; as well as The master node schedules the pod on the selected worker node.

2. The method according to claim 1, wherein selecting the selected working node comprises: Select a worker node configured with a virtual router, which operates in a virtual router mode indicated by the one or more pod features.

3. The method according to claim 2, wherein the virtual router mode includes the Data Plane Development Kit (DPDK) mode.

4. The method according to any one of claims 1 to 3, wherein the one or more parameters include one or more of the following: CPU pinning parameters, NUMA (Non-Uniform Memory Access) binding parameters, and virtual router configuration parameters.

5. The method according to any one of claims 1 to 3, further comprising: The master node provides one or more CPU allocations for the pod to the corresponding agent of the selected worker node, the CPU allocations being based at least in part on one or more node characteristics of the selected worker node.

6. The method according to any one of claims 1 to 3, wherein selecting the selected worker node to receive the pod comprises: The selected worker node is chosen based on the best match between the multiple node characteristics and the one or more pod characteristics.

7. The method according to any one of claims 1 to 3, further comprising: After the worker node is initialized, the master node receives resource usage information associated with the worker node from the corresponding agent. Selecting the selected worker node to receive the pod includes: selecting the worker node to receive the pod based on the resource usage information.

8. The method according to claim 7, further comprising: The master node determines the number of CPUs currently available on the worker node based on the resource usage information; The desired number of CPUs for a given pod is determined based on the characteristics of the one or more pods. as well as The selected worker node is selected based on the currently available CPUs on the selected worker node and the desired number of CPUs for the pod.

9. The method according to claim 8, further comprising: The master node determines the first pool of reserved CPUs and the second pool of non-reserved CPUs based on the characteristics of the multiple nodes. The currently available CPUs include available CPUs from the first pool of reserved CPUs.

10. The method according to any one of claims 1 to 3, wherein the node profile includes a first node profile, and wherein the working node includes a first working node, wherein the method further comprises: The master node receives, via a user interface device, an instruction for a profile change of the working node, from the first node profile to the second node profile; In response to receiving the instruction that the profile has been changed, the master node migrates the workload from the first working node to a second working node associated with the first node profile. as well as After migrating the workload, the master node configures the first worker node according to the second node profile.

11. A master node, comprising: Multiple processors; A memory coupled to the plurality of processors, wherein the memory stores instructions that, when executed by the plurality of processors, cause the master node to perform the following operations: Each agent executing on a corresponding worker node in a cluster receives multiple node characteristics associated with the corresponding worker node, wherein the cluster is a container-based orchestration system cluster, and wherein the master node manages the orchestration of workloads on the worker nodes of the cluster. For each of the plurality of working nodes, one or more parameters corresponding to the plurality of node characteristics associated with the corresponding working node and the node profile of the corresponding working node are determined, and the one or more parameters are provided to a corresponding agent executing on the working node, wherein the corresponding agent configures the working node according to the one or more parameters; Receive requests to deploy pods with one or more container group pod characteristics; A selected worker node is chosen from the plurality of worker nodes based on the characteristics of the plurality of nodes and the characteristics of the one or more pods, wherein the pod is to be deployed on the selected worker node; and Schedule the pod on the selected worker node.

12. The master node of claim 11, wherein the instructions for selecting a selected worker node include instructions for: Select a worker node configured with a virtual router, which operates in a virtual router mode indicated by the one or more pod features.

13. The master node according to claim 11 or 12, wherein the instructions further include instructions for: Provide one or more CPU allocations for the pod to the corresponding agent of the selected worker node, the CPU allocations being based at least in part on one or more node characteristics of the selected worker node.

14. The master node according to claim 11 or 12, wherein the instructions further include instructions for: After the worker node is initialized, the master node receives resource usage information associated with the worker node from the corresponding agent. And the instructions for selecting the chosen worker node to receive the pod include instructions for the following: The worker node is selected to receive the pod based on the resource usage information.

15. A container-based orchestration system, comprising: Master node; as well as Multiple worker nodes, communicatively coupled to a master node, wherein the master node manages the orchestration of workloads on each of the worker nodes. The master node includes a memory for storing instructions that, when executed by the master node's processing circuitry, cause the master node to: Each agent executing on a corresponding worker node in the cluster receives multiple node characteristics associated with that corresponding worker node; For each of the plurality of working nodes, determine one or more parameters corresponding to the plurality of node characteristics associated with the corresponding working node and the node profile of the corresponding working node; And provide the one or more parameters to the corresponding agent executing on the worker node, wherein the corresponding agent configures the worker node according to the one or more parameters; Receive requests to deploy pods with one or more pod characteristics; Based on the characteristics of the plurality of nodes and the characteristics of the one or more pods, a selected worker node is chosen from the plurality of worker nodes to receive the pod; as well as Schedule the pod on the selected worker node, and The corresponding agent for the selected worker node is configured as follows: Receive an indication that the pod has been scheduled to the selected worker node; and Configure the selected worker node to perform the workload of the pod according to one or more parameters.

16. The container-based orchestration system of claim 15, wherein the selected working node further comprises: Virtual router The corresponding agent of the selected working node is further configured as follows: In response to determining one or more parameters indicating that the virtual router is configured as a DPDK virtual router, the workload is configured to be executed on the NUMA node of the selected worker node with the DPDK virtual router.

17. The container-based orchestration system of claim 16, wherein the virtual router executes on a first subset of the plurality of processors of the selected worker node, wherein the corresponding agent is further configured to execute the workload on a second subset of the plurality of processors, wherein the first subset has no processors shared with the second subset.

18. The container-based orchestration system according to any one of claims 15 to 17, The corresponding agent is also configured to provide resource usage information associated with the selected worker node; and The master node selects the worker node to receive the pod based on the resource usage information.

19. The container-based orchestration system of claim 18, wherein the master node is further configured to: The number of CPUs currently available on the selected working node is determined based on the resource usage information. The desired number of CPUs for the pod is determined based on the one or more pod characteristics; and The selected worker node is selected based on the number of currently available CPUs on the selected worker node and the desired number of CPUs for the pod.

20. The container-based orchestration system according to any one of claims 15 to 17, wherein the worker node includes a first worker node, and wherein the corresponding agent is further configured to: In response to determining that the workload of the first worker node does not conform to the NUMA node allocation, the non-compliance is reported to the master node, wherein in response to receiving the non-compliance report, the master node reschedules the pod with the workload to a second worker node with the same node profile as the first worker node.

21. A computer-readable storage medium encoded with instructions for causing one or more programmable processors to perform the method according to any one of claims 1 to 10.

Images