Automated lifecycle management with flexible scaling and dynamic resource allocation for virtualized cable data plane applications.

Abstract

SYSTEM AND METHOD FOR SUPPORTING FLEXIBLE SCALING AND DYNAMIC RESOURCE ALLOCATION FOR VIRTUALIZED CABLE DATA PLANE APPLICATIONS The system includes a head end operating with nodes to provide data to customer devices. The head end includes at least one server running a container that includes a data plane application that provides data packets to the nodes for transmission. The data plane application is instantiated in at least one of a virtual networking function and a computing resource function.

Description

Background Art

[0001] Cross - reference to Related Applications This application claims the benefit of U.S. Provisional Patent Application No. 62 / 939,832, filed on November 25, 2019.

[0002] The subject matter of this application relates to systems and methods for supporting lifecycle management for virtualized cable data plane applications.

[0003] Cable television (CATV) services generally provide content from a central distribution unit, generally referred to as a "head - end", to a large number of customers (e.g., subscribers). The head - end distributes content channels to customers from this central distribution unit through an access network that includes a hybrid fiber - coaxial (HFC) cable plant, including associated components (nodes, amplifiers, and taps). However, modern cable television (CATV) service networks not only provide media content such as television channels and music channels to customers, but also host, for example, Internet services, telephone services such as video - on - demand, VoIP, and digital communication services such as home automation / security. These digital communication services not only require communication to subscribers in a downstream direction from the head - end, typically through HFC while typically forming a branch network, but also require communication from subscribers to the head - end in an upstream direction, typically through the HFC network.

[0004] For this purpose, CATV headends traditionally included a separate Cable Modem Termination System (CMTS) used to deliver high-speed data services such as Cable Internet and Voice over Internet Protocol to cable television customers and video headend systems, and to deliver video services such as broadcast video and video on demand (VOD). Typically, a CMTS includes both Ethernet® interfaces (or other more traditional high-speed data interfaces) and radio frequency (RF) interfaces, so that traffic coming from the internet can be routed (or bridged) through the Ethernet interface, through the CMTS, and then to an optical RF interface connected to the cable company's hybrid fiber coaxial (HFC) system. Downstream traffic is delivered from the CMTS to the customer's home cable modem and / or set-top box, while upstream traffic is delivered from the customer's home cable modem and / or set-top box to the CMTS. The video headend system similarly delivers video to either a set-top, a TV with a video decoding card, or other devices that can demodulate and decode the encrypted video services it receives. Many modern CATV systems combine CMTS functionality and video distribution systems (EdgeQAM - quadrature amplitude modulation) into a single platform called the Integrated Cable Access Platform (CCAP).

[0005] Furthermore, many modern architectures relocate the physical layer (PHY) of a traditional CMTS or CCAP to the network's fiber nodes (referred to as remote PHY or R-PHY architectures). Thus, while the core performs upper-layer processing in the CMTS / CCAP, the R-PHY devices at the remote nodes convert downstream data transmitted from the core from digital to analog and transmit it at radio frequencies to cable modems and / or set-top boxes, and convert upstream radio frequency data transmitted from cable modems and / or set-top boxes from analog to digital format and transmit it optically to the core. Additionally, other modern CATV systems relocate the control layer or MAC layer to the fiber nodes (referred to as R-MACPHY architectures), or relocate other components to the nodes. Such architectures are generally referred to as distributed access architectures (DAA), distributed CMTS (D-CMTS), etc., in contrast to integrated architectures (e.g., I-CMTS) where all physical and control layers are located at the headend. For simplicity, this disclosure later describes and explains an “I-CMTS” architecture in which all CMTS functions are located at the headend; however, those skilled in the art will understand that in a system containing CCAP, such description includes an integrated CCAP architecture in which all CCAP functions are located at the headend. Similarly, this disclosure later describes and explains a D-CMTS architecture in which the physical parts of the CMTS are pushed to the nodes; however, those skilled in the art will understand that such description includes distributed CCAP functions as well as other distributed architectures such as R-MACPHY, as well as where the system uses CCAP.

[0006] Unfortunately, partial system virtualization tends to pose problems for real-time applications. Therefore, improved systems and methods for partial system virtualization are desirable. [Brief explanation of the drawing]

[0007] To better understand the present invention and to show how it can be carried out, the following accompanying drawings are referenced here as examples.

[0008] [Figure 1] An example of an integrated cable modem termination system is provided. [Figure 2] A distributed cable modem termination system is given as an example. [Figure 3] A layered network stack is illustrated as an example. [Figure 4] An example of a server system having a resource allocation manager and a container orchestration system is provided. [Figure 5] An example of a server system having containers and a container orchestration system is provided. [Figure 6] A resource pool is given as an example. [Figure 7A] This example illustrates the initialization of a vCore instance. [Figure 7B] This example illustrates the initialization of a vCore instance. [Modes for carrying out the invention]

[0009] Referring to Figure 1, the integrated CMTS system 100 may include data 110 that is transmitted to and received by the integrated CMTS 130 (or integrated CCAP) via the Internet (or other network), typically in the form of packetized data. The integrated CMTS / CCAP 130 may also receive downstream video 120, typically in the form of packetized data from an operator video aggregation system. For example, broadcast video is typically acquired from a satellite delivery system and preprocessed for delivery to subscribers via either a CCAP or QAM system coexisting with the CMTS and headend. Also, for example, internet-based video (e.g., YouTube®) is typically delivered to the CMTS via a common internet data pipeline. The integrated CMTS system 100 receives and processes the received data 110 and downstream video 120. The CMTS130 (or CCAP) integrates cable modem termination, switching, routing, and QAM functions at the headend, thereby enabling processing of all data, video, and audio functions over IP before conversion to RF or optical signals. The CMTS130 may transmit downstream data 140 and downstream video 150 to the customer's cable modem and / or set-top box 160 over the network, which may include other devices such as amplifiers and splitters. The CMTS130 may receive upstream data 170 from the customer's cable modem and / or set-top box 160 over the network, which may include other devices such as amplifiers and splitters. The CMTS130 may include multiple devices to achieve its desired capabilities.

[0010] Referring to Figure 2, as a result of increasing bandwidth demands, limited facility space for an integrated CMTS, and power consumption considerations, it is desirable to include a D-CMTS system 200 (e.g., a Distributed Integrated Cable Access Platform (CCAP)). The D-CMTS system 200 distributes some of the downstream functions of the I-CMTS system 100 to remote locations such as fiber nodes using network packetized data. An exemplary D-CMTS system 200 may include a remote PHY architecture, where the remote PHY (R-PHY) is preferably an optical node device located at the fiber-coaxial junction. Generally, the R-PHY often includes the MAC layer and PHY layer of a part of the system. The D-CMTS system 200 may also include a D-CMTS core 230, which contains data 210 that is typically transmitted and received over the Internet (or other network) in the form of packetized data. The D-CMTS core 230 may also receive downstream video 220, which is typically received in the form of packetized data from an operator video aggregation system. The D-CMTS core 230 receives and processes the received data 210 and downstream video 220. The remote fiber node 280 preferably includes a remote PHY device 290. The remote PHY device 290 may transmit the downstream data 240 and downstream video 250 to the customer's cable modem and / or set-top box 260 via a network which may include other devices such as amplifiers and splitters. The remote PHY device 290 may also receive upstream data 270 from the customer's cable modem and / or set-top box 260 via a network which may include other devices such as amplifiers and splitters. The remote PHY device 290 may include multiple devices to achieve its desired capabilities.The remote PHY device 290 primarily includes PHY-related circuitry such as downstream QAM modulators and upstream QAM demodulators, along with pseudowire logic, and connects to the D-CMTS 230 using network-packetized data. The remote PHY device 290 and D-CMTS 230 may include data and / or video interconnects such as downstream data, downstream video, and upstream data 295. In some embodiments, video traffic may be sent directly to the remote physical device, thereby bypassing the RPHY Core. Depending on the circumstances, remote PHY and / or remote MAC PHY functionality may be provided to the headend.

[0011] As an example, the remote PHY device 290 may convert downstream DOCSIS (i.e., Data Over Cable Service Interface Specification) data (each of which is incorporated herein by reference in its entirety, e.g., DOCSIS 1.0, 1.1, 2.0, 3.0, 3.1, and 4.0), video data, and out-of-band signals received from the D-CMTS 230 into analog for transmission over RF or linear optics. As an example, the remote PHY device 290 may convert upstream DOCSIS and out-of-band signals received from analog media such as RF or linear optics into digital for transmission to the D-CMTS 230. As observed, depending on the particular configuration, the R-PHY may move all or part of the DOCSIS MAC and / or PHY layer to the fiber node.

[0012] An I-CMTS device is typically a custom-built hardware device consisting of a single chassis with a series of slots, each receiving its own line card containing a processor, memory, and other computing and networking capabilities supported on it. Each line card contains the same hardware configuration, processing power, and software. Each line card performs the functions of the I-CMTS device, including MAC and PHY functions. As the system expands to support additional customers, additional line cards are included in the system to expand the system's processing power. Unfortunately, dynamically scaling the number of line cards in real time to meet the demands of a particular network is problematic.

[0013] The computing power of microprocessor-based common off-the-shelf (COTS) server platforms is increasing, while the cost of such systems is decreasing over time. Using such systems, computing systems may be virtualized and operated using one or more COTS servers, if desired, generally referred to herein as virtual machines. Using container technology running on COTS servers and / or virtual machines, a COTS server may run on only a single operating system. Each virtualized application may then be isolated using software containers, so that a virtualized application may not see or be aware of other virtualized applications running on the same machine. Typically, each COTS server includes one or more Intel / AMD processors (or other processing devices) with associated memory and networking capabilities to run operating system software. Typically, a COTS server includes a framework and an operating system, where user applications run on such a framework, and the operating system is abstracted away from the actual operating system. Each virtual machine may be instantiated and operated as one or more software applications running on a COTS server. Multiple software containers may be instantiated and run on the same COTS server and / or the same virtual machine. Multiple COTS servers are typically located in one or more data centers, each communicating with the others. Multiple COTS servers may be located in different geographical regions to provide geographical redundancy. In some embodiments, containers may contain the same functionality as virtual machines, or vice versa. In some embodiments, a group of containerized components, commonly referred to as a pod, may take the form of a virtual machine.

[0014] In some embodiments, the COTS server may typically be a “bare metal” server containing an operating system along with drivers and a portion of a container orchestration system. One or more containers are then added to the “bare metal” server while being managed by the container orchestration system. The container orchestration system described herein may also, as desired, be run as a virtual machine orchestration system or be referred to as such. In some embodiments, the “bare metal” server may be used as a pod running on the operating system along with drivers and the container orchestration system. In some embodiments, the virtual machine may be omitted from the COTS server.

[0015] Selected software processes contained within line cards and / or remote PHY devices may include software containers and run on COTS servers, and may also run on "bare metal" servers and / or virtual machines, including both "active" and "backup" software processes. The functionality provided by such "bare metal" servers and / or virtual machines may include higher-level functionalities such as packet processing including routing internet packet provisioning, Layer 2 virtual private networking operating over pseudowires, and multiprotocol label switching routing. The functionality provided by such "bare metal" servers and / or virtual machines may include DOCSIS functions such as DOCSIS MAC and encapsulation, channel provisioning, service flow management, quality of service and rate limiting, scheduling, and encryption. The functionality provided by such "bare metal" servers and / or virtual machines may include video processing such as EQAM and MPEG processing.

[0016] Each of the COTS servers and / or virtual machines and / or software containers may include different hardware profiles and / or frameworks. For example, each of the COTS servers and / or “bare metal” servers and / or virtual machines and / or software containers may include different processor types, different numbers of processing cores per processor, different amounts of memory for each processor type, different amounts of memory per processing core, different encryption capabilities, different amounts of available off-processor memory, different memory bandwidth (DDR) speeds, and diverse types and capabilities of network interfaces, such as Ethernet cards. Thus, different COTS servers and / or “bare metal” servers and / or virtual machines and / or software containers may have different processing capabilities depending on the specific hardware. Each of the COTS servers and / or “bare metal” servers and / or virtual machines and / or software containers may include different software profiles. For example, each of the COTS servers and / or “bare metal” servers and / or virtual machines and / or software containers may include different software operating systems and / or other services that run them, which are generally referred to as frameworks in this specification. Thus, different COTS servers and / or "bare metal" servers and / or virtual machines and / or software containers may have different software processing capabilities that vary depending on the specific software profile.

[0017] Referring to Figure 3, for data processing and data transfer over a network, the hardware and / or software architecture may consist of multiple different planes, each performing a different set of functions. In the relevant part, the layered architecture may include different planes such as the management plane 300, control plane 310, data plane 320, and switch fabric 330, which can perform the sending and receiving of data packets.

[0018] For example, the management plane 300 can generally be thought of as user interaction or a general software application that runs on it. The management plane typically configures, monitors, and provides management, monitoring, and configuration that is provided to all layers of the network stack and other parts of the system.

[0019] For example, the control plane 310 is a component of the switching functionality, often involving system configuration, management, and the exchange of routing table and forwarding information. Typically, the exchange of routing table information is performed relatively infrequently. The route controller of the control plane 310 exchanges topology information with other switches and builds routing tables based on routing protocols. The control plane may also create forwarding tables for the forwarding engine. Generally, the control plane can be thought of as the layer that determines where traffic is sent. Because control functions are not performed for each individual packet arriving, they tend not to have strict speed constraints.

[0020] For example, the data plane 320 parses packet headers for switching and manages quality of service, filtering, media access control, encapsulation, and / or queuing. Generally speaking, the data plane carries data traffic, which might be equivalent to a cable distribution network. Generally, the data plane can be thought of as the layer that, through the switch fabric, primarily forwards traffic to the next hop along a path to a selected destination, according to control plane logic. Because the data plane performs functions for each individual packet arriving, it tends to be highly speed-constrained.

[0021] For example, the switch fabric 330 provides a network topology that interconnects network nodes via one or more network switches.

[0022] As the system expands to support additional customers, additional COTS servers and / or "bare metal" servers and / or virtual machines and / or software containers are included in the system, and the overall processing capacity of the system is increased. To provide processing redundancy, one or more additional COTS servers and / or "bare metal" servers and / or virtual machines and / or software containers may be included that are assigned as "backups" to be swapped with "active" processes upon detection of a failure event. Scaling of the data plane 320 on the COTS servers and / or "bare metal" servers and / or virtual machines and / or software containers must ensure sufficient high-speed processing of data packets and sufficient bandwidth for transmitting data packets to service dynamically varying processing requirements and prevent loss in other ways.

[0023] It is desirable to virtualize all or part of the data plane, particularly the remote PHY functions on COTS servers and / or "bare metal" servers. In this way, the MAC core for the cable distribution system can be run on a COTS server and / or "bare metal" server. As referred to herein, the virtual remote PHY MAC core may be referred to as a vCore instance herein.

[0024] Referring to FIG. 4, it is desirable to incorporate a platform as a service that uses operating system-level virtualization to package and deliver software, generally referred to as a container 410. Each of the containers is isolated from each other and bundles its own software, libraries, and configuration files. The containers may communicate with each other using defined channels. As a general matter, one or more applications and their dependencies can be packed into virtual containers that can run on COTS servers and / or "bare metal" servers and / or virtual machines. This containerization improves flexibility and portability when running applications on-premises COTS servers, "bare metal" servers, public cloud COTS servers, private cloud COTS servers, or other means. If each container is relatively lightweight, a single COTS server and / or "bare metal" server, and / or a virtual machine operating on a COTS server and / or "bare metal" server can run multiple containers simultaneously. Further, the COTS server and / or "bare metal" server, and / or virtual machine and / or container may be distributed within a cable distribution system.

[0025] COTS servers and / or “bare metal” servers and / or virtual machines may include a container orchestration system 420 for automating the application deployment, scaling, and management of containers 410 across one or more COTS servers and / or “bare metal” servers and / or virtual machines. The computing device running the container orchestration system 420 is preferably isolated from the computing device providing containers for data plane applications. As can be understood, in the virtual machine illustrated in Figure 4, COTS B, etc., may be omitted. Application deployment, scaling, and container management may include a cluster across multiple hosts, such as multiple COTS servers. Container deployment, maintenance, and scaling may be based on underlying system capability characteristics, such as different processor types, different numbers of processing cores per processor, different amounts of memory for each processor type, different amounts of memory per processing core, different amounts of available off-processor memory, different memory bandwidth (DDR) speeds, different frameworks, and / or diverse types and capabilities of network interfaces, such as Ethernet cards. Furthermore, the container orchestration system 420 may allocate different amounts of underlying system capabilities, such as a specific processor type, a selected number of processors (e.g., one or more), a specific number of processing cores per selected processor, a selected amount of memory for each processor type, a selected amount of memory per processing core, a selected amount of available off-processor memory, a selected framework, and / or a selected amount and / or type of network interfaces, such as an Ethernet card. A corresponding agent for the container orchestration system 420 may be included in each COTS server (e.g., COTS A and / or COTS B).

[0026] The container orchestration system 420 may include a group of containerized components, generally referred to as a pod 430. A pod consists of one or more containers coexisting on the same COTS server and / or "bare metal" server and / or the same virtual machine, and the containers can share the resources of the same COTS server and / or "bare metal" server and / or the same virtual machine. Each pod 430 is preferably assigned a unique pod IP address within the cluster, so that applications can use ports without the risk of conflict. Within a pod 430, each container may refer to one another based on localhost or other addressing services, but it is preferable that a container in one pod does not have a way to directly address another container in another pod, and therefore it is preferable to use the pod's IP address or other addressing services.

[0027] Conventional D-CMTS RPHY cores may be implemented as custom-built appliances, including both software and hardware, to achieve desired performance characteristics, such as ensuring data packet forwarding timing. Custom-built appliances, due to their fixed characteristics, do not support automated deployment or scaling. In contrast to custom-built appliances, vCore instances are preferably implemented as software running on COTS servers and / or "bare metal" servers on operating systems such as Linux®. vCore instances are preferably implemented in a way that facilitates automated techniques such as lifecycle management, flexible scaling, health monitoring, and telemetry. Unfortunately, running vCore instances on COTS servers and / or "bare metal" servers tends to present several challenges, primarily related to data plane components. One of the main challenges is ensuring that data is delivered to the network in a timely and effective manner to achieve the real-time characteristics of a cable data delivery environment. Cable data delivery environments involve real-time constraints on the timing of data packet delivery, which are not present in typical web-based or database environments.

[0028] Each vCore instance is preferably implemented within a container, and the size of each container (e.g., scale) is translated into the amount of server hardware and software resources allocated to a particular vCore instance. The amount of server hardware and software resources allocated to each particular vCore instance is preferably a function of the number of customer groups (e.g., service groups) and / or the number of customers on which the vCore instance can easily provide RPHY MAC Core services. For example, a limited amount of server hardware and software resources may be allocated to a particular vCore instance having a limited number of customers and / or groups of customers. For example, a substantial amount of server hardware and software resources may be allocated to a particular vCore instance having a substantial number of customers and / or a substantial number of customers. For example, selected server hardware resources are preferably allocated across different vCore instances in a non-overlapping manner so that each vCore instance has a dedicated and predictable amount of server hardware resources. For example, selected software resources are preferably allocated across different vCore instances in a non-overlapping manner so that each vCore instance has a dedicated and predictable amount of software resources.

[0029] For example, the preferred number of CPU cores to be allocated to each vCore instance (Cc) may be a function of the total USSG (Upstream Service Group - a group of customer modems and / or set-top boxes) and the total DSSG (Downstream Service Group - a group of customer modems and / or set-top boxes) connected through that vCore instance (DSsg). This can be expressed as vCore:Cc=f1(USsg,DSsg). Other hardware and / or software characteristics may be allocated similarly as needed.

[0030] For example, the network capacity allocated to each vCore instance (Cbw) may be a function of the total USSG (upstream service group - group of customer modems and / or set-top boxes) (USsg) and total DSSG (downstream service group - group of customer modems and / or set-top boxes) (DSsg) connected to that vCore instance. This can be expressed as Cbw = f2(USsg, DSsg). Other hardware and / or software characteristics may be allocated similarly as needed.

[0031] Scaling vCore instances can refer to the ability to automatically create and deploy vCore instances within containers on COTS servers and / or "bare metal" servers and / or virtual machines, appropriately sized to accommodate a specific set of remote physical devices and / or service groups (e.g., a set of cable television customers) and / or cable television customers. In some cases, scaling vCore instances can also include the ability to automatically modify the hardware and / or software characteristics of existing vCore instances within containers on COTS servers and / or "bare metal" servers and / or virtual machines, appropriately sized to accommodate a specific set of remote physical devices and / or service groups (e.g., a set of cable television customers) and / or cable television customers.

[0032] The resource allocation manager 470 may allocate or reallocate appropriate amounts of hardware and software, COTS server and / or “bare metal” server resources, to each specific vCore instance (e.g., CPU cores, and / or memory, and / or network capacity). The amount of these COTS server and / or “bare metal” server hardware and software resources allocated or reallocated to each vCore instance is a scaling function, and may also be other functions, such as various other resource allocations. An agent corresponding to the resource allocation manager 470 may be included in each COTS server (e.g., COTS A, COTS B).

[0033] A vCore instance includes data plane software for forwarding data packets and other functions of the data plane. The data plane software may include a set of data plane libraries and network interface controller (NIC) drivers used to manage data packets for the data plane. Preferably, the data plane software operates in user space, as opposed to kernel space like typical network processing software, and therefore does not use the operating system kernel and container-managed network drivers and plugins. For example, the data plane software may include a queue manager, buffer manager, memory manager, and / or a packet framework for packet processing. The data plane software may use CPU cores isolated from the kernel, meaning that processes scheduled by the operating system are not running on these isolated CPU cores. CPU core isolation between the data plane software and the operating system software ensures that tasks performed by the operating system software do not interfere with the data plane software processing data packets in a timely manner. Furthermore, CPU core isolation between the data plane software and the operating system software allows the same physical central processing unit to be used, even if it is a different core. In addition, other hardware and / or software capabilities may similarly be separated into, for example, selected processors (e.g., one or more), a specific number of processing cores per selected processor, a selected amount of memory for each processor type, a selected amount of memory per processing core, a selected amount of available off-processor memory, a selected framework, and / or a selected amount of and / or network interfaces.

[0034] Furthermore, it is desirable that each vCore instance has its own dedicated network bandwidth functionality, separate from other vCore instances and operating system software. To provide dedicated network bandwidth to vCore instances, the physical network interface card may be virtualized so that multiple different software applications can utilize the same network interface card, each with its own guaranteed bandwidth. The network interface card is preferably virtualized using a single Root I / O Virtualization (SR-IOV) technology. SR-IOV divides the physical functionality of the NIC (e.g., PF) into one or more virtual functionality (VF). The capabilities of PF and VF generally differ. Generally, PF supports queuing, description, offloading, hardware locking, hardware link control, etc. Generally, VF supports networking functionality based on queues and descriptors.

[0035] The automatic creation, deployment, and deletion of vCore instances may be performed by the container orchestration system 420.

[0036] Referring to Figure 5, a vCore instance 530 can run on a COTS server and / or “bare metal” server 500 and function as a remote PHY MAC core for one or more remote physical devices connected via an integrated interconnect network, typically located within the same hub. A vCore instance 530 may include data plane software 532. Each of the vCore instances 530 may be a group of vCore instances, commonly referred to as a POD, if desired. The COTS server 500 may communicate with the internet 560, a set of networking switches 570, remote physical devices 580, and customers 590. The COTS server and / or “bare metal” server, including the vCore instances running on it, is typically a relatively high-performance server having one or more of the following characteristics:

[0037] Hardware: At least one management NIC 510 is typically connected to another management network 512. The management NIC 510 is primarily used for orchestrating and managing data traffic.

[0038] Preferably, for hardware timestamping functionality of data packets, at least two (for redundancy) data plane NICs 514 (i.e., data plane physical network interfaces) are included, along with SR-IOV and PTP (IEEE 1588) 522. The data plane NICs 514 are used to provide connectivity to remote physical devices and customer modems, and / or to provide set-top boxes / customer terminals behind such remote physical devices. Each vCore instance 530 may include a virtual function 534 network interface to each of the data plane NICs 514.

[0039] Furthermore, the hardware may include a dedicated device for DES encryption.

[0040] software: The operating system on the COTS server and / or "bare metal" server is preferably a Linux OS such as Ubuntu or Red Hat.

[0041] COTS servers and / or "bare metal" servers and / or virtual machines include container software.

[0042] COTS servers and / or "bare metal" servers and / or virtual machines and / or other servers include, at a minimum, part of a container orchestration system.

[0043] COTS servers and / or “bare metal” servers and / or virtual machines and / or other servers include, at least in part, a Resource Allocation Manager (RAM) 520 that manages the server allocation of software and / or hardware resources for vCore instances, including, for example, CPU cores, memory, VF, WAT VF, MAC address, etc. The RAM 520 may also provide server configuration, including diagnostics and health monitoring, such as OS configuration and driver support. COTS servers and / or “bare metal” servers and / or other servers may include, at least in part, an Orchestration Application 540 that manages the management of vCores (e.g., containers and / or pods).

[0044] COTS servers and / or “bare metal” servers and / or other servers may run a PTP application 522 that synchronizes the system clocks of the COTS servers and / or “bare metal” servers and / or virtual machines and / or vCore instances 520 based on the system-wide grandmaster clock. For improved accuracy, the PTP application 522 is preferably based on hardware time stamping and an accurate hardware clock present on the NIC 514.

[0045] Container initialization and resource allocation may be performed in a distributed manner. Initial vCore initialization 582 may be used to perform or otherwise have the default configuration of instantiated vCores. vCore orchestration 584 may be used to perform or otherwise have the management of instantiated vCores, along with the allocation of resources to specific vCores. Thus, initial vCore initialization 582 and vCore orchestration 584 cooperate to instantiate vCores, allocate resources to vCores, and manage the resource-allocated instantiated vCores. Initial vCore initialization 582 preferably works with an orchestration application 540 on the server to instantiate default vCores. vCore orchestration 584 preferably works with an orchestration application 540 on the server to perform vCore orchestration. vCore orchestration 584 preferably works with RAM 520 to allocate resources to vCores.

[0046] As mentioned above, a COTS server, including a vCore instance, has resource allocations managed at least partially by RAM520. During the COTS server startup phase, RAM creates multiple resource pools (CPU cores, data plane network VFs, encryption VFs, etc.), and then RAM can allocate or lease resources from each pool to vCore PODs when requested by the container orchestration system 540. Furthermore, RAM520 may manage data encryption and decryption, which can be selectively offloaded to dedicated hardware as needed.

[0047] RAM520 may include a REST API that can be used to allocate and release resources, and to determine resource availability and allocation status. RAM520 may also periodically checkpoint the resource pool state to a durable in-memory key-value database cache, which can then be used if the COTS server crashes. The in-memory key-value database cache is preferably unsuitable for easy random access and suitable for reconstructing data into memory if the COTS server crashes.

[0048] Referring also to Figure 6, the RAM520 resource pool 600 may include, for example, several allocated hardware and / or software resources.

[0049] A single resource pool may contain 610 CPU cores. From the total number of physical CPU cores available on the server (Tc), the COTS server startup configuration can allocate a number of scheduled CPU cores (Sc) and isolated CPU cores (Ic) for multiple operating systems, where Sc + Ic = Tc. Sc CPU cores are used by non-dataplane applications (OS, RM, PTP apps, control plane, management plane, etc.), while Ic CPU cores are used only by dataplane-based software. RAM may create and manage a CPU core pool 610 consisting of Ic cores identified by CPU core IDs.

[0050] Another resource pool may include dataplane NIC VF620. When the COTS server starts, dataplane NIC VFs may be created on the vCore instances. The number of dataplane NIC VFs created must be greater than the predicted number of vCore instances that may be deployed on the COTS server. Dataplane NIC VF pool 620 may contain PCI addresses, or if not, may contain the PCI addresses of all dataplane NIC VFs created at startup.

[0051] Another resource pool may include an encryption VF630. In a similar manner to the dataplane NIC VF620, a VF may be created at server startup based on the dedicated portion of encryption devices available to the vCore instance. The encryption VF pool 639 may include PCI addresses, or if not, the PCI addresses of all encryption VFs created at startup.

[0052] Another resource pool may include dataplane MAC address pool 640. Often, NIC VF534 receives a “random” MAC address assigned via the operating system kernel or a driver within dataplane 532. Using “randomized” MAC addresses for vCore instances is not optimal and requires complex MAC address management. Dataplane MAC address pool 640 may use a locally managed range that is unique for each server of the vCore instance.

[0053] Another resource pool may include a network capacity 650. SR-IOV does not support bandwidth division, which results in that a PF or VF on a data plane NIC can use some or all of the bandwidth at any point available on that NIC. Providing bandwidth division of network capacity can be implemented as follows. The system may assume that a data plane NIC on a specific server with vCore instances has a total bandwidth of Tbw, and each vCore instance deployed on that server requires some capacity calculated based on the above formula (Cbw = f2(USsg, DSsg)). Then, the total capacity required by all vCore instances deployed on a COTS server is smaller than the available total bandwidth (Cbw1 + Cbw2 +... + CbwN < Tbw). Therefore, the network capacity "pool" 650 can be the total bandwidth (Tbw) available on the data plane NIC. Next, the RAM 520 can reserve the network capacity of vCore instances in response to requests for Tbw.

[0054] Other resource pools may be included as well, as required.

[0055] A vCore instance configuration typically consists of at least two parts. The first part may be an RPHY MAC Core configuration. The RPHY MAC Core configuration may include, for example, DOCSIS, RF, RPD, cable-mac, IP addressing, routing, etc. The second part may be a data plane configuration 532. The data plane configuration 532, in particular the virtualized data plane for the RPHY MAC Core device configuration, may include, for example, the CPU Core ID used by the data plane 532, the data plane network VF address used by the data plane 432, the MAC address of the interface, the encryption VF address used for encryption offload, memory allocation, etc. In many embodiments, the RPHY MAC Core configuration is provided by multiple system operators before the actual configuration. The vCore instances of the data plane 532 may be determined based on resource information received from RAM 520 by the vCore instance itself during the initialization phase.

[0056] Referring to Figures 7A and 7B, an exemplary initialization process is illustrated. First, multiple system operators 700 create vCore instances with specific names in 710. Multiple system operators 700 provide RPHY MAC Core configurations in 712 for the vCore instances created in 710. Multiple system operators 700 provide scaling for the vCore instances created in 710 in 714. Multiple system operators 700 provide other deployment parameters for the vCore instances created in 710 in 716. Based on the provided data, multiple system operators 700 deploy the vCore instances created in 710 in 718. The vCore orchestration system 730 creates deployment specifications for the vCore instances created in 710 in 732. The vCore orchestration system 730 translates the scaling and deployment parameters into vCore instance-specific hardware and / or software allocations in 734. The vCore orchestration system 730 deploys the vCore instances created in 710 to the vCore orchestration API 740 at 736. Based on the vCore orchestration API 740, the vCore orchestration API 740 deploys pods of the vCore instances created in 710 onto the COTS server and / or "bare metal" server and / or virtual machine 750 at 738. The vCore orchestration API 740 notifies the vCore orchestration system 730 at 742 that the pods have been deployed. The COTS server and / or "bare metal" server and / or virtual machine 750 initializes pod 738 at 760. Pod 762 requests resources from the resource management application 770 at 764. The resource management application 770 allocates these resources at 722. Pod 762 generates the data plane configuration on 774, starts the data plane with the configuration on 774 on 776, starts the control and management plane on 778, and pushes the RPHY MAC Core configuration on 780.vCore POD762 responds that the vCore instance is running on 782 (e.g., initializing and configuring) to the vCore orchestration system 730. The vCore orchestration system 730 responds that the vCore is running on 784 to multiple system operators 700.

[0057] Furthermore, each functional block or various feature in each of the embodiments described above may be implemented or executed by a circuit, which is typically an integrated circuit or a combination of integrated circuits. Circuits designed to perform the functions described herein may include general-purpose processors, digital signal processors (DSPs), application-specific or general-purpose integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic, or discrete hardware components, or combinations thereof. The general-purpose processor may be a microprocessor, or alternatively, the processor may be a conventional processor, controller, microcontroller, or state machine. The general-purpose processor or each of the circuits described above may be composed of digital circuits or analog circuits. Furthermore, as advances in semiconductor technology lead to the emergence of technologies for creating integrated circuits that replace currently existing integrated circuits, integrated circuits using these technologies may also be used.

[0058] The present invention is not limited to the specific embodiments described, and variations thereof may be made without departing from the scope of the invention as defined in the appended claims, such that they may be interpreted in accordance with the principle of superiority law, including several other principles that extend the enforceable scope of the claims beyond the literal scope of the equivalent doctrine or thereto. Unless the context indicates otherwise, a claim reference to a number of examples of an element, whether it is a reference to one example or to more examples, requires at least a predetermined number of examples of the element, but is not intended to exclude the scope of a claimed structure or method having more examples of that element than described. When used in the claims, the term “includes” or its derivatives is used in a non-exclusive sense and is not intended to exclude the presence of other elements or steps in the claimed structure or method.

Claims

1. It is a cable system, (a) A headend connected to a plurality of customer devices via a transmission network including remote fiber nodes that convert digital data to analog data suitable for the plurality of customer devices, wherein the headend includes at least one server including a processor, (b) A container running on at least one server, the container including a data plane application that provides packets of data to be transmitted to the remote fiber node, (c) The data plane application, (1) A virtual networking function that provides a dedicated networking function for a portion of the physical networking of the server, wherein the dedicated networking function is associated with the allocation of the virtual networking function based on the plurality of customer devices, and the virtual networking function provides the data plane application with bandwidth when the data plane application transmits packets to the remote fiber node, and (2) A computing resource function that provides dedicated computing resources for a portion of the server, wherein the dedicated computing resources provide a dedicated portion of the server function of the server, with the allocation of the computing resource function based on the plurality of customer devices, and the computing resource function provides the data plane application to the data plane application, the data plane application being contained within the container instantiated by at least one of the computing resource functions, (d) The data plane application that receives data from a data source and provides data packets for transmission to the remote fiber node, A cable system, including a cable system.

2. The cable system according to claim 1, wherein the headend receives packetized video, receives packetized data from the network, and transmits the packetized data to the network using the data plane application.

3. The cable system according to claim 2, wherein the headend transmits downstream data to one of the selected customer devices using the data plane application, transmits downstream video to one of the selected customer devices using the data plane application, and receives upstream data from one of the selected customer devices using the data plane application.

4. The cable system according to claim 1, wherein the transmission network includes a remote PHY that includes at least one of an orthogonal amplitude modulator and an orthogonal frequency division modulator.

5. The system further comprises a second container running on at least one of the servers, the second container including a second data plane application that provides packets of data for transmission to the remote fiber node, (a) The second data plane application is (1) A second virtual networking function that provides a dedicated networking function for a portion of the physical networking of the server, wherein the dedicated networking function involves the allocation of the second virtual networking function based on the plurality of customer devices, and the second virtual networking function provides the data plane application with bandwidth when the data plane application transmits packets to the remote fiber node, and (2) A computing resource function that provides a second dedicated computing resource for a portion of the server, wherein the second dedicated computing resource provides a dedicated portion of the server function of the server, with the allocation of the computing resource function based on the plurality of customer devices, and the computing resource function provides the data plane application to the data plane application, the second data plane application contained in the second container which is instantiated by at least one of the computing resource functions that provides the dedicated computing resource, (b) The second data plane application that receives data from the data source and provides data packets for transmission to the remote fiber node, The cable system according to claim 1, including the following:

6. The cable system according to claim 5, wherein the dedicated computing resource includes at least one of (a) processor type, (b) number of processing cores per processor, (c) amount of memory per processor type, (d) amount of memory per processor core, (e) encryption capability, (f) amount of available off-processor memory, (g) memory bandwidth (DDR) speed, (h) network interface type, and (i) network capability type.

7. The cable system according to claim 5, wherein the dedicated computing resource includes at least one of (a) a software profile, (b) an operating system, (c) a software service, and (d) a framework.

8. The cable system according to claim 1, wherein the data plane application includes a remote PHY function.

9. The cable system according to claim 1, wherein the data plane application includes remote MAC functionality.

10. The cable system according to claim 1, wherein the data plane application includes remote PHY MAC functionality.

11. The cable system according to claim 1, wherein the data plane application contained within the container is instantiated with the virtual networking function and the computing resource function.

12. The cable system according to claim 1, wherein the dedicated computing resources include selected processing cores.

13. The cable system according to claim 1, wherein the dedicated computing resources include network capacity.

14. The cable system according to claim 1, wherein the data plane application does not use the operating system kernel of the at least one server.

15. The cable system according to claim 1, wherein the dedicated computing resources include processing cores that are not shared with tasks performed by the operating system of the at least one server.

16. The cable system according to claim 1, wherein the dedicated networking function includes a guaranteed minimum bandwidth.

17. The cable system according to claim 1, further comprising an encryption device separated from the data plane application.

18. It is a cable system, (a) A headend connected to a plurality of customer devices through a transmission network including a node that converts digital data into analog data suitable for a plurality of customer devices, wherein the headend includes at least one server including a processor, (b) A container running on at least one server, the container including a data plane application that provides packets of data to be sent to the node, (c) The data plane application, (1) A virtual networking function that provides a dedicated networking function for a portion of the physical networking of the server, wherein the dedicated networking function is associated with the allocation of the virtual networking function based on the plurality of customer devices, and the virtual networking function provides the data plane application with bandwidth when the data plane application transmits packets to remote fiber nodes, and (2) A computing resource function that provides dedicated computing resources for a portion of the server, wherein the dedicated computing resources are allocated to the computing resource function based on the plurality of customer devices, and the computing resource function provides the dedicated computing resources to the data plane application, provides a dedicated portion of the server function of the server, and the data plane application is contained within the container instantiated by at least one of the computing resource functions, (d) A cable system including the data plane application that receives data from a data source and provides packets of data for transmission to the node.

19. The head end of the cable system, (a) The headend is connected to the plurality of customer devices through a transmission network that includes remote nodes that convert received digital data into transmission data suitable for the plurality of customer devices, and the headend includes at least one server including a processor, (b) A data plane application running on at least one server, wherein the data plane application provides packets of data to be transmitted to the remote node, (c) The data plane application is (1) The data plane application running on the at least one server communicates with the provided data packets to the remote node using a physical networking device outside the data plane application, the physical networking device includes a function to provide the remote node with a maximum level of bandwidth, the data plane application running on the at least one server provides data to a virtual networking function running on a network interface controller configured to provide the physical networking device with a first level of bandwidth for the data packets, which is less than the maximum level of bandwidth, the physical networking device allocates the first level of bandwidth only to the virtual networking function, the physical networking device does not include a function to share the first level of bandwidth with other data plane applications in the cable system, and the physical networking device is configured so that the first level of bandwidth is not used by the other data plane applications when the virtual networking function is not using the first level of bandwidth. (2) A computing resource function that provides dedicated computing resources for a portion of at least one server, wherein one of the at least one servers includes at least one processor, and each of the at least one processor includes a plurality of cores. The data plane application operates on one of the at least one servers using the at least one processor, each of the at least one processors comprising the plurality of cores, and the control system operates on one of the at least one servers using the at least one processor, each of the at least one processors comprising the plurality of cores, and the control system operates on at least a first processor of the at least one processor, each of the at least one processors comprising a first plurality of cores, The data plane application runs on the first processor of the at least one processor, the first processor includes a first plurality of cores, one first set of the first plurality of cores runs the data plane application, the first set of the first plurality of cores is isolated from an operating system running on at least one other core of the first plurality of cores, the operating system is configured such that only the data plane application can run on one of the first plurality of cores, and the headend is configured such that the control system cannot run on one or more of the first set of cores of the first plurality of cores. It is configured to include at least one of the following: (d) The data plane application receives data from a data source and provides data packets to transmit to the remote node, The head end, including the head end.

20. The headend according to claim 19, wherein the headend receives packetized video, receives packetized data from the network, and transmits the packetized data to the network using the data plane application.

21. The headend according to claim 20, wherein the headend transmits downstream data to one of the selected customer devices using the data plane application, transmits downstream video to the selected customer device using the data plane application, and receives upstream data from the selected customer device using the data plane application.

22. The headend according to claim 19, wherein the transmission network includes a remote PHY that includes at least one of an orthogonal amplitude modulator and an orthogonal frequency division modulator.

23. The headend according to claim 19, wherein the computing resource function includes at least one of (a) processor type, (b) number of processing cores per processor, (c) amount of memory per processor type, (d) amount of memory per processing core, (e) encryption capability, (f) amount of available off-processor memory, (g) memory bandwidth (DDR) speed, (h) network interface type, and (i) network capability type.

24. The headend according to claim 23, wherein the computing resource function includes at least one of (a) a software profile, (b) an operating system, (c) a software service, and (d) a framework.

25. The headend according to claim 19, wherein the data plane application includes remote PHY functionality.

26. The headend according to claim 19, wherein the data plane application includes remote MAC functionality.

27. The headend according to claim 19, wherein the data plane application includes remote PHY MAC functionality.

28. The headend according to claim 19, wherein the data plane application contained within the container instantiates the virtual networking function and the computing resource function.

29. The headend according to claim 19, wherein the computing resource function includes selected processing cores.

30. The headend according to claim 19, wherein the computing resource function includes network capacity.

31. The headend according to claim 19, wherein the data plane application operates in user space and does not utilize the operating system kernel of the at least one server.

32. The headend according to claim 19, wherein the computing resource function includes a guaranteed minimum bandwidth.

33. The headend according to claim 19, further comprising an encryption device separated from the data plane application.

Images