Virtual IP for container POD

Abstract

Systems and methods for implementing a virtual IP for a container pod are disclosed. In some embodiments, a method may include operating a cloud-based network system in a containerized software environment to assign a virtual internet protocol (VIP) address to an application pod of the containerized software environment, the VIP being directly accessible from a network external to the containerized software environment. The method may include retaining a range of Internet Protocol (IP) addresses for use as VIP addresses, assigning a first fixed IP address to a first application pod, assigning a first VIP address in the range of IP addresses to the first application pod, and routing traffic directed to the first VIP address to the first fixed IP address.

Description

Technical Field

[0001] Various embodiments of this technology generally relate to improvements in Internet Protocol (IP)-based communication capabilities for software container environments such as Kubernetes® (sometimes specifically K8s). More specifically, embodiments of this technology relate to systems and methods for improved networking capabilities in cloud-based environments, such as implementing Session Border Controllers (SBCs). Background Technology

[0002] Mobile communications, including cellular and Voice over Internet Protocol (VoIP), are a fundamental aspect of modern society. The infrastructure supporting mobile communications can include numerous components and network functions that may need to meet high standards in terms of data and media throughput, failover handling, performance, and reliability. One such network function may include a Session Border Controller (SBC), which may include dedicated components for regulating and protecting Internet Protocol (IP) communication flows such as VoIP and IP video streams. However, components such as SBCs used in mobile communication networks have traditionally comprised expensive, large pieces of equipment that can be difficult to maintain or scale as needed.

[0003] Shifting communications infrastructure functionality from hardware-based implementations to software-defined networking can alleviate some of these challenges. Containerized software deployment and orchestration systems such as Kubernetes are a popular approach for deploying software systems in cloud environments, enabling companies to quickly and easily scale the resources they need on hosted cloud servers. Communications service providers using cloud-based deployments have more freedom and speed to scale their services rather than making extensive changes to the private server infrastructure used for their communications networks.

[0004] However, systems such as Kubernetes have significant limitations when providing mobile communication services and similar networking functions. These limitations can include limited media throughput, a lack of reliable and consistent IP address availability, a lack of uniformity in ingress and egress communication paths, and limited failover capabilities to maintain uninterrupted high availability (HA) service in the event of component failures during a communication session. Therefore, there is a need to improve the implementation of cloud-based networking functions. Summary of the Invention

[0005] This summary is provided to introduce a series of concepts in a simplified form, which will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0006] Various embodiments herein relate to systems, methods, and computer-readable storage media for implementing virtual IPs for container pods. In one embodiment, a cloud-based network system may include one or more processors and a memory having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform the process of assigning virtual Internet Protocol (VIP) addresses to application pods in a containerized software environment, the VIPs being directly reachable from a network outside the containerized software environment. The cloud-based network system may reserve a certain range of Internet Protocol (IP) addresses for use as VIP addresses, assign a first fixed IP address to a first application pod, assign a first VIP address within that IP address range to the first application pod, and route traffic destined for the first VIP address to the first fixed IP address.

[0007] In some embodiments, a cloud-based network system can determine when a first application pod has restarted as a new application pod, assign a new static IP address to the new application pod, and assign a first VIP address to the new application pod, the first VIP address being configured to route traffic to the new static IP address. In some examples, the cloud-based network system can assign a Virtual Network Interface Controller (VNIC) to the first application pod to act as an externally accessible port not managed by the containerized software environment, assign a first static IP address to the VNIC, and assign a first VIP address to the VNIC so that traffic destined for the first VIP address is routed to be received at the first static IP address. The cloud-based network system can assign the first application pod to an active role, assign a second static IP address to a second application pod, assign the second application pod to a standby role, and switch the second application pod from the standby role to the active role in response to a crash of the first application pod. In some embodiments, the cloud-based network system can determine when a first application pod has crashed and, in response to the crash, assign a first VIP address to a second application pod, the first VIP address being configured to route traffic to a second fixed IP address. This cloud-based network system can operate in a public cloud environment, request a public cloud operator to associate the first VIP address with the first application pod, and, in response to the first application pod crash, request the public cloud operator to associate the first VIP address with the second application pod. In another example embodiment, the cloud-based network system can assign a first VIP address to a second application pod, route traffic destined for the first VIP address to a first fixed IP address while the first application pod is running, and route traffic destined for the first VIP address to a second fixed IP address in response to the first application pod crash. This cloud-based network system can operate in a private cloud environment and, based on the Free Address Resolution Protocol (GARP), route traffic destined for the first VIP address to be received at either the first or second fixed IP address. This cloud-based network system can also assign a third fixed IP address to a third application pod, and assign a first VIP address to the first application pod, the second application pod, and the third application pod. The first application pod and the third application pod are both in the primary role, and the second application pod is in the backup role.This cloud-based network system can assign a second VIP address to a first application pod, a second application pod, and a third application pod. When both the first and third application pods are in primary use, traffic is routed via the first and second VIP addresses to the first and third fixed IP addresses, respectively, based on GARP. Furthermore, when either the first or second application pod crashes, traffic intended for that crashed pod is routed to the second fixed IP address via GARP. In some examples, the containerized software environment may include a Kubernetes cluster.

[0008] In some embodiments, a method may include operating a cloud-based networking system within a containerized software environment to assign virtual Internet Protocol (VIP) addresses to application pods within the containerized software environment. These VIPs are directly reachable from networks outside the containerized software environment. The method may include reserving a range of Internet Protocol (IP) addresses for use as VIP addresses, assigning a first fixed IP address to a first application pod, assigning the first VIP address within that IP address range to the first application pod, and routing traffic destined for the first VIP address to the first fixed IP address. Attached Figure Description

[0009] Many aspects of this disclosure can be better understood with reference to the following accompanying drawings. Components in the drawings are not necessarily drawn to scale. Furthermore, similar reference numerals designate corresponding parts in several views. While several embodiments have been described in conjunction with these drawings, this disclosure is not limited to the embodiments disclosed herein.

[0010] Figure 1 This is an illustration of a system configured to implement a virtual IP for a container pod according to certain embodiments of the present disclosure.

[0011] Figure 2 This is an illustration of a system configured to implement a virtual IP for a container pod according to certain embodiments of the present disclosure.

[0012] Figure 3 This is an illustration of a system configured to implement a virtual IP for a container pod according to certain embodiments of the present disclosure.

[0013] Figure 4 This is an illustration of a system configured to implement a virtual IP for a container pod according to certain embodiments of the present disclosure.

[0014] Figure 5 This is an illustration of a system configured to implement a virtual IP for a container pod according to certain embodiments of the present disclosure.

[0015] Figure 6This is a process flowchart of a system configured to implement a virtual IP for a container pod, according to certain embodiments of the present disclosure.

[0016] Figure 7 This is a process flowchart of a system configured to implement a virtual IP for a container pod, according to certain embodiments of the present disclosure.

[0017] Figure 8 This is a process flowchart of a system configured to implement a virtual IP for a container pod, according to certain embodiments of the present disclosure.

[0018] Figure 9 This is a process flowchart of a system configured to implement a virtual IP for a container pod, according to certain embodiments of the present disclosure.

[0019] Figure 10 This is a flowchart of an example method for implementing a virtual IP for a container pod, according to certain embodiments of this disclosure.

[0020] Figure 11 This is a flowchart of an example method for implementing a virtual IP for a container pod, according to certain embodiments of this disclosure.

[0021] Figure 12 This is a flowchart of an example method for implementing a virtual IP for a container pod, according to certain embodiments of this disclosure.

[0022] Figure 13 This is a flowchart of an example method for implementing a virtual IP for a container pod, according to certain embodiments of this disclosure.

[0023] Figure 14 This is a flowchart of an example method for implementing a virtual IP for a container pod, according to certain embodiments of this disclosure.

[0024] Figure 15 This is a flowchart of an example method for implementing a virtual IP for a container pod, according to certain embodiments of this disclosure.

[0025] Figure 16 This is an illustration of a system configured to implement a virtual IP for a container pod according to certain embodiments of the present disclosure.

[0026] Figure 17 This is an illustration of a system configured to implement a virtual IP for a container pod according to certain embodiments of the present disclosure.

[0027] Figure 18 This is an illustration of a system configured to implement a virtual IP for a container pod according to certain embodiments of the present disclosure.

[0028] Figure 19 This is an illustration of a system configured to implement a virtual IP for a container pod according to certain embodiments of the present disclosure.

[0029] Figure 20 This is an illustration of a system configured to implement a virtual IP for a container pod according to certain embodiments of the present disclosure.

[0030] For the purpose of discussing some embodiments of this technology, some components or operations may be separated into different blocks or combined into a single block. Furthermore, while the technology is capable of various modifications and alternative forms, specific embodiments are illustrated by way of example in the accompanying drawings and described in detail below. However, it is not intended to limit the technology to the specific embodiments described. Rather, the technology is intended to cover all modifications, equivalents, and alternatives falling within the scope of the technology defined by the appended claims. Detailed Implementation

[0031] In the following detailed description of certain embodiments, reference is made to the accompanying drawings, which form part of this document, and exemplary embodiments are illustrated therein. It should also be understood that features of the embodiments and examples herein may be combined, interchanged, or removed without departing from the scope of this disclosure, other embodiments may be utilized or created, and structural changes may be made.

[0032] According to various embodiments, the methods and functions described herein can be implemented as one or more software programs running on a computer processor or controller. Dedicated hardware implementations, including but not limited to application-specific integrated circuits, programmable logic arrays, and other hardware devices, can also be configured to implement the methods and functions described herein. The methods and functions can be executed by modules or nodes that may include one or more physical components (e.g., logic, circuitry, processors, etc.) of a computing device configured to perform a particular task or job, or may include instructions, or any combination thereof, that, when executed, cause the processor to perform a particular task or job. Furthermore, the methods described herein can be implemented as a computer-readable storage medium or memory device including instructions that, when executed, cause the processor to perform the methods.

[0033] Figure 1A diagram illustrating a system 100 configured to implement virtual IPs for container pods according to certain embodiments of the present disclosure is provided. System 100 may include one or more Kubernetes containerized software environments 102, one or more external components 104, and one or more networks 106. The Kubernetes environment 102 may include a load balancer module 108, and one or more microservices, applications, or compute pods, such as a configuration module 110, an xcode module 112, a signaling module 114, and a media module 116. Modules within the Kubernetes environment 102 may communicate via an internal pod networking path 122. Components of the Kubernetes environment 102 may connect to the external network 106 via an entry path 118 into the Kubernetes environment 102 and an exit path 120 out of the Kubernetes environment. Components of system 100 may be implemented via computers, servers, hardware and software modules, or other system components. Components of system 100 may also include or have access to one or more data storage devices, data storage media, data storage servers, and associated data structures (such as databases) that may store data files, executable code, or other information.

[0034] In Kubernetes environment 102, applications or microservices can be executed by pods, which can be units of computation. In system 100, various microservices (such as configuration 110, transcoding 112, signaling 114, and media 116) can execute on one or more pods in Kubernetes environment 102. These microservices can be used to manage and process communication flows from mobile communication network providers. For example, external element 104 can include communication devices or networking components involved in a communication session. Media from external element 104 can be transmitted to Kubernetes environment 102 via network 106 (e.g., the Internet or some other data network outside of Kubernetes environment 102) along ingress path 118.

[0035] Before communication messages are delivered to a pod or application such as configuration module 110, ingress path 118 may traverse Kubernetes load balancer 108. Load balancer 108 can process incoming messages to distribute workloads across resources within Kubernetes environment 102 and may always reside on ingress path 118 for network communication in a default Kubernetes setup. From a mobile communication service provider's perspective, the additional overhead of load balancer 108 may be undesirable for Real-Time Transport Protocol (RTP) and User Datagram Protocol (UDP) traffic. Data from network 106 intended for specific microservices 110-116 may first need to pass through load balancer 108 and potentially be routed through Kubernetes' internal pod network 122. A pod may have only one interface (e.g., to Kubernetes pod network 122) and trusted / untrusted isolation cannot be achieved at the network level. Therefore, external communication (e.g., from network 106) may not directly reach the intended microservice or pod. The IP address used for a pod may be temporary (e.g., generated and subsequently removed as needed), making it impossible to know in advance what the IP address for the pod will be, and the IP address may change if the pod or application is restarted. The default Kubernetes pod interface may be low-capacity and unsuitable for media traffic (such as VoIP and IP video communication streams), and may not be able to run accelerated networking technologies such as single-root I / O virtualization (SR-IOV or SRIOV) or the Data Plane Development Kit (DPDK). Furthermore, messaging leaving the Kubernetes environment 102 may take an exit path 120 different from the ingress path 118, thus bypassing the load balancer 108.

[0036] To efficiently deliver mobile network services on cloud-based environments such as Kubernetes 102, many aspects of the deployment of system 100 can be improved. Additional interfaces can be added to application pods to provide capabilities not offered by the Kubernetes cluster. A mechanism can be provided where external work or messaging (e.g., from network 106) can be directly reached by externally reachable IPs for microservices 110-116, and the IP address used for the microservice can be known in advance. For example, a known IP address reachable from an external network can be used to handle Session Initiation Protocol (SIP) phone calls, where a signaling path will be established. The IP address used for the microservice can be configured to remain consistent even after a pod or microservice restart or failure. The same ingress and egress paths can be provided via a direct connection between the pod and external network 106. Example use cases could include handling media packets in RTP communication, where media packets should enter and exit along the same media path. Additional interfaces can provide support for Virtual IP (VIP) addressing. Using the additional interfaces and associated handling described herein, mobile service providers can handle high-throughput media and packet routing via cloud applications in systems such as Kubernetes. These improvements can be applied to various networking and protocol types with similar requirements, such as 4G, 5G, Internet of Things (IoT), SIP, Diameter, RTP, Real-time Transmission Control Protocol (RTCP), Internet Protocol Security (IPsec), Internet Key Exchange (IKE), etc. Figure 2 An example system implementing these modifications is shown in the figure.

[0037] Figure 2 A diagram illustrating a system 200 configured to implement a virtual IP for a container pod according to certain embodiments of the present disclosure is provided. Specifically, system 200 may be depicted in an embodiment where a containerized application or pod has additional interfaces added to enable direct connectivity to an external network (this direct connection is not controlled by a containerized application system such as Kubernetes). System 200 may include external components 204 (e.g., a SIP peer), one or more external networks 206, and multiple containerized applications or microservices running on the pod (such as transcoding module 212, signaling module 214, and media module 216). Furthermore, the additional interfaces enable modules 212-216 to communicate via an internal network 224, which can be used with… Figure 1 The Kubernetes pod network 122 is different.

[0038] These additional interfaces can be implemented via a Virtual Network Interface Controller or Card (Vnic) 226 that is not directly managed by Kubernetes. Vnic 226 enables bypassing the Kubernetes infrastructure and its inherent limitations. Pods or microservices 212-216 with Vnic 226 can communicate directly with the external network 206 via consistent ingress and egress paths, without going through the Kubernetes load balancer 108. Furthermore, persistent VIPs can be assigned to Vnics, which can be known before the instantiation of pods or microservices 212-216 and remain consistent even if the pod or microservice restarts.

[0039] Vnic implementation can be automated and controlled via one or more operators, similar to a control system, added to the control plane of the Kubernetes environment or other containerized software systems. Automation can be achieved through one or more operators configured to know which application to launch, how to launch it, what additional resources the application requires, and on which node the application will be spawned. On that node, the one or more operators can create the necessary network interfaces, inject them into the appropriate pod or application, and associate them with virtual IPs, while bypassing the limitations of the Kubernetes system. About Figure 3 The example implementation is described.

[0040] Figure 3 A diagram illustrating a system 300 configured to implement virtual IPs for container pods according to certain embodiments of the present disclosure is provided. Specifically, system 300 may be depicted as an embodiment in which a custom operator in a containerized software environment manages interfaces added to applications or pods. System 300 may include a cloud-native application (CnApp) operator 302, a CnApp resource 304, a VnicSet operator 306, a VnicSet resource 308, a StatefulSet 312 comprising multiple pods 320, and one or more external networks 306 not managed by a containerized software environment such as Kubernetes.

[0041] A StatefulSet 312 can be a sample Kubernetes workload application programming interface (API) object used to manage stateful applications, where each pod 320 created from the same specification has a persistent identity, rather than being interchangeable. In some examples, each pod 320 in a StatefulSet 312 can be of the same type (e.g., ...). Figure 2 The media pod 216), and different types of pods 320 or applications have different StatefulSets 312.

[0042] The externally reachable Vnic interfaces proposed in this paper can work for StatefulSet 312 as well as general pod 320, Deployment, Job and similar resources. These interfaces allow pod 320 to be reached via a consistent ingress and egress path using a predictable and fixed VIP and via an external network 306 not managed by Kubernetes.

[0043] In Kubernetes, an operator can be a method for packaging, deploying, and managing applications within Kubernetes. An operator can be an application-specific controller that extends the functionality of the Kubernetes API to create and manage applications for users, and can include application-specific information to automate the entire lifecycle of the software it manages. An operator can be a custom Kubernetes controller that uses custom resources (CRs), providing settings for values ​​defined in custom resource definition (CRD) files (which can refer to any form of resource definition data, not strictly a file system file) to manage applications and their components. CRs can provide high-level configuration and settings, which the operator can translate into low-level actions based on logic embedded within the operator. An operator can monitor specific CR types and take application-specific actions to align the current state of the managed application with the desired state specified in the resource. The example cloud-based SBC (cbSBC) or cloud-native SBC (cnSBC) scenarios supported by Vnic described in this paper can be managed by two example custom operators: CnApp operator 302 and VnicSet operator 306. Custom operators can be invoked or initiated by an administrator or user providing the operator's definition file and executing it at the Kubernetes control plane (e.g., via the apply command), causing the control plane to create and run the operator.

[0044] The CnApp operator 302 can manage one or more applications and pods 320 (such as those related to...). Figures 1-2The described media application, signaling application, transcoding application, etc.), and (one or more) StatefulSet 312 or other application deployments or instances. The CR used for CnApp operator 302 can be CnApp resource 304, which can be submitted by the user to define the application to be managed by CnApp operator 302. Once in use, CnApp operator 302 can monitor CnApp resource 304 and can create the desired application (e.g., implemented via StatefulSet 312 and pod 320). CnApp resource 304 can also define what Vnic resource requirements the created application and associated pod 320 have. Based on the Vnic resource requirements, CnApp operator 302 can create a VnicSet resource 308 yaml definition file, which indicates the Vnic resources required by the created StatefulSet 312 and pod 320.

[0045] VnicSet operator 310 can be configured to detect VnicSet resource creation events (e.g., triggered by the generation of a VnicSet resource CRD file by CnApp operator 302), causing VnicSet operator 310 to create the VnicSet resource 308 object itself. The VnicSet resource 308 object can be created through a mechanism by which VnicSet operator 310 assigns or injects a Vnic to pod 320 and enables a VIP on the Vnic for interfacing with external network 306. Resource files for pod 320 (e.g., CnApp resource 304) can include references to the corresponding VnicSet resource 308. Accordingly, CnApp operator 302 and VnicSet operator 310 can work together to implement cnSBC as described herein.

[0046] The implementation scheme for Vnics and VIPs can vary depending on whether the implementation is for a private network (e.g., via OpenStack) or a public cloud environment (such as using OCI (Oracle Cloud Infrastructure), AWS (Amazon Web Services), and Microsoft Azure). This paper describes example implementations for both scenarios. In one implementation, the three-part solution may include: 1) CnApp operator 302 starts application pod 320 (e.g., based on CnApp resource 304) and creates a trigger event for Vnic creation by generating a VnicSet resource definition file 308; 2) VnicSet operator 310 handles the trigger event and creates and injects the Vnic into pod 320; and 3) VnicSet operator 310 enables VIP support on the Vnic, enabling failover or switching scenarios between pods 320 with a fixed VIP that is always reachable from the external network 306. Because CnApp operator 302 and VnicSet operator 310 can work together to implement the network scheme described in this paper, they can be collectively referred to as Netverse operators. Netverse operators provide a complete automation solution for managing network applications with Virtual IP (VIP), High Availability (HA), and dynamic service management. About Figure 4 An example architecture of a Kubernetes system configured to implement the scheme described in this article is described.

[0047] Figure 4A diagram illustrating a system 400 configured to implement virtual IPs for container pods according to certain embodiments of this disclosure is provided. Specifically, system 400 may depict an example Kubernetes architecture employing a custom operator to implement cnSBC. System 400 may include a control plane 402 that can receive and implement user modifications 404, and create and manage applications in application plane 406. Control plane 402 may include a controller manager 408, an API server 410, a scheduler 412, a VnicSet operator 414, a VnicSet resource 416, a CnApp operator 418, and a CnApp resource 420. Application plane 406 may include multiple worker nodes or hosts, including node 1 422, node 2 424, and node 3 426. Each node 422-426 may include one or more pods 428, and each pod may include one or more containers 430. Although depicted as part of control plane 402, elements such as VnicSet manipulator 414 and CnApp manipulator 418 can execute on worker nodes 422-426 and can be configured to hook into control plane 402 to extend its functionality. System 400 may correspond to Figures 2-3 The system described in [the document].

[0048] When Kubernetes is deployed, a cluster is provided, consisting of one or more worker machines referred to as nodes 422-426. Each node 422-426 can host one or more pods 428, which can be the smallest deployable unit of computing that can be managed in Kubernetes. Each pod 428 can run one or more containers 430, which can be a bundle of software and all its dependencies (e.g., an application or microservice). Through nodes 422-426 and pods, Kubernetes can manage the resources required to execute, scale, and manage applications or services in a cloud environment. These applications and resources can be managed via components running in the control plane 402.

[0049] Control plane 402 can be a container orchestration layer that exposes APIs and interfaces to define, deploy, and manage the lifecycle of container 430 and the nodes 422-426 and pod 428 running container 430. Only sample systems that can be included in control plane 402 are depicted in system 400.

[0050] API server 410 can provide a front end for control plane 402, through which all other components can interact. API server 410 can expose the Kubernetes API and receive user-modified 404 responses and other input. User-modified 404 responses can be received in the form of kubectl command-line interface (CLI) instructions, which can include objects defined via YAML or JSON files. API server 410 can validate and configure data for API objects (including pod 428).

[0051] Controller Manager 408 can run controller processes, where the controller can be a control loop that monitors the shared state of the cluster via API Server 410 and makes changes to move the current state toward a desired state (e.g., the desired state can be specified via user modification 404). Controller Manager 408 can run controller processes for the default Kubernetes controller, as well as controller processes for controllers implemented by custom operators (e.g., VnicSet operator 414 and CnApp operator 418). Accordingly, Controller Manager 408 can implement management functions controlled by custom operators such as VnicSet operator 414 and CnApp operator 418.

[0052] Scheduler 412 can monitor newly created pod 428 or other resources that are not assigned to nodes 422-426, and can select which nodes to run the resource on. The decision of which nodes 422-426 to assign to the newly created pod 428 can be based on resource requirements and availability (e.g., based on node load distribution), data locality, and other factors. The operator responsible for creating pod 428 (e.g., CnApp operator 418) can also influence or control which nodes 422-426 the pod is assigned to.

[0053] Such as about Figure 3As described, a Kubernetes operator can be a method for packaging, deploying, and managing applications in Kubernetes. The operator can implement a custom controller to manage the application based on values ​​defined in a custom resource (CR) (e.g., VnicSet resource 416 and CnApp resource 420). In some embodiments, the operator (e.g., VnicSet operator 414 and CnApp operator 418) can run on worker nodes 422-426. Since the operator can implement the controller on worker nodes 422-426, but the controller can run on control plane 402, the operator can effectively extend control plane 402 to worker nodes 422-426. The operator can add itself to the list of controller managers 408, thereby extending that list to application plane 406, and can begin monitoring the operator's resources via API server 410.

[0054] Application pod 428 can be created as part of StatefulSet 312 resource. Scheduler 412 can decide which nodes 422-426 to assign pod 428 to based on resource availability. Controller manager 408 can listen to API server 410 for the resources it subscribes to. If a StatefulSet 312 resource is modified, change notification can be triggered by controller manager 408, which invokes the corresponding controller method for that resource to bring the resource back from its current state to the desired state.

[0055] When a custom operator (e.g., VnicSet operator 414 and CnApp operator 418) adds a hook to the controller manager 408, the controller manager monitors the API server 410 for notifications of changes to custom resources associated with the custom operator (e.g., VnicSet resource 416 or CnApp resource 420), such as creation, modification, or deletion. Any notification can indicate that the state of the custom resource has changed (e.g., a change to the desired or current state may lead to a determination that the current and desired states do not match), and trigger a response to the API server 410. Based on this notification, the controller manager 408 can perform controller actions on the custom resource to restore equilibrium between the application pod's current and desired states. Essentially, an operator can add an endpoint to the Kubernetes API called a custom resource (CR), along with a control plane component 402 or hook that monitors and maintains this new type of resource. Custom operators can include a coordinator module or process 432 that can instruct the controller manager 408 on how to bring the custom resource from its current state to its desired state.

[0056] Accordingly, the operator may include coordinator 432 as part of a controller method that continuously loops to determine the state of associated custom resources (e.g., VnicSet operator 414 can continuously monitor the state of VnicSet resource 416 via API server 410 with the help of coordinator 432 and controller manager 408). The operator may receive a notification whenever a change event occurs to the custom resource (e.g., based on user modification 404). The operator can then adjust the state of the custom resource to bring it from its current state to the desired state. In the event of any error or other failure to bring the custom resource to the desired state, the operator and coordinator 432 may continue to loop through a specific set of operations until the custom resource reaches the desired state.

[0057] Such as about Figure 3 One part of the described example three-part scheme for implementing cnSBC includes a CnApp operator 418. The CnApp operator 418 can be configured to manage its own needs, as well as create, monitor, and manage application pod 428 and its associated resources. Furthermore, when pod 428 is created, updated, or deleted, the CnApp operator 418 can dynamically create, update, delete, and apply the VnicSet resource 416 definition file. The VnicSet operator 414 can then take action on those change notifications to the VnicSet resource 416. For example, based on a change notification to the VnicSet resource 416 from the CnApp operator 418, the VnicSet operator 414 can create a Vnic and inject it into nodes 422-426 or pod 428, or enable a VIP on the Vnic associated with pod 428.

[0058] CnApp operator 418 can be configured to dynamically scale the application and associated resources (e.g., changing the number of pods from four to two, or from four to six, etc.). CnApp operator 418 can adjust VnicSet resource 416 based on the scaled number of pods or other resources. The adjusted VnicSet resource 416 can trigger events that cause VnicSet operator 414 to adjust the number of Vnics inserted at nodes 422-426 (e.g., increasing the number of Vnics when the number of pods 428 increases proportionally, and deleting and removing Vnics when the number of pods 428 decreases proportionally).

[0059] CnApp operator 418 can manage StatefulSets or Deployments for pod 428, as well as other similar resources with multiple replicas. For example, CnApp operator 418 can also dispose of persistent volume claims (PVCs), ConfigMaps, Secrets, or other resources used for storage. It can support all constructs and attributes of Kubernetes resources in custom bundles (e.g., node selectors, images, replicas, ContainerPorts, etc.) as part of its own manifest file. CnApp operator 418 can also create Services (a way to make an application running on a group of pods 428 available on the network so that clients can interact with it) and manage the association between Services and pods 428. Services can be of default types (e.g., ClusterIP, NodePort, Loadbalancer, Headless, etc.) or unique or custom Services. Created Services can be associated with a specific resource comprising multiple pods 428 (e.g., StatefulSet 312) or with a single pod 428. For example, per-pod Services can be specified in the CnApp operator definition file (used to create the CnApp operator 418 itself), so that if a StatefulSet 312 with four pods is created, the Service will be associated with each of the four pods. If pods 428 are added or removed proportionally, the CnApp operator 418 can dynamically add or remove the corresponding Services proportionally.

[0060] Kubernetes has mechanisms for managing application pods that operate on its default resources, such as StatefulSets, Deployments, ReplicaSets, Jobs, etc. However, the CnApp operator 418 can operate on its own custom resource, the CnApp resource 420. The CnApp resource CRD yaml file can be provided via user commit or modification 404 to instruct the CnApp operator 418 on how to configure the application deployment (e.g., using StatefulSets and replica counts for the pod, the IP address to be assigned to the VNIC injected into the pod, and other details). The CnApp operator 418 can use the CRD to generate the CnApp resource 420 itself. The CnApp resource 420 can integrate any resources required for the managed application to function, such as StatefulSets, Services, PVCs, ConfigMaps, Secrets, etc. Therefore, returning to... Figure 3 A CnApp application can include four pods (320) of type StatefulSet 312. If the CnApp application is to scale from four pods (320) to six, this involves not only scaling up the StatefulSet 312 (the default Kubernetes resource), but also scaling up the CnApp resource (304). Scaling up the CnApp resource (304) can include scaling up the StatefulSet 312 replica size from four to six (for the two additional pods (320)), plus two additional PVCs, two additional Services, two additional ConfigMaps, additional Secrets, etc. Therefore, all resources make the application a complete, self-contained box. The CnApp operator (302) can be configured to scale up or down all associated resources together based on the CnApp resource (304).

[0061] Return to Figure 4 In addition to managing (e.g., creating, monitoring, and deleting) applications and their associated pods 428 and resources, the CnApp operator 418 can dynamically create, delete, and update the VnicSet resource 416 custom resource definition (CRD) YAML file and associate it with application resources such as StatefulSets and Deployments. Similarly, the CnApp operator 418 can trigger creation or deletion events for the VnicSet resource 416 based on the VnicSet resource CRD and the requirements of the managed application. These events can cause the VnicSet operator 414 to create the VnicSet resource 416 itself and inject Vnics into pods 428 based on the VnicSet resource 416. Together, the CnApp operator 418 and the VnicSet operator 414 enable the management of pods 428 as part of a StatefulSet or Deployment, using custom behaviors for virtual interfaces, PVCs, per-pod associated services, and default Kubernetes resources. Accordingly, managed application pod 428 can have external (e.g., outside the Kubernetes cluster) connectivity, persistent IP addressing, virtual or floating IP (VIP), a default cluster IP service per pod, high availability (HA) support, symmetric ingress and egress paths, and dynamic service updates.

[0062] CnApp resources are resilient by storing data in a resource metadata section, allowing metadata (e.g., tags and comments) to be retrieved during operator 418 and application restarts to recreate the desired state. CnApp operator 418 may also include a coordinator 432 to monitor changes to custom and owned resources and bring them from their current state to the desired state. Coordinator 432 may include a finalizer associated with CnApp resource 420 to perform bundle cleanup before the resource is deleted. For example, in the event of a crash or interruption, the finalizer can be executed to perform cleanup when the system recovers. Finalizer cleanup may include checking what the associated resources are, including StatefulSet, Deployment, Service, PVC, ConfigMap, Secret, etc.

[0063] Such as about Figure 3 The first part of the example three-part scheme for implementing cnSBC described includes the CnApp operator 418 starting the application pod 428 and creating a triggering event for Vnic creation by generating a VnicSet resource 416. The second part includes the VnicSet operator 414 being configured to handle the triggering event, and creating and injecting the Vnic into pod 428.

[0064] The VnicSet operator 414 can create a VnicSet resource 416 and an associated Vnic, and inject the Vnic into pod 428 to provide a network interface for that pod, as described herein. The Vnic created by the VnicSet operator can be part of or associated with a cloud provider (e.g., OpenStack). Accordingly, upon initialization, the VnicSet operator 414 can execute the API protocol to authenticate with the cloud provider using credentials provided in the VnicSet operator 414 definition file. This authentication process can grant the VnicSet operator 414 privileges to create and manage Vnics on host worker nodes 422-426. Once authentication is successful, the operator 414 can re-authenticate upon receiving a 401 response from the cloud provider. If an ongoing API call fails due to an unauthentication error, the underlying framework can attempt to re-authenticate using the credentials provided during initialization. If re-authentication fails, the operator can retry authentication after rereading the credentials from the installed secret, and can retry reading and re-authentication a selected number of times before declaring an API failure. Upon receiving any cloud API failure, the operator event or request can be requeued, and after a fixed delay, the VnicSet operator 414 and coordinator 432 can be invoked again. In this round, operator 414 can identify itself as unauthenticated and attempt to trigger authentication. Requests can be requeued up to a selected number of times, after which the CRD status can be updated to "authentication failed." All subsequent new requests can declare a failure only after the selected number of re-authentication attempts have been triggered.

[0065] The VnicSet operator 414 can be configured to hook into the Kubernetes control plane 402 to create, monitor, and manage its custom resource (VnicSet resource 416), which can maintain details for one or more Vnics. The VnicSet resource 416 definition file can include podID entries to identify a specific target pod 428. The podID for application pod 428 can be a name or a number and can be entered manually or automatically generated. The definition file can also accept input such as the number of Vnics to create for target pod 428, the Vnic IP, and which application pod 428 the Vnics should be injected into (e.g., identified by pod name or pod ID number). Based on the pod 428 to which the Vnics should be injected, the VnicSet operator 414 can identify the worker nodes 422-426 running on the target pod 428.

[0066] VnicSet operator 414 can create the desired number of Vnics on worker nodes 422-426 running on the target pod 428, and these Vnics can have IPs that are reachable from outside the cluster without requiring Kubernetes management. Vnics can be created with predetermined static IP addresses, which can be provided as part of the CRD file of VnicSet resource 416. Since CnApp operator 418 can generate the VnicSet resource CRD, the CnApp operator can obtain IP address configuration information from the CnApp resource definition file 420 (e.g., from information provided by the user via user modification 404). If a specific IP is not provided or is not provided to it by VnicSet operator 414, it can be dynamically assigned via DHCP (Dynamic Host Configuration Protocol).

[0067] Once a Vnic is created on nodes 422-426 where the target pod 428 resides, the VnicSet operator 414 can create a job on the same worker nodes 422-426. This job can inject one or more Vnics from worker nodes 422-426 into container 430 within the target pod 428. In the Kubernetes architecture, pod 428 can run in a “silo” called a namespace, which can correspond to Linux control groups (cgroups), isolating resource groups within a single cluster. Injecting a Vnic from worker node 422 into pod 428 may involve exposing the namespace of host node 422 where the Vnic was created and moving that Vnic to the namespace of the target container 430. The Vnic can be initially created on host node 422, where the target pod 428 is running, and jobs executing on node 422 can remove the Vnic from that host namespace and cgroups, change its reference, and move it to the namespace of the specific application pod 428 or pod container 430. For example, a Vnic can be injected into container 430 within pod 428, allowing each container 430 to have its own Vnic that cannot be used by other containers within pod 428. By injecting a Vnic, an external interface into the pod can be created, which has characteristics such as a consistent or fixed IP address known before the pod and Vnic are created, high capacity, failover support with HA enabled, identical ingress and egress paths, and potentially lower latency than traffic managed by Kubernetes.

[0068] The VnicSet operator 414 can be configured to operate in all or selected namespaces. During initialization, environment variables can be used with the VnicSet operator 414. The operator scope can be determined based on the value configured for this environment variable: for example, to operate in all namespaces, the "WATCH_NAMESPACE" environment variable can be omitted or left blank, where an empty value means the operator can run within a cluster scope (e.g., all namespaces within the cluster). To operate only within a specified namespace, the "WATCH_NAMESPACE" variable can be defined using namespace names, with multiple values ​​separated by commas.

[0069] The processing of resource creation, updating, and deletion via CnApp operator 418 and VnicSet operator 414 can be managed by the corresponding coordinator element 432. Regarding... Figure 5 An example of the coordinator 432 is described in more detail.

[0070] Figure 5 A diagram depicts a system 500 configured to implement a virtual IP for a container pod according to certain embodiments of the present disclosure. In particular, system 500 may depict an example coordinator module 532, which may correspond to... Figure 4 The coordinator module 532 may include a finalizer module 502, an obsolete resource disposer module 504, and a resource creation and update module 506. As described herein, the coordinator 532 may be a module or process contained within a custom operator (e.g., in a Kubernetes architecture). The coordinator 532 may loop to determine the state of associated custom resources and make changes to those resources to bring them from their current state to a desired state.

[0071] The coordinator 532 logic can follow a sequence flow to determine what changes might be needed to bring the resources to the desired state. In the depicted example, this logic can proceed from the finalizer 502 to the stale resource disposer 504, and then to the resource creation and update module 506.

[0072] Finalizer 502 can be a routine or module executed before a resource is deleted to perform cleanup. Finalizer 502 cleanup may include checking what resources are associated with the application, determining the condition of the resources to determine when it is safe to delete or detach them, and deleting or detaching the appropriate resources. When Kubernetes receives an indication that an application has been deleted or will be deleted, it can create a deletion timestamp to indicate when the resource was marked for deletion. Finalizer 502 used by operators can check this timestamp to determine whether to execute cleanup code or delete the resource.

[0073] The obsolete resource handler 504 can be a routine or module that determines whether a resource associated with the application has become "obsolete" (meaning the resource has been terminated or not found (e.g., no longer exists)). If a resource is determined to be obsolete, the obsolete resource handler 504 can initiate the deletion of any components or resources associated with that obsolete resource. Accordingly, the obsolete resource handler 504 can execute cleanup routines or delete components, even if they have not yet been marked for deletion, unlike the terminator 502.

[0074] The Create or Update Resource module 506 can be configured to check the current state of a resource and compare it with a desired state (e.g., as specified in a CRD yaml file). If the current state does not match the desired state, the Create or Update Resource module 506 can generate or modify the resource until it matches the desired state.

[0075] When an operator is initiated or restarted, it can invoke coordinator 532 to scan the cluster or namespace in which the operator is running to find all resources of its type and check if any of these resources need to be cleaned up, updated, or changed. This scan may include API server 410 in access control plane 402, which can provide information about all resources of a given type (e.g., VnicSet, CnApp, etc.). Based on the results of its scan, the operator can update its own metadata with information about the discovered resources (e.g., resource IDs, status information, etc.). The details of how the elements of coordinator 532 function or the processes they perform may depend on the operators associated with those elements (e.g., VnicSet operator 414 or CnApp operator 418).

[0076] For example, when the CnApp operator 418 or the VnicSet operator 414 starts, boots, or restarts, they can (e.g., by checking the etcd database) scan the cluster to find all or specific namespaces based on their configuration. The corresponding operator can identify all existing resources matching the type (e.g., CnApp for CnApp operator 418, or VnicSet for VnicSet operator 414). The Kubernetes control plane 402 can generate a CREATE event for the corresponding operator for the existing identified resources, as the operator may be identifying these resources for the first time. A predicate function can pass this CREATE event, and the coordinator 532 of the corresponding operator can be invoked. The coordinator 532 of the corresponding operator can determine if there are any finalizers pending on or associated with its identified resources, and if so, execute the finalizer.

[0077] For CnApp operator 418, coordinator 532 can identify CnApp instances for identified resources, read metadata from those CnApp instances (e.g., owned StatefulSets, pods, services, PVCs, associated VnicSets, Vnics), scan the CRD file and bring the resources to the desired state, then update the metadata or status accordingly. For VnicSet operator 414, coordinator 532 can determine if a pod ID in the VnicSet resource has been terminated or not found, and if so, can invoke stale resource disposer 504 to delete the resource, updating the operator metadata and status accordingly. If the pod ID is valid, coordinator 532 can check the VnicSet resource metadata and verify that Vnic injection is not required (e.g., the Vnic on the pod is expected and the current Vnic count is correct). Coordinator 532 can update the VnicSet operator metadata and status to reflect the resource information in the VnicSet operator, and repeat this process until all pods are verified. The VnicSet coordinator 532 can scan the VnicSet resource CRD yaml file to bring the current state of the VnicSet resource to the desired state.

[0078] Even if all operators and all pods are running on the same worker node 422, and that node crashes (or some other combination of crash and resources and operators on the function node), the startup and restart process described above can still be performed. Operator pods can be scheduled onto another available node 424, and all resources can be coordinated to bring them to the desired state. Recovery is possible even in the event of a crash in all components because metadata can be stored on the etcd component (not shown) in the control plane 402, which can be a consistent and highly available key-value store for cluster data. As used herein, "crash" can refer to a node or pod becoming unresponsive, interrupted, lost, faulty, or otherwise unavailable.

[0079] Coordinator 532 can be configured to run concurrently against the controller, allowing multiple worker nodes 422-426 to handle CRD operations (e.g., create, update, and delete). The number of concurrent coordinators 532 can be changed by setting the value of the MaxConcurrentReconciles property under controller.Options.

[0080] The following figures illustrate more specific examples of operations implemented by the custom operator and its coordinator 532. In particular, Figures 6-9This provides an overview of how the CnApp operator 418 and VnicSet operator 414 manage application creation, deletion, pod crashes and restarts, and updates. Figures 10-15 A more detailed processing flow of the coordinator 532 component of the CnApp operator 418 and the VnicSet operator 414 is described. About Figure 6 This describes the application creation event. Once the application pod 428 cluster is generated by the CnApp operator 418, it can trigger the VnicSet operator 414 to create and dispose of Vnics to provide an auxiliary network interface that is not directly managed by Kubernetes.

[0081] Figure 6 This is a process flowchart of a system 600 configured to implement a virtual IP for a container pod, according to certain embodiments of this disclosure. Specifically, Figure 6 The diagram depicts a management module 604, a control plane 602, a CnApp operator 618, and a VnicSet operator 614 (which can correspond to...). Figure 4 The sequence of operations and data transfers between the control plane 402, CnApp operator 418, and VnicSet operator 414. Although for clarity... Figure 6 Not shown in the diagram, but may also be found in components of system 600 and Figure 4 Information is exchanged between other components. The operation of system 600 can be an example of creating a CnApp resource and subsequently a VnicSet resource to create a Vnic in the Kubernetes environment and inject the Vnic into the pod. The operations performed by the CnApp operator 618 and the VnicSet operator 614 can be performed by the corresponding coordinator module (e.g., the resource creation and update module for each custom operator).

[0082] Management module 604 may include a user interface, such as a command-line interface (CLI), an API interface, or other mechanisms, through which the containerized software system can receive initial commands for implementing cnSBC or creating CnApp applications or resources. Management module 604 may issue a "Create CnAppResource" command 606 or instruction to control plane 602. The Create CnAppResource command 606 may include a custom resource definition (CRD) YAML file for the CnApp resource, which may define the characteristics of the CnApp resource to be created and the specifications of the associated VnicSet resource to be created, such as the ReplicaCount field (with a replica count value indicating the number of application pods to be initialized), Vnic details, one or more network names, and Virtual IP (VIP) details. Based on the Create CnAppResource command 606, control plane 602 may trigger or provide a "Create CnAppResource event" notification 608 to CnApp operator 618.

[0083] In response to the CnAppResource creation event notification 608, the CnApp operator 618 can perform several operations. It can create CnApp resources for management. Based on the CRD configuration file, the CnApp operator 618 can create pod resources for the CnApp resource, such as a StatefulSet (or Deployment, or other resource), and Kubernetes can create pods for a StatefulSet. When the CnApp operator 618 determines that one or more pods are running and awaiting Vnic injection, the CnApp operator 618 can initiate the creation of one or more VnicSet resources (e.g., one VnicSet resource per pod, and each VnicSet resource can have one or more Vnics for the associated pods). To do this, the CnApp operator 618 can create a CRD for each VnicSet resource (e.g., a VnicSet resource YAML definition file, such as VnicSetResourceApp.yaml). The CnApp operator 618 can programmatically apply the CRD to the control plane 602 to issue the "Create VnicSetResourceApp" command 610. The Kubernetes-assigned IDs of these VnicSet resources can be stored in the pod metadata. Based on the CnApp configuration or definition file, the CnApp operator 618 can also configure or manage Services (e.g., Cluster IP Service), PVCs, ConfigMaps, Secrets, or other resources for one or more application pods. The CnApp operator 618 can mark itself as the owner of resources created by the CnApp operator 618 for resource cleanup.

[0084] Receiving the "Create VnicSetResourceApp" command 610 at control plane 602 can trigger a "Create VnicSetResource Event" notification 612, which can be sent to or noticed by the VnicSet operator 614. Notification 612 may include data such as a VnicSetResourceApp.yaml CRD file created by the CnApp operator 618, which may include the pod ID, network name, primary IP (e.g., a static IP address for the Vnic), one or more secondary IPs (e.g., one or more virtual IPs to be assigned to the Vnic), and CnApp owner data. Secondary IPs may be included in the CRD or left blank, in which case the VIP can be dynamically assigned by DHCP. The secondary IP, or VIP, assigned to the Vnic can be routed to the primary IP or static IP.

[0085] In response to the Create VnicSetResource event 612 notification (and associated CRD), the VnicSet operator 614 can perform several operations. It can perform a Create VnicSet event handling operation. This operation may include determining that a pod with the podID from the Create VnicSetResource event notification 612 is running on that worker node. The VnicSetOperator 614 can create a cloud Vnic on the identified worker node and store the following data in the VnicSet resource object: a) the name of the worker node assigned by Kubernetes; b) the cloud-assigned worker node ID; c) the cloud-assigned Vnic ID; d) the MAC (Media Access Control) address of the cloud-assigned Vnic; e) the primary IP of the cloud-assigned Vnic; f) one or more VIPs assigned to the Vnic; and g) the CnApp owner. Finally, the VnicSet operator 614 can create a job on the identified worker node that injects the cloud Vnic into the target pod. The VnicSet operator 614 can mark itself as the owner of the resources created by the VnicSet operator 614 in order to perform resource cleanup.

[0086] In some examples, the VnicSet operator 614 can be deployed without a corresponding CnApp operator 618. For instance, an administrator can manually set up a StatefulSet along with its associated pods and resources, and submit the VnicSetResourceApp.yaml CRD file to the Kubernetes control plane 602 to trigger the creation and insertion of Vnics on the specified pods. The CnApp operator 618 provides additional automation and resource disposal capabilities, allowing resources for containerized software applications to be disposed of (e.g., scaled up or down), and associated Vnics to be created and managed dynamically. Figure 7 Example processing for handling deletion events involving CnApp operator 718 and VnicSet operator 714 is described.

[0087] Figure 7 This is a process flowchart of a system 700 configured to implement a virtual IP for a container pod, according to certain embodiments of this disclosure. Specifically, Figure 7 The diagram depicts a management module 704, a control plane 702, a CnApp operator 718, and a VnicSet operator 714 (which can correspond to...). Figure 4 The sequence of operations and data transfers between the control plane 402, CnApp operator 418, and VnicSet operator 414. Although for clarity... Figure 7Not shown in the diagram, but may also be found in components of system 700 and Figure 4 Information is exchanged between other components. The operation of system 700 can be an example delete CnApp method, including CnApp operator 718 and VnicSet operator 714, used to delete or release resources in a Kubernetes environment. The operations performed by CnApp operator 718 and VnicSet operator 714 can be executed by the corresponding coordinator module (e.g., by the finalizer module of each custom operator).

[0088] The management module 704 can issue a "Delete CnAppResource" command 706 or instruction to the control plane 702, for example, in response to a client or user instruction. The "Delete CnAppResource" command 706 can identify the name of the resource to be deleted, or even identify one or more resources to be deleted using a custom resource definition (CRD) YAML file for that CnApp resource. The CRD may include a ReplicaCount field for the number of pods, Vnic details, network name, Virtual IP (VIP) details, and the CnApp owner identifier. Based on the "Delete CnAppResource" command 706, the control plane 702 can provide a "Delete CnAppResource Event" notification 708 to the CnApp operator 718.

[0089] In response to the CnAppResource deletion event notification 708, the CnApp operator 718 may trigger the finalizer module of the coordinator 432 element of the CnApp operator 718. For the CnApp operator 718, the finalizer may determine the VnicSet resources associated with the CnApp resource to be deleted, such as by identifying the associated VnicSet resources for the pod associated with the CnApp resource. For example, the CnApp resource metadata may have a list of all mappings of podID and VnicSet resource ID. The finalizer may trigger one or more deletion events for all identified VnicSet resources for the VnicSet operator 714. These events can be triggered by sending a delete VnicSetResourceApp command 710 from the CnApp operator 718 to the control plane 702. The CnApp operator 718 may then exit the finalizer and continue deleting resources associated with or marked as owned by the CnApp operator 718, including CnApp resource 420, any StatefulSet and pod, Service, PVC, ConfigMap, Secret, etc. In some examples, the CnApp operator 718 can delete the CnApp resource element itself, which can trigger a Kubernetes process to remove standard K8s elements such as pods and services.

[0090] When control plane 702 receives the delete VnicSetResourceApp command 710, it can trigger a delete VnicSetResource event 712, which can be sent to or noticed by the VnicSet operator 714. The notification 712 may include the name or identifier of the VnicSet resource to be deleted, and in some examples may include data such as a VnicSetResourceApp.yaml file containing podID, network name, primary IP, secondary IP, and CnApp owner data.

[0091] In response to the VnicSetResource deletion event notification 712, the VnicSet operator 714 can perform several operations. In one example, the VnicSet operator 714 can invoke the finalizer of its coordinator module 432, which can identify the pod associated with the VnicSet resource from the podID field. The finalizer can assess the condition of the identified pod to determine how to dispose of the Vnic associated with that pod. If the pod is terminated (e.g., not running) or not found, the finalizer can retrieve all associated Vnics from the VnicSet resource (e.g., by VnicId) and delete or detach them one by one. However, if there are pending operations on the pod or worker node, or any temporary errors that could lead to a retry, it may be necessary to postpone the pod deletion. If the pod is still running (e.g., busy with operations), the deletion or detachment operation can be postponed by marking the pod in the VnicSet resource as to be deleted, which can be checked in the next coordination cycle after a certain delay after event 712 by performing requeuing. When Coordinator 432 is invoked, it can call the Deprecated Resource Disposer Component API to coordinate the state of any existing resources marked for deletion. When it is determined that the associated pod has been terminated or not found, it can delete or detach the Vnic.

[0092] Now go to Figure 8 This is a process flowchart of a system 800 configured to implement a virtual IP for a container pod, according to certain embodiments of this disclosure. Specifically, Figure 8 The control plane 802, the CnApp operator 818, and the VnicSet operator 814 (which can correspond to) are depicted. Figure 4 The sequence of operations and data transfers between the control plane 402, CnApp operator 418, and VnicSet operator 414. Although for clarity... Figure 8 Not shown in the diagram, but may also be found in components of system 800 and Figure 4 Information is exchanged between other components. The operation of system 800 can be a sample process for handling pod failures, crashes, or restarts in a Kubernetes environment via CnApp operator 818 and VnicSet operator 814. The operations performed by CnApp operator 818 and VnicSet operator 814 can be performed by the corresponding coordinator modules (e.g., by the legacy resource modules of each custom operator and the create or update resource modules).

[0093] This process can begin when a crashed pod is restarted via control plane 802 (804). Restarting a pod in Kubernetes when it crashes can include deleting the crashed pod and creating a new pod via the StatefulSet controller in the Kubernetes control plane 802. When a CnApp pod is a restarted or crashed pod, the CnApp operator 818 can be used to clean up the pod's resources (e.g., ConfigMap, Secret, PVC, VnicSet resources). Accordingly, control plane 802 can cause one or more CnApp StatefulSet update events 806 for StatefulSets owned by CnApp. The CnApp operator 818 can monitor triggered events for CnApp resources and StatefulSet resources owned by CnApp. Update event 806 can notify the CnApp operator 818 that the CnApp pod in the StatefulSet has crashed and a new pod has been created.

[0094] In response to event 806, the CnApp operator 818 may trigger the startup of its coordinator module at 808. The CnApp coordinator may first check for the coordination of stale resources, resulting in the deletion of Vnics associated with crashed or deleted pods via a deleteVnicSetResourceApp command 810 issued from the CnApp operator 818 to the control plane 802. The coordinator's creation and update module may then determine that there are new pods in a running state that require Vnics, resulting in the creation of VnicSet resources via a createVnicSetResourceApp command 812 issued from the CnApp operator 818 to the control plane 802.

[0095] In response to the delete VnicSetResourceApp command 810, control plane 802 can trigger a delete VnicResource event 816. VnicSet operator 814 can notice event 816 and trigger its own coordinator at 820 to delete any stale VnicSet resource IDs marked.

[0096] The `CreateVnicSetResourceApp` command 812 causes the control plane 802 to trigger the `CreateVnicResource` event 822. This allows the VnicSet operator 814 coordinator to create a Vnic at 822 and inject the Vnic into the newly created pod. For example, the VnicSet operator 814's create and update coordinator module can determine the worker node for the new pod, create a cloud Vnic on that worker node, and store detailed information in the VnicSet resource object, such as about... Figure 6 As described for Vnic creation. The VnicSet operator 814 creates and updates the coordinator module, which can then create jobs on worker nodes to inject the newly created Vnic into the new pod.

[0097] Once the crashed or faulty pod resources have been removed and the newly created pod has been updated with Vnic, the CnApp operator coordinator can exit at 824. Next, regarding... Figure 9 Describe the CnApp update event.

[0098] Figure 9 This is a process flowchart of a system 900 configured to implement a virtual IP for a container pod, according to certain embodiments of this disclosure. Specifically, Figure 9 The diagram depicts the management module 904, control plane 902, CnApp operator 918, and VnicSet operator 914 (which can correspond to...). Figure 4 The sequence of operations and data transfers between the control plane 402, CnApp operator 418, and VnicSet operator 414. Although for clarity... Figure 9 Not shown in the diagram, but may also be found in components of system 900 and Figure 4 Information is exchanged between other components. The operation of system 900 can be an example of handling CnApp update events in a Kubernetes environment via CnApp operator 918 and VnicSet operator 914. Operations performed by CnApp operator 918 and VnicSet operator 914 can be performed by the corresponding coordinator modules (e.g., legacy resource modules and create or update resource modules for each custom operator). Example scenarios for updating application pods can include changes to image names, replica sizes (e.g., the number of pods), or the addition, deletion, or updating of VIPs for Vnics injected into pods.

[0099] The management module 904 can issue an "Update CnAppResource" command 906 or instruction to the control plane 902, for example, in response to a client or user instruction or a Kubectl apply operation. The Update CnAppResource command 906 can include a CnAppResource Custom Resource Definition (CRD) YAML file for the CnApp resource. This file can include a ReplicaCount field for the number of pods, Vnic details and Vnic name, network name, and Virtual IP (VIP) details. Some attribute or parameter values ​​in the CnAppResource CRD file may differ from those initially submitted to create the resource (e.g., in...). Figure 6 The CRD in the middle has been changed. Based on the update CnAppResource command 906, the CnApp operator 918 can detect the "update CnAppResource event" notification 908 from the control plane 902.

[0100] In response to the CnAppResource update event notification 908, the CnApp operator 918 can trigger its coordinator 432 element. Depending on the type of update, the resource can be deleted or removed (e.g., by the obsolete resource handler module of coordinator 432), created or updated (e.g., by the create or update resource module), or both (e.g., the old version of the resource can be deleted, and the updated version can be created to replace it). Figure 9 In the illustrated example, CnApp operator 918 can send a delete VnicSetResourceApp command 912 to control plane 902 (e.g., from the obsolete resource disposal module) and send a create VnicSetResourceApp command 916 to control plane 902 (e.g., from the create / update module).

[0101] When control plane 902 receives the delete VnicSetResourceApp command 912, it can trigger a delete VnicResource event 920, which can be sent to or noticed by the VnicSet operator 914. In response to the delete VnicResource event notification 920, the VnicSet operator 914 can invoke its coordinator module 432 at 922, which can handle the deletion of the specified VnicSet resource (e.g., as per [reference to...]). Figure 7 (as described).

[0102] When control plane 902 receives the createVnicSetResourceApp command 916, it can trigger a createVnicResource event 924, which can be sent to or noticed by the VnicSet operator 914. In response to the createVnicResource event notification 924, the VnicSet operator 914 can invoke its coordinator module 432 at 926, which can handle the creation of the specified VnicSet resource (e.g., as per...). Figure 6 (as described).

[0103] In some examples, such as depending on the nature of update command 906, CnApp and VnicSet resources can be updated without deleting or creating resources (e.g., via the create or update resource module). Figures 10-15 A more detailed processing flow of the coordinator 532 component of the CnApp operator 418 and the VnicSet operator 414 is described, in which... Figures 10-12 The process of using the coordinator 532 of the CnApp operator 418, including the terminator 502, the obsolete resource handler 504, and the creation or updating of resource modules 506, is described.

[0104] Figure 10 This is a flowchart 1000 of an example method for implementing a virtual IP for a container pod, according to certain embodiments of this disclosure. Specifically, method 1000 may depict an example processing flow of the terminator component 502 of the coordinator module 432 of the CnApp operator 418. This method can be... Figure 4 The CnApp operator 418 can be implemented alone or in combination with other components of the system 400.

[0105] This method may include initiating a coordinator module or process at 1002. As described herein, the coordinator for an operator may be a processing loop for monitoring resources associated with the operator and bringing them from their current state to a desired state. The condition of a resource may change and require a return to the desired state, or the desired state may change (e.g., through user input) such that the resource should be changed from its current state to the updated desired state. The coordinator process may include a terminator element (process flow 1022), a stale resource disposer element, and a resource creation or update element. These elements may each perform a series of condition checks and potential changes on the resource, and may be executed in a selected order to appropriately remove, add, or adjust resources. Figures 10-12 In an example embodiment, the order in which the coordinator processes data can be determined from the order in which the finalizer ( Figure 10 It begins with ) and then proceeds to the legacy resource processor ( Figure 11 Then proceed to create or update the resource handler ( Figure 12Then exit the coordinator or restart the loop.

[0106] After the coordinator is started at 1002 and the process proceeds to CnApp finalizer handling flow 1022, the method may include checking at 1004 whether the CnApp application or instance is marked for deletion. Finalizer 1022 can check whether the CnApp is marked for deletion by determining whether Kubernetes has set a deletion timestamp for the CnApp. Based on the presence of the deletion timestamp, the method may include determining at 1006 whether the CnApp is marked for deletion. If no timestamp is set, the CnApp may not be marked for deletion, and the process may exit finalizer 1022 and proceed to the obsolete resource disposal module at 1020.

[0107] If a deletion timestamp has been set and the CnApp resource is marked for deletion, the method could include determining at 1008 whether the CnApp resource object contains a finalizer. For example, when a resource is created, a finalizer can be associated with that resource by adding a link or URL (Uniform Resource Locator) to the finalizer API call in the resource data. If no finalizer is available for the resource, this indicates that the resource is not a custom CnApp resource and can be disposed of by other Kubernetes processes. Accordingly, when no finalizer is available for the resource, the method could include exiting the CnApp coordinator at 1010 so that the resource can be disposed of by other Kubernetes systems. However, if the CnApp resource does contain a finalizer, the method could include reading the CnApp resource metadata at 1012 to obtain information such as the associated pod ID and VnicSet ID.

[0108] Based on this metadata, the method can then include an execution loop where, for each identified pod, the finalizer finds or determines the associated VnicSet resource at position 1014. For example, the CnApp resource metadata can provide information indicating which pods have been injected with or associated with a Vnic, as well as a specific identifier for each pod. The loop can then include deleting the associated VnicSet resource determined for each pod at position 1016. Deleting the VnicSet resource can include the CnApp operator sending or providing an instruction to the Kubernetes control plane for deleting the VnicSet resource (e.g., this might cause the control plane to set a deletion timestamp for the specified VnicSet resource). Sending the deletion instruction to the control plane can trigger an event that causes the VnicSet operator to delete the specified VnicSet resource.

[0109] Once the loop for each pod is complete, the method can include removing the finalizer at 1018 to clean up the CnApp resource. Without a finalizer hook, the Kubernetes control plane can clean up standard Kubernetes resources. Cleanup operations can include deleting the StatefulSet or pod, Service, PVC, ConfigMap, Secret, and other resources associated with the CnApp resource. At this point, the method can then exit finalizer 1022 and proceed to the obsolete resource disposal module at 1020. About Figure 11 The example implementation discusses the operation of legacy resource processors.

[0110] Figure 11 This is a flowchart 1100 of an example method for implementing a virtual IP for a container pod, according to certain embodiments of this disclosure. Specifically, method 1100 may depict an example processing flow of the legacy resource handler component 504 of the coordinator module 432 of the CnApp operator 418. This method can be... Figure 4 The CnApp operator 418 can be implemented alone or in combination with other components of the system 400.

[0111] The processing flow of this method can begin at point 1102, where the processing flow of the CnApp terminator ends. At this point, the stale CnApp resource disposer processing flow 1118 can begin. This method may include reading or obtaining CnApp resource metadata at point 1104. This metadata may include information such as a list of pods and any associated Vnics (VnicSet), as well as the ID of each pod.

[0112] Based on this metadata, the method can then include an execution loop where, for each identified pod ID, the stale resource handler 1118 checks the pod status. A determination can be made at 1108 whether the pod status is terminated or not found. If not, the pod is likely running normally, and the method can include doing nothing at 1110. However, if the pod status is terminated or not found, the method can include deleting the associated VnicSet resource at 1112 and then removing the pod entry from the CnApp metadata at 1114. Similar to the finalizer, deleting a VnicSet resource can include the CnApp operator sending or providing an indicator to the Kubernetes control plane for deleting the VnicSet resource (e.g., causing the control plane to set a deletion timestamp for the specified VnicSet resource), which can trigger an event that causes the VnicSet operator to delete the specified VnicSet resource. Once the loop completes for all identified pods, the method can exit the stale CnApp resource handler 1118 and continue to the create or update CnApp resource module at 1116. About Figure 12 The example implementation discusses the operations for creating or updating CnApp resource modules.

[0113] Figure 12 This is a flowchart 1200 of an example method for implementing a virtual IP for a container pod, according to certain embodiments of this disclosure. Specifically, method 1200 may depict an example processing flow of the coordinator module 432 of the CnApp operator 418 creating or updating the CnApp resource handler component 506. This method can be... Figure 4 The CnApp operator 418 can be implemented alone or in combination with other components of the system 400.

[0114] The process flow of this method can begin at point 1202, where the processing flow of the obsolete CnApp resource handler ends. At this point, the creation or updating of the CnApp resource processing flow 1238 can begin. This method may include accessing the CnApp resource's Custom Resource Definition (CRD) file at point 1204, and if a finalizer is not present, it may include adding a finalizer to the CnApp resource (e.g., a link to finalizer processing). At point 1206, if a tag is not present in the CnApp metadata, the method may include adding a tag to the CnApp metadata (e.g., adding a CnApp tag "owner = cnapp-operator"). For example, the owner tag may not be a mandatory component of a user-provided CRD, but it can be added to the CnApp resource for resource cleanup and management.

[0115] At point 1208, it's possible to determine whether a StatefulSet (or an appropriate number of pods, ReplicaSets, etc.) has been created. If so, the method can include checking and coordinating the StatefulSet's properties (e.g., replica count or image changes).

[0116] However, if a StatefulSet has not yet been created, the method can include starting a loop to create the appropriate StatefulSet(s), pod(s), and associated resources. This loop can include creating a Service for each pod at 1212 (if the Service exists in the CRD yaml file). At 1214, the method can include adding a tag to each Service (e.g., a Service tag with "owner=cnapp-operator"). The method can include creating the StatefulSet and associated pod at 1216, and then adding metadata to the StatefulSet and pod(s) at 1218. For example, the metadata could be a StatefulSet tag with "owner=cnapp-operator", and tags like "owner = cnapp-operator", "vnics_needed = ...<number_of_vnics> "and "svc_label =<svc_name> The pod tag is now added. At this point, the loop can end.

[0117] Once one or more StatefulSets have been created or reconciled, the method can enter another loop for each pod, starting at 1220 to determine if a VnicSet resource has been created for each pod. If not, the method can include creating the VnicSet resource YAML file at 1222 and adding the VnicSet to the CnApp metadata (e.g., by associating the VnicSet ID with each pod ID). The method can then include applying the VnicSet resource YAML at 1226, which can trigger a Create VnicSet event. The Create VnicSet event can include the CnApp operator sending or providing an indicator to the Kubernetes control plane for creating the VnicSet resource, which can trigger an event that causes the VnicSet operator to create and inject the specified VnicSet resource. The VnicSet operator can also associate a VIP with the injected Vnic.

[0118] Alternatively, if the appropriate number of VnicSet resources have already been created for each pod at 1220, the loop can include checking and coordinating the Virtual IP Address (VIP) associated with the Vnic at 1228. A determination can be made at 1230 whether the VIP has changed. If not, the VnicSet resources can be left unchanged at 1232. If the VIP has changed, it can trigger an update VnicSet event at 1234. Again, this update event can include the CnApp operator sending or providing an indicator to the Kubernetes control plane for updating the VnicSet resources, which can trigger an event causing the VnicSet operator to update the specified VnicSet resources. Once the desired VnicSet resources have been created or updated for each pod, the loop can end, and the CnApp resource creation or update processing at 1238 can also end. With the completion of the CnApp resource creation or update processing at 1238, the coordinator processing flow can also complete, so the coordinator can exit at 1236 or loop back to the finalizer. Figures 13-15 Example processing of coordinator 532, terminator 502, obsolete resource handler 504, and resource creation or update module 506 for VnicSet operator 414 is described.

[0119] Figure 13 This is a flowchart 1300 of an example method for implementing a virtual IP for a container pod, according to certain embodiments of this disclosure. Specifically, method 1300 may depict an example processing flow of the terminator component 502 of the coordinator module 432 of the VnicSet operator 414. This method can be... Figure 4 The VnicSet operator 414 can be implemented alone or in combination with other components of the system 400.

[0120] This method may include initiating a coordinator module or process at 1302. The coordinator process may include a terminator element (process flow 1326), a stale resource disposer element, and a resource creation or update element. These elements may each perform a series of status checks and potential changes on the resources, and may be executed in a selected order to appropriately remove, add, or adjust resources. Figures 13-15 In an example embodiment, the order in which the coordinator processes data can be determined from the order in which the finalizer ( Figure 13 It begins with ) and then proceeds to the legacy resource processor ( Figure 14 Then proceed to create or update the resource handler ( Figure 15 Then exit the coordinator or restart the loop.

[0121] After the coordinator is started at 1302 and the process proceeds to VnicSet finalizer flow 1326, this method may include checking at 1304 whether the VnicSet resource is marked for deletion. Finalizer 1326 can check whether the VnicSet resource is marked for deletion by determining whether Kubernetes has set a deletion timestamp for the resource. Based on the presence of the deletion timestamp, this method may include determining at 1306 whether the VnicSet resource is marked for deletion. If no timestamp is set, the VnicSet resource may not be marked for deletion, and this process may exit finalizer 1326 and proceed to the obsolete resource disposal module at 1324.

[0122] If a deletion timestamp has been set and the VnicSet resource is marked for deletion, the method may include determining at 1308 whether the VnicSet object contains a finalizer. If not, the resource may not be a custom VnicSet resource, and the method may include exiting the VnicSet coordinator at 1310 to allow the Kubernetes system to dispose of the resource. However, if the VnicSet object does contain a finalizer, the method may include reading the VnicSet resource metadata at 1312 to obtain information such as the associated pod ID and Vnic ID.

[0123] Based on this metadata, the method can then include checking the pod status of the pod ID from the metadata at 1314. A determination can be made at 1316 whether the pod status is terminated or not found. If so, the method can include detaching or deleting all(one or more) Vnics associated with that pod ID at 1318 and removing the finalizer at 1320 to clean up the VnicSet resources. However, if the pod status is not terminated or not found (e.g., if the pod is still running), the method can include not deleting the Vnics(one or more) associated with the pod ID and requeuing the deletion event at 1322 (e.g., by enabling the "tombstone" flag). For example, deleting a Vnic on a running pod might interrupt ongoing operations and cause service loss or errors, so deletion can be delayed until the pod is terminated or not found. Once the Vnic is deleted, or the deletion event is requeued, the method can exit finalizer processing at 1326 and proceed to the obsolete VnicSet resource disposal module at 1324. (About...) Figure 14 The example implementation discusses the operation of the legacy VnicSet resource handler.

[0124] Figure 14This is a flowchart 1400 of an example method for implementing a virtual IP for a container pod, according to certain embodiments of this disclosure. Specifically, method 1400 may depict an example processing flow of the legacy resource disposal component 504 of the coordinator module 432 of the VnicSet operator 414. This method can be... Figure 4 The VnicSet operator 414 can be implemented alone or in combination with other components of the system 400.

[0125] The processing flow of this method can begin at point 1402, where the processing flow of the VnicSet finalizer ends. At this point, the deprecated VnicSet resource disposer processing flow 1418 can begin. This method may include reading or obtaining VnicSet resource metadata at point 1404. This metadata may include information such as the associated pod ID and Vnic ID.

[0126] Based on this metadata, the method can then include checking the pod status of the pod ID from the metadata at 1406. A determination can be made at 1408 whether the pod status is terminated or not found. If so, the method can include detaching or deleting all(one or more) Vnics associated with that pod ID at 1412 and updating the VnicSet resource metadata at 1414 to reflect the deletion(one or more). However, if the pod status is not terminated or not found (e.g., if the pod is still running), the method can include doing nothing at 1410 and not deleting or detaching any(one or more) Vnics used for that pod. After disposing of(one or more) Vnics based on the pod status, the method can exit the stale VnicSet resource disposer processing at 1418 and proceed to the create or update VnicSet resource module at 1416. About Figure 15 The example implementation discusses the operations of creating or updating the VnicSet resource module.

[0127] Figure 15 This is a flowchart 1500 of an example method for implementing a virtual IP for a container pod, according to certain embodiments of this disclosure. Specifically, method 1500 may depict an example processing flow of the creation or updating of the VnicSet resource handler component 506 by the coordinator module 432 of the VnicSet operator 414. This method can be... Figure 4 The VnicSet operator 414 can be implemented alone or in combination with other components of the system 400.

[0128] The process flow of this method can begin at point 1502, where the process flow of the legacy VnicSet resource disposer ends. At this point, the creation or updating of the VnicSet resource process flow 1516 can begin. This method may include, at point 1504, accessing the VnicSet resource custom resource definition (CRD) file, which indicates the pods for which Vnics should be created, and how many Vnics should be created. This method may also include, at point 1506, determining the worker nodes running within the specified pods.

[0129] The method can then initiate a loop to process each Vnic to be created for the pod. At 1508, the method may include creating the cloud Vnic on the determined worker node and saving metadata for that Vnic (e.g., by associating the Vnic ID of the created Vnic with the pod ID). The method may then include creating a job at 1510 on the same worker node to inject the Vnic into the selected pod, which may result in the cloud Vnic being injected into the pod at 1512.

[0130] Once the desired Vnic is inserted into the selected pod, the loop can end, and the VnicSet creation or update resource processing step 1516 can also end. With the completion of VnicSet creation or update resource processing step 1516, the coordinator processing flow can also complete, and the coordinator can exit at step 1514 or loop back to the finalizer. About Figure 16 Discuss the operations of the coordinator 432 and the CnApp operator 418 in terms of Service behavior.

[0131] Figure 16A diagram illustrating a system 1600 configured to implement virtual IPs for container pods according to certain embodiments of this disclosure is provided. Specifically, system 1600 may depict example customizable Service behavior to enable per-pod Services or StatefulSet-based Services. System 1600 may include multiple media pods 1602 configured for media management, and a StatefulSet Sig-CRD 1604 with a replica count of 2 to contain two pods for signaling management, Sig-CRD-0 1606 and Sig-CRD-1 1608. In an example embodiment of system 1600, each signaling pod of StatefulSet 1604 has a corresponding Service, where pod Sig-CRD-0 has Service Sig-CRD-0 1610, and pod Sig-CRD-1 has Service Sig-CRD-1 1612. System 1600 may be implemented via CnApp operator 418 and associated coordinator 432, nodes 422-426, and... Figure 4 Other components are used to implement and manage this.

[0132] According to default Kubernetes behavior, support for a cluster IP Service for a StatefulSet (which may contain multiple pods) results in a single Service for all pods in that StatefulSet. A Service in Kubernetes can be a way to expose a networked application running as one or more pods in a cluster. Therefore, for example, a signaling Service provided by a Sig-CRD StatefulSet 1604 can be available to other applications (e.g., media pod 1602) or external clients via signaling services 1610-1612. In general Kubernetes applications, pods are transient, and therefore a Service for a StatefulSet or Deployment allows a frontend pod 1602 to access functionality provided by pods 1606-1608 without having to keep track of individual pods and their associated IP addresses (which may be changing). However, the approach presented in this paper involves making pods and their associated IP addresses more consistent and knowable. Therefore, the CnApp operator 418 can be configured to support customizable Service behavior, either applying Kubernetes' default per-StatefulSetService functionality or enabling per-pod Service functionality.

[0133] The CnApp coordinator 418 CRD yaml file can include a "perpod" variable (e.g., under the Service subsection). When CnApp coordinator 432 is invoked and perpod is set to False, a Service can be created with its name matching the StatefulSet name tag, and this Service will be associated with all pods in that StatefulSet. However, when CnApp coordinator 432 is invoked with perpod set to True or not set (making the default perpod=true), a Service can be created for each pod in the StatefulSet, with each Service's name matching the corresponding pod name (e.g., where the pod name can be derived from StatefulSetName-index). The Service's port and other details can be specified under the Service subsection of the CnApp CRD yaml file. In the perpod implementation, each frontend pod 1602 can connect to each Service 1610-1612 to access the functionality of the pods in StatefulSet 1604. About Figures 17-19 The discussion covers the handling of assigning Virtual IP Addresses (VIPs) to Vnics.

[0134] Figure 17 A diagram depicts a system 1700 configured to implement virtual IPs for container pods according to certain embodiments of this disclosure. Specifically, system 1700 may depict an example implementation for providing VIP support in pods within a private environment (as opposed to a public cloud environment). System 1700 may include a VnicSet operator 1714 that extends the functionality of the Kubernetes control plane 1702. System 1700 may also include a first worker node (worker node 11722) on which a first pod (pod-1 1704) is running. System 1700 may also include a second worker node (worker node 2 1724) on which a second pod (pod-2 1706) and a third pod (pod-3 1708) are running. VnicSet operator 1714 can manage multiple resources, including VnicSet resource 1 1716 associated with pod-1 1704, VnicSet resource 2 1718 associated with pod-2 1706, and VnicSet resource 3 1720 associated with pod-3 1708. Elements of system 1700 may correspond to or be implemented by elements of system 400.

[0135] Such as about Figure 3The first part of the described example three-part scheme for implementing cnSBC includes CnApp operator 418 starting application pod 428 and creating a trigger event for Vnic creation by generating VnicSet resource 416. The second part includes VnicSet operator 414 being configured to handle the trigger event and create and inject the Vnic into pod 428. The third part includes VnicSet operator 1714 enabling VIP support on the Vnic, thereby enabling failover or switching scenarios between pods with a fixed VIP that is always reachable from the external network. VIP and failover support can be implemented automatically.

[0136] VIP support for pods in StatefulSets, Deployments, Jobs, or other resources can provide pods with static IPs (even after a pod restart), thus solving the problem of ephemeral IPs for pods in Kubernetes. Unlike native Kubernetes networking, VIPs and VMACs (Virtual MAC addresses) can also provide high availability (HA) support within pods. HA support can be achieved by assigning the same IP to two or more pods working in primary and standby roles (e.g., N primary pods to M standby pods), so that if the primary pod fails, the standby pods can immediately take over using the same IP address. Multiple VIP interfaces can be created within a single pod, such as by injecting multiple Vnics, each with an associated VIP assigned to it, or the same Vnic having multiple assigned VIPs (e.g., dual-stack), or combinations thereof. Example use cases for VIP support can include handling SIP (Session Initiation Protocol) or RTP (Real-Time Transport Protocol) communication, which may require IP addresses to be known at the remote end for communication, and the IP addresses should remain consistent during restarts, as SIP messages may be sent on the same IP address.

[0137] As described herein, CnApp operator 418 can create a pod, determine the number of Vnics to be included in that pod, and then generate a VnicSet resource YAML file. Based on the VnicSet resource YAML, VnicSet operator 1714 can generate the desired number of Vnics on worker nodes with the identified pod-IDs. Furthermore, the VnicSet operator can be authenticated by a cloud service provider and determine at least one static IP address, MAC address, VIP(s), and other information associated with the Vnics. Vnic details can be provided with the VnicSet resource YAML file, assigned by the cloud service provider or DHCP, or a combination thereof.

[0138] For example, the VnicSet operator 1714 can reserve a certain range of IP addresses as an IP pool for each subnet for Vnic injection from that subnet. This IP range can be defined by the user, in a ConfigMap, in the VnicSet operator definition file, or by other methods for the VnicSet operator 1714. Once the VnicSet operator 1714 starts, it can allocate and mask this range for VIP management. The VIP pool can be a reserved subset of a larger IP address range (which can be used to assign primary (e.g., fixed) addresses to Vnics) so that primary addresses are not assigned from the reserved VIP pool. Furthermore, the reserved IP range can be dynamically maintained by the VnicSet operator 1714 during restarts, crashes, allocations, cancellations, or reallocations.

[0139] Once the Vnic for injection is created, the VIP can be enabled and associated with the Vnic and pod based on the VnicSet resource CRD definition file. In some examples, all pods associated with the target application can have a Vnic with the same VIP. This VIP can be enabled only on the primary pod. Multiple Vnics with a VIP can be injected to provide multiple interface support, for example, for network isolation. The same VIP can be associated with both the primary and standby pods to make the same interface available during pod switching or failover events.

[0140] In private environments (such as OpenStack), traffic to assigned VIPs can be handled via GARP (Granted Address Resolution Protocol). For example, when the VnicSet operator 1714 makes an OpenStack API call for authentication, it can identify the Vnic and the associated static IP and VIP for both the primary and standby pods. GARP and ARP tables can manage which VIPs are primary on which pods. For example, when multiple primary pods have Vnics with the same VIP, GARP can manage which traffic is routed to which Vnic and pod. GARP can also automatically manage transitions from standby to primary, and the ARP table can be updated when a pod becomes non-primary or primary. For example, if pod 21706 is about to crash and pod 3 1708 becomes primary to compensate, an internal GARP request can be issued to activate the VIP assigned to the newly activated pod. Accordingly, GARP can ensure that pod 3 1708 will start using the same VIP as the stopped pod to receive appropriate traffic.

[0141] Example VnicSet resource type patterns for Kubernetes may include:

[0142] type VnicSetResource {

[0143] / / The name of the pod that Vnic will attach to

[0144] PodName string 'json:"podname"'

[0145] / / The pod ID that Vnic will attach to

[0146] PodID string 'json:"podid"'

[0147] / / Vnic's array

[0148] Vnics []VnicType 'json:"vnics"'

[0149] }

[0150] / / VnicType data

[0151] type VnicType struct {

[0152] / / Network Name

[0153] NetworkName string `json:"network-name"`

[0154] / / List of secondary IPs / virtual IPs

[0155] VirtualIPs []string `json:"virtualIps"` <<<< VIP support

[0156] }

[0157] Example CRD resource definition files for VnicSet may include:

[0158] apiVersion: <group> / <domain.com> / v1

[0159] kind: VnicSet

[0160] spec:

[0161] podid: <pod-id>

[0162] nickname: <pod-name>

[0163] vnics:

[0164] - network-name: <network-name-1>

[0165] fixedIP: <ip1>

[0166] VirtualIps: <IP List> <<< VIP Support

[0167] - network-name: <network-name-2>

[0168] fixedIP: <ip2>

[0169] VirtualIps: <IP List> <<< VIP Support

[0170] The primary or fixed IP for a Vnic can optionally be provided to the VnicSet Operator 1714 from the CnApp Operator 418. Note that if no IP is provided in the CRD file, an IP address that is externally reachable IP can be dynamically assigned via DHCP. In addition to the fixed IP, multiple VIPs can be assigned to a single Vnic. The primary IP can be assigned from a first allocation range and a secondary IP range (provided in a ConfigMap, for example) can be reserved by the IP Management Module (IPMM) 1710 of the VnicSet Operator 1714 for the application to use as VIPs.

[0171] During initialization of the VnicSet Operator 1714, the IPMM 1710 can reserve a pool of virtual / secondary IPs to ensure that the network does not assign overlapping IPs to interfaces (e.g., the primary IP is assigned from this pool of VIPs). Once reserved, these IPs can only be used within the purview of the VnicSet Operator. There can be a fixed naming convention for creating a Vnic, port, or interface to ensure that the port can be retrieved in the event of a reboot of the VnicSet Operator 1714. For example, a combination of "Name,” "DeviceOwner,” and "DeviceID” can be set during creation of the port to ensure that it will be unique when the port is retrieved. The following is an example naming convention that can be used when creating a port:

[0172] Name: VnicSet-Operator-<operator name as mentioned in the Deployment>-Port

[0173] DeviceOwner: VnicSet-Operator

[0174] DeviceID: VnicSet-Operator-network-name

[0175] When the VnicSet operator 1714 receives a creation event (which has a CRD specification containing details of the identified pod 1704 and associated Vnics), it can identify the worker node 1722 on which pod 1704 is running. The VnicSet operator 1714 can create one or more Vnics (e.g., Vnic1 and Vnic2) on that worker node 1722 and associate the primary (fixed) and secondary (virtual) IPs configured in the CRD specification with the Vnics. The VnicSet operator 1714 can create a job to inject the created Vnics from worker node 1722 into pod 1704. In the depicted example, the VnicSet operator 1714 can generate a VnicSet resource 11716 for pod-1 1704, detailing that Vnic1 is associated with fixed IP1, VIP1, and VIP2, and Vnic2 is associated with fixed IP2 and VIP3. Since pod 2 (1706) and pod 3 (1708) can be associated with the same application as pod 1 (1704), Vnic1 and Vnic2 for these pods can also be associated with VIP 1, VIP 2, and VIP 3, but with different fixed IPs. The VnicSet operator 1714 can monitor the lifecycle events of pods 1704-1708, and if a pod restarts, operator 1714 can associate the same VIP with the Vnic created on the target worker node and inject it into the restarted pod.

[0176] When the VnicSet operator 1714 receives an update event CRD, it can monitor this change and update the IP address in the associated Vnic if the administrator updates the VIP settings in any network (e.g., by modifying VnicSet resource files 1716-1720, by deleting an existing Vnic and creating a new one, or by creating a new Vnic and adding it to an existing pod). Upon reboot of the VnicSet operator 1714, the IPMM 1710 can retrieve the created Vnic or port information from the hypervisor and create a new record for the retrieved Vnic ID. This information can be used to delete the associated port during operator cleanup operations.

[0177] In the example depicting System 1700, pod 1 (1704) and pod 2 (1706) can be in the primary state, while pod 3 (1708) can be in the standby state. This provides a primary:standby pod N:M ratio of 2:1. If pod 1 (1704) or pod 2 (1706) crashes, pod 3 (1708) can switch from standby to primary. For example, the application controller can detect the crash and pass it to Kubernetes via the service manager. Because pod 3 (1708) shares the VIP address with pod 1 (1704) and pod 2 (1706), it can immediately take over the traffic that would otherwise be handled by the crashed pod (e.g., handled by GARP and updating the ARP table) while the crashed pod restarts. Figures 18-19 It solves the application of Vnic and VIP in public cloud environments that may not rely on GARP.

[0178] Figure 18 A diagram depicts a system 1800 configured to implement virtual IPs for container pods according to certain embodiments of this disclosure. Specifically, system 1800 may depict an example implementation for providing VIP support in pods within a public cloud environment (as opposed to a private environment). System 1800 may include a VnicSet operator 1814 that extends the functionality of the Kubernetes control plane 1802 and interacts with a public cloud control plane 1826. System 1800 may also include a first worker node (worker node 1 1822) on which a first pod (pod-1 1804) is running. System 1800 may also include a second worker node (worker node 2 1824) on which a second pod (pod-2 1806) and a third pod (pod-3 1808) are running. VnicSet operator 1814 can manage multiple resources, including VnicSet resource 1 1816 associated with pod-1 1804, VnicSet resource 2 1818 associated with pod-2 1806, and VnicSet resource 3 1820 associated with pod-3 1808. Elements of system 1800 may correspond to or be implemented by elements of system 400.

[0179] In public cloud environments, such as platforms like OCI (Oracle Cloud Infrastructure), AWS (Amazon Web Services), and Microsoft Azure, the mechanisms for creating Vnics may differ from those in private environments. Furthermore, public cloud implementations may not rely on GARP to handle traffic failover between a crashed primary and standby pod. Instead, the VnicSet operator 1814 can issue specific requests to the public cloud control plane 1826 to enable or disable specific VIPs on a pod. Accordingly, dynamic automation schemes for creating pods with externally reachable interfaces, IP addresses with restart and failover resilience, and identical ingress and egress paths bypassing the Kubernetes load balancer can be implemented in both private environments and public cloud platforms. The examples provided in this article may be for StatefulSet resources, but these ideas can be extended to Deployments or other resources.

[0180] In a public cloud environment, the VnicSet operator 1814 can perform the same operations discussed above regarding the management of VnicSet resources, monitor the lifecycle events of one or more target pods, and take appropriate actions via the coordinator. The VnicSet operator 1814 can still enable VIPs, associate VIPs with Vnics, and inject Vnics into pods.

[0181] However, differences can exist. The primary IP, or fixed IP, used for the Vnic can be a mandatory field in the VnicSet resource CRD file, rather than an optional field. The VnicSet operator 1814 can authenticate to the public cloud (rather than a private cloud environment). Once the Vnic for injection is created, the VnicSet operator 1814 can request the public cloud 1826 to enable the VIP and associate the VIP only with the Vnic ID of the primary pods 1804-1806 (e.g., unlike in a private environment where the VIP can be associated with a standby pod, and the switch from standby to primary can be handled automatically via GARP). The primary pods 1804-1806 can be determined based on the primary:standby count provided in the CnApp resource CRD file, which may have fields for the primary and standby counts for the pods. If the public cloud 1826 does indeed enable the VIP for the Vnic, the VnicSet operator 1814 coordinator can proceed to the next step. However, in the event of failure, the VnicSet operator 1814 coordinator can clean up the Vnic and requeue the coordination request. During pod restarts, updates, or crashes, when primary pods 1804-1806 stop, the VnicSet operator 1814 can request the public cloud control plane 1826 to associate the VIP of the old primary pods 1804-1806 with the standby pod 1808. This standby pod will then take over the role of the stopped primary pods. Simultaneously, the old primary pods can be regenerated as standby pods with a Vnic without a VIP.

[0182] In the example of system 1800, the VnicSet operator 1814 can detect creation notifications for Vnics on pods 1804-1808, with a CRD specifying the fixed IP value of the Vnic. Accordingly, VnicSet resource 1 1816 can associate Vnic1 (with VnicId1) with fixed IP1 and VnicId2 with fixed IP2; VnicSet resource 2 1818 can associate VnicId3 with fixed IP3 and VnicId4 with fixed IP4; and VnicSet resource 3 1820 can associate VnicId5 with fixed IP5 and VnicId6 with fixed IP6.

[0183] The VnicSet operator 1814 can send a request to the public cloud control plane 1826 based on the primary:standby ratio to associate VIPs with the primary pod Vnic. The VnicSet operator 1814 can specify which VIPs are associated with which Vnics, for example, based on information provided in the CnApp resource CRD file and also provided to the VnicSet operator in the VnicSet resource CRD file. In this case, the request to the public cloud control plane could be: associating VnicId1 with VIP1 and VIP2; associating VnicId2 with VIP3; associating VnicId3 with VIP4 and VIP5; and associating VnicId4 with VIP6. VnicId5 and VnicId6 of the standby pod 3 1808 may not have VIP association requests. The public cloud control plane 1826 can provide a successful response (or alternatively, a failed response or no response, to which the VnicSet operator 1814 can clean up one or more Vnics and requeue the association request). After successful association, a fixed IP and VIP can be associated with a Vnic, as described in system 1800, and the Vnic can be injected into the associated pods 1804-1808 by the VnicSet operator 1814.

[0184] Now suppose pod 2 1806 crashes. The VnicSet operator 1814 can detect this event, and its coordinator can remove the associated VnicSet resource 2 1818, thus freeing up the Vnics (VnicId3 and VnicId4). Regarding... Figure 19 Discussions resumed after the crash.

[0185] Figure 19 A diagram illustrating a system 1900 configured to implement virtual IPs for container pods according to certain embodiments of the present disclosure is depicted. Specifically, system 1900 may depict an example implementation for handling switchover or failover events between a primary pod and a standby pod in a public cloud environment. System 1900 may include a VnicSet operator 1914 that extends the functionality of the Kubernetes control plane 1902 and interacts with a public cloud control plane 1926. System 1900 may also include worker node 1 1922 running pod-1 1904, and worker node 2 1924 running pod-2 1906 and pod-3 1908. VnicSet operator 1914 can manage multiple resources, including VnicSet resource 1 1916 associated with pod-1 1904, VnicSet resource 2 1918 associated with pod-2 1906, and VnicSet resource 3 1920 associated with pod-3 1908. The components of system 1900 may correspond to or be implemented by the components of system 400 and system 1800.

[0186] After pod 2 1906 crashes and is deleted, pod 2 1906 can be regenerated (e.g., by the CnApp operator). The newly generated pod 2 1906 can have a new pod ID, pod-id4. The VnicSet operator 1914 can create a new VnicSet resource 2 1918 and create a new Vnic on worker node 2 1924, which was regenerated from pod 2 1906. The new Vnic can have a new VnicId (e.g., VnicId7 and VnicId8) but with the same static IPs (static IP3 and static IP4). The role or status of pod 2 1906 can be switched from primary to standby.

[0187] Pod 3 (1908) can take over the primary role, and the VnicSet operator (1914) can send a request to the public cloud control plane (1926) to associate VnicId5 with VIP4 and VIP5, and VnicId6 with VIP6, which was previously associated with pod 2 (1906). Pod 2 (1906) may now have no associated VIPs. Therefore, pod 3 (1908) can take over the handling of calls or messages that were handled by pod 2 (1906) before the crash, without service interruption.

[0188] The VnicSet operator 1914 can also contact the public cloud control plane 1926 when VIPs need to be adjusted in other ways. For example, in the case of an operator CRD update (where an administrator updates the set of reserved VIPs in any network), the VnicSet operator 1914 can detect this change and request the public cloud control plane 1926 to update the IP addresses in one or more associated Vnics. About Figure 20 A computational system configured to perform the operations described in the above figures and descriptions is described.

[0189] Figure 20 The illustration shows an apparatus 2000 including a computing system 2001, which represents any system or collection of systems in which the various processes, systems, programs, services, and scenarios disclosed herein can be implemented. For example, the computing system 2001 could be... Figure 1 Network 106 or Kubernetes environment 102, or Figure 4 Examples of the control plane 402, application plane 406, worker nodes 422-426, VnicSet operator 414, CnApp operator 418, API server 410, or other components, or any combination thereof. Examples of the computing system 2001 include, but are not limited to, desktop computers, laptop computers, server computers, routers, web servers, cloud computing platforms and data center equipment, and any other type of physical or virtual server machine, physical or virtual router, container, or any variation or combination thereof.

[0190] The computing system 2001 can be implemented as a single device, system, or apparatus, or it can be implemented in a distributed manner as multiple devices, systems, or apparatuses. The computing system 2001 may include, but is not limited to, a processing system 2002, a storage system 2003, software 2005, a communication interface system 2007, and a user interface system 2009. The processing system 2002 can be operatively coupled to the storage system 2003, the communication interface system 2007, and the user interface system 2009.

[0191] Processing system 2002 can load and execute software 2005 from storage system 2003. Software 2005 may include and implement VIP processing 2006, which may represent any operation for implementing an externally accessible, persistent IP interface for a containerized software environment, as discussed with respect to the foregoing figures. When executed by processing system 2002, software 2005 may instruct processing system 2002 to operate as described herein to at least implement the various processing, operational scenarios, and sequences discussed in the foregoing embodiments. For brevity, computing system 2001 may optionally include additional devices, features, or functions not discussed.

[0192] In some embodiments, the processing system 2002 may include a microprocessor and other circuitry for retrieving and executing software 2005 in the storage system 2003. The processing system 2002 may be implemented within a single processing device, but may also be distributed across multiple processing devices or subsystems that collaboratively execute program instructions. Examples of the processing system 2002 may include a general-purpose central processing unit, a graphics processing unit, a dedicated processor and logic device, and any other type of processing device, combination thereof, or variations thereof.

[0193] Storage system 2003 may include any memory device or computer-readable storage medium that is readable by processing system 2002 and capable of storing software 2005. Storage system 2003 may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Examples of storage media include random access memory, read-only memory, magnetic disks, optical disks, optical media, flash memory, virtual and non-virtual memory, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other suitable storage medium. In no case is a computer-readable storage medium used for transmitting signals.

[0194] In addition to a computer-readable storage medium, in some embodiments, the storage system 2003 may also include a computer-readable communication medium through which at least some of the software 2005 can communicate internally or externally. The storage system 2003 may be implemented as a single storage device, but may also be implemented across multiple storage devices or subsystems co-located or distributed relative to each other. The storage system 2003 may include additional elements, such as a controller, capable of communicating with the processing system 2002 or potentially other systems.

[0195] Software 2005 (which, among other functions, includes VIP processing 2006) can be implemented in program instructions that, when executed by processing system 2002, instruct processing system 2002 to operate as described herein in various operating scenarios, sequences, and processes.

[0196] Specifically, program instructions may include various components or modules that cooperate or otherwise interact to perform the various processing and operational scenarios described herein. These components or modules may be implemented in compiled or interpreted instructions, or in some other variation or combination of instructions. These components or modules may execute synchronously or asynchronously, serially or in parallel, in a single-threaded environment or in multiple threads, or according to any other suitable execution paradigm, variation, or combination thereof. Software 2005 may include additional processing, programs, or components, such as operating system software, virtualization software, or other application software. Software 2005 may also include firmware or some other form of machine-readable processing instructions executable by processing system 2002.

[0197] Generally, when software 2005 is loaded into and executed by processing system 2002, it can transform a suitable apparatus, system, or device (represented by computing system 2001) from a general-purpose computing system into a dedicated computing system tailored to implement the systems and processes described herein. In practice, software 2005 encoded on storage system 2003 can transform the physical structure of storage system 2003. Specific transformations of the physical structure can depend on various factors in the different embodiments described herein. Examples of such factors may include, but are not limited to, the technology used to implement storage system 2003, whether the computer storage medium is characterized as primary or secondary storage, and other factors.

[0198] For example, if the computer-readable storage medium is implemented as a semiconductor-based memory, then software 2005 can, when program instructions are encoded therein, transform the physical state of the semiconductor memory, such as by changing the state of the transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory. Similar transformations may occur with respect to magnetic or optical media. Other transformations of the physical medium are also possible without departing from the scope of this description; the foregoing examples are provided only for the convenience of this discussion.

[0199] The communication interface system 2007 may include communication connections and devices that allow communication with other computing systems (not shown) via a communication network (not shown). Examples of connections and devices that commonly allow inter-system communication may include network interface cards, antennas, power amplifiers, radio frequency (RF) circuitry systems, transceivers, and other communication circuitry systems. The connections and devices may communicate via a communication medium to exchange communication with other computing systems or networks of systems (such as metal, glass, air, or any other suitable communication medium).

[0200] Communication between the computing system 2001 and other computing systems (not shown) can occur through one or more communication networks and according to various communication protocols, combinations of protocols, or variations thereof. Examples include intranets, internets, the Internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software-defined networks, data center buses and backplanes, or any other type of network, combination of networks, or variations thereof.

[0201] As those skilled in the art will recognize, aspects of the present invention can be implemented as systems, methods, computer program products, and other configurable systems. Accordingly, aspects of the present invention can take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware aspects, which are generally referred to herein collectively as "circuit," "module," or "system." Furthermore, aspects of the present invention can take the form of computer program products implemented on one or more memory devices or (one or more) computer-readable media, thereon having computer-readable program code embodied.

[0202] Unless the context explicitly requires otherwise, throughout the specification and claims, the terms "comprise," "comprising," "including," etc., shall be interpreted as encompassing, not exclusive or exhaustive; that is, in the sense of "including but not limited to." As used herein, the terms "connection," "coupling," or any variation thereof refer to any connection or coupling between two or more elements, whether direct or indirect; the coupling or connection between elements may be physical, logical, or a combination thereof. Furthermore, the terms "this article," "above," "below," and similar terms, when used in this application, refer to the application as a whole, and not any particular part thereof. Where the context permits, singular or plural terms used in the above specific embodiments may also include either the plural or the singular, respectively. Except for selection or identification of alternatives, when referring to a list of two or more items, the word "or" covers all of the following interpretations of the word: any item in the list, all items in the list, and any combination of items in the list.

[0203] The phrases "in some embodiments," "according to some embodiments," "in the illustrated embodiments," "in other embodiments," etc., generally mean that the specific feature, structure, or characteristic following the phrase is included in at least one embodiment of the present technology, and may be included in more than one embodiment. Furthermore, such phrases do not necessarily refer to the same embodiment or different embodiments.

[0204] The specific embodiments of this technology described above are not intended to be exhaustive or to limit the technology to the precise forms disclosed above. While specific examples of the technology have been described above for illustrative purposes, various equivalent modifications can be made within the scope of this technology, as will be recognized by those skilled in the art. For example, although processes or blocks are presented in a given order, alternative embodiments may execute routines with steps, or employ systems with blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and / or modified to provide alternatives or sub-combinations. Each of these processes or blocks can be implemented in various different ways. Moreover, although processes or blocks are sometimes shown as being executed sequentially, these processes or blocks may instead be executed or implemented in parallel, or may be executed at different times. Additionally, any specific numbers mentioned herein are merely examples: alternative embodiments may employ different values ​​or ranges.

[0205] The teachings of the techniques provided herein can be applied to other systems, not necessarily those described above. Elements and actions from the various examples described above can be combined to provide further implementations of the technique. Some alternative implementations of the technique may include not only additional elements of those implementations described above, but may also include fewer elements.

[0206] Based on the specific embodiments described above, these and other modifications can be made to the technology. While the above description depicts certain examples of the technology and describes the expected best mode, the technology can be practiced in many ways, however detailed it may appear in the text above. The details of the system can vary considerably in its specific embodiments, but are still covered by the technology disclosed herein. As stated above, specific terms used in describing certain features or aspects of the technology should not be construed as implying that the term is redefined herein as limited to any specific feature, characteristic, or aspect of the technology associated with that term. Generally, the terms used in the following claims should not be construed as limiting the technology to the specific examples disclosed in the specification, unless such terms are expressly defined in the Detailed Description section above. Accordingly, the actual scope of this technology covers not only the disclosed examples but also all equivalent ways of practicing or implementing the technology under the claims.

[0207] To reduce the number of claims, certain aspects of the present technology are presented below in certain claim forms, but the applicant contemplates any number of aspects of the present technology in any number of claim forms. For example, while only one aspect of the present technology is described as a computer-readable medium claim, other aspects may equally be implemented as computer-readable medium claims, or in other forms, such as being implemented as means-plus-function claims. Any claim intended to be treated under 35 USC § 112(f) will begin with the phrase "means for," but the use of the term "for" in any other context is not intended to invoke treatment under 35 USC § 112(f). Accordingly, the applicant reserves the right to pursue additional claims after the filing of this application in order to pursue such additional claim forms in this application or a continuation thereof.

[0208] The aspects of the subject matter described herein are set forth in the following numbered clauses:

[0209] 1. A cloud-based network system, comprising:

[0210] One or more processors;

[0211] A memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the process of assigning Virtual Internet Protocol (VIP) addresses to application pods in a containerized software environment, the VIP addresses being directly accessible from a network outside the containerized software environment, the process including:

[0212] Reserve an Internet Protocol (IP) address range for use as a VIP address;

[0213] Assign a first fixed IP address to the first application pod;

[0214] Assign a first VIP address from the IP address range to the first application pod; and

[0215] Traffic destined for the first VIP address is routed to the first fixed IP address.

[0216] 2. The cloud-based network system as described in Clause 1 further includes instructions that, when executed, cause the one or more processors to:

[0217] Determine when the first application pod has been restarted as a new application pod;

[0218] Assign a new static IP address to the new application pod; and

[0219] The first VIP address is assigned to the new application pod, and the first VIP address is configured to route traffic to the new static IP address.

[0220] 3. The cloud-based network system as described in Clause 1 further includes instructions that, when executed, cause the one or more processors to:

[0221] Assign a Virtual Network Interface Controller (Vnic) to the first application pod to act as an externally accessible port that is not managed by the containerized software environment;

[0222] Assign the first fixed IP address to Vnic; and

[0223] Assign the first VIP address to Vnic so that traffic destined for the first VIP address is routed to be received at the first fixed IP address.

[0224] 4. The cloud-based network system as described in Clause 1 further includes instructions that, when executed, cause the one or more processors to:

[0225] Assign the primary role to the first application pod;

[0226] Assign the second fixed IP address to the second application pod;

[0227] Assign the second application pod to a standby role; and

[0228] In response to the crash of the first application pod, the second application pod is switched from standby role to primary role.

[0229] 5. The cloud-based network system as described in Clause 4 further includes instructions that, when executed, cause the one or more processors to:

[0230] Determine when the first application pod crashed; and

[0231] In response to the crash of the first application pod, the first VIP address is assigned to the second application pod, and the first VIP address is configured to route traffic to the second fixed IP address.

[0232] 6. The cloud-based network system as described in Clause 5 further includes instructions that, when executed, cause the one or more processors to:

[0233] Operating cloud-based network systems in a public cloud environment;

[0234] Request the public cloud operator to associate the first VIP address with the first application pod; and

[0235] In response to the crash of the first application pod, a request is made to the public cloud operator to associate the first VIP address with the second application pod.

[0236] 7. The cloud-based network system as described in Clause 4 further includes instructions that, when executed, cause the one or more processors to:

[0237] Assign the first VIP address to the second application pod;

[0238] While the first application pod is running, traffic destined for the first VIP address will be routed to the first fixed IP address; and

[0239] In response to the crash of the first application pod, traffic destined for the first VIP address is routed to the second fixed IP address.

[0240] 8. The cloud-based network system as described in Clause 7 further includes instructions that, when executed, cause the one or more processors to:

[0241] Operating cloud-based network systems in a private cloud environment; and

[0242] Traffic destined for the first VIP address is routed to be received at either the first fixed IP address or the second fixed IP address based on the Free Address Resolution Protocol (GARP).

[0243] 9. The cloud-based network system as described in Clause 8 further includes instructions that, when executed, cause the one or more processors to:

[0244] Assign a third fixed IP address to a third application pod;

[0245] The first VIP address is assigned to the first application pod, the second application pod, and the third application pod, with the first application pod and the third application pod playing the primary role and the second application pod playing the backup role.

[0246] The second VIP address is assigned to the first application pod, the second application pod, and the third application pod.

[0247] When both the first and third application pods are in primary use, traffic is routed via GARP to the first and third fixed IP addresses, respectively, through the first and second VIP addresses; and

[0248] When the first or second application pod crashes, GARP-based traffic intended for the crashed pod will be routed to the second fixed IP address.

[0249] 10. The cloud-based network system as described in Clause 1 further includes: the containerized software environment including a Kubernetes cluster.

[0250] 11. A method comprising:

[0251] Operating a cloud-based networking system within a containerized software environment to assign Virtual Internet Protocol (VIP) addresses to application pods within the containerized software environment, wherein the VIPs are directly accessible from networks outside the containerized software environment, includes:

[0252] Reserve an Internet Protocol (IP) address range for use as a VIP address;

[0253] Assign a first fixed IP address to the first application pod;

[0254] Assign a first VIP address from the IP address range to the first application pod; and

[0255] Traffic destined for the first VIP address is routed to the first fixed IP address.

[0256] 12. The method as described in Clause 11 further includes:

[0257] Determine when the first application pod has been restarted as a new application pod;

[0258] Assign a new static IP address to the new application pod; and

[0259] The first VIP address is assigned to the new application pod, and the first VIP address is configured to route traffic to the new static IP address.

[0260] 13. The method as described in Clause 11 further includes:

[0261] Assign a Virtual Network Interface Controller (Vnic) to the first application pod to act as an externally accessible port that is not managed by the containerized software environment;

[0262] Assign the first fixed IP address to Vnic; and

[0263] Assign the first VIP address to Vnic so that traffic destined for the first VIP address is routed to be received at the first fixed IP address.

[0264] 14. The method as described in Clause 11 further includes:

[0265] Assign the primary role to the first application pod;

[0266] Assign the second fixed IP address to the second application pod;

[0267] Assign the second application pod to a standby role; and

[0268] In response to the crash of the first application pod, the second application pod is switched from standby role to primary role.

[0269] 15. The method as described in Clause 14, further comprising:

[0270] Determine when the first application pod crashed; and

[0271] In response to the crash of the first application pod, the first VIP address is assigned to the second application pod, and the first VIP address is configured to route traffic to the second fixed IP address.

[0272] 16. The method as described in Clause 15 further includes:

[0273] Operating cloud-based network systems in a public cloud environment;

[0274] Request the public cloud operator to associate the first VIP address with the first application pod; and

[0275] In response to the crash of the first application pod, a request is made to the public cloud operator to associate the first VIP address with the second application pod.

[0276] 17. The method as described in Clause 14 further includes:

[0277] Assign the first VIP address to the second application pod;

[0278] When the first application pod is running, traffic destined for the first VIP address will be routed to the first fixed IP address; and

[0279] In response to the crash of the first application pod, traffic destined for the first VIP address is routed to the second fixed IP address.

[0280] 18. The method as described in Clause 17 further includes:

[0281] Operating cloud-based network systems in a private cloud environment; and

[0282] Traffic destined for the first VIP address is routed to be received at either the first fixed IP address or the second fixed IP address based on the Free Address Resolution Protocol (GARP).

[0283] 19. The method as described in Clause 18 further includes:

[0284] Assign a third fixed IP address to a third application pod;

[0285] The first VIP address is assigned to the first application pod, the second application pod, and the third application pod, with the first application pod and the third application pod playing the primary role and the second application pod playing the backup role.

[0286] The second VIP address is assigned to the first application pod, the second application pod, and the third application pod.

[0287] When both the first and third application pods are in primary use, traffic is routed via GARP to the first and third fixed IP addresses, respectively, through the first and second VIP addresses; and

[0288] When the first or second application pod crashes, GARP-based traffic intended for the crashed pod will be routed to the second fixed IP address.

[0289] 20. The method of Clause 11 further includes: the containerized software environment includes a Kubernetes cluster. < / group>

Claims

1. A method comprising: Operating a cloud-based networking system within a containerized software environment to assign Virtual Internet Protocol (VIP) addresses to application pods within the containerized software environment, wherein the VIPs are directly accessible from networks outside the containerized software environment, the assignment includes: Reserve an Internet Protocol (IP) address range for use as a VIP address; Assign a first fixed IP address to the first application pod; Assign a first VIP address from the IP address range to the first application pod; and Traffic destined for the first VIP address is routed to the first fixed IP address.

2. The method of claim 1, further comprising: Determine when the first application pod has been restarted as a new application pod; Assign a new static IP address to the new application pod; as well as The first VIP address is assigned to the new application pod, and the first VIP address is configured to route traffic to the new fixed IP address.

3. The method of claim 1 or claim 2, further comprising: Assign a Virtual Network Interface Controller (Vnic) to the first application pod to act as an externally accessible port that is not managed by the containerized software environment; Assign the first fixed IP address to the Vnic; as well as The first VIP address is assigned to the Vnic so that traffic destined for the first VIP address is routed to be received at the first fixed IP address.

4. The method as described in any of the preceding claims, further comprising: Assign the primary role to the first application pod; Assign the second fixed IP address to the second application pod; Assign the second application pod to a standby role; and In response to the crash of the first application pod, the second application pod is switched from standby role to primary role.

5. The method of claim 4, further comprising: Determine when the first application pod crashed; as well as In response to the crash of the first application pod, the first VIP address is assigned to the second application pod, and the first VIP address is configured to route traffic to the second fixed IP address.

6. The method of claim 5, further comprising: Operating cloud-based network systems in a public cloud environment; Request the public cloud operator to associate the first VIP address with the first application pod; as well as In response to the crash of the first application pod, a request is made to the public cloud operator to associate the first VIP address with the second application pod.

7. The method of any one of claims 4 to 6, further comprising: Assign the first VIP address to the second application pod; While the first application pod is running, traffic destined for the first VIP address will be routed to the first fixed IP address; as well as In response to the crash of the first application pod, traffic destined for the first VIP address is routed to the second fixed IP address.

8. The method of claim 7, further comprising: Operating cloud-based network systems in a private cloud environment; as well as Traffic destined for the first VIP address is routed to be received at either the first fixed IP address or the second fixed IP address based on the Free Address Resolution Protocol (GARP).

9. The method of claim 8, further comprising: Assign a third fixed IP address to a third application pod; The first VIP address is assigned to the first application pod, the second application pod, and the third application pod, with the first application pod and the third application pod playing the primary role and the second application pod playing the backup role. The second VIP address is assigned to the first application pod, the second application pod, and the third application pod. When both the first application pod and the third application pod are in primary use, traffic is routed to the first fixed IP address and the third fixed IP address via the first VIP address and the second VIP address based on GARP. as well as When the first or second application pod crashes, GARP-based traffic intended for the crashed pod will be routed to the second fixed IP address.

10. The method as described in any of the preceding claims, wherein the containerized software environment includes a Kubernetes cluster.

11. A cloud-based network system, comprising: One or more processors; A memory having instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform the method as described in any of the preceding claims.

12. A computer program product comprising a program that, when executed, performs the method as described in any one of claims 1 to 10.

13. The computer program product of claim 12, comprising a computer-readable storage medium containing the program.

Images