User-level shared internet bandwidth control method and system for multi-tenant pods

Abstract

The application provides a user-level shared Internet bandwidth control method and system for a multi-tenant Pod, which comprises the following steps: in response to receiving a user export declaration, performing a user export resource chain arrangement operation to generate an export resource of the user; in response to the Pod being created, finding a unique and state-ready export resource, injecting a managed network accessory definition corresponding to the export resource into a network annotation of the Pod, and retaining a network accessory of an original master container of the Pod; performing permission verification logic to prohibit Pods without a user identifier from mounting the managed network accessory definition, and allowing Pods with a user identifier to mount only the managed network accessory definition belonging to themselves; configuring a master network interface and a secondary network interface for the Pod, retaining a network segment route corresponding to a cluster public inter-class domain routing set in the master network interface, and switching a default route of the remaining Internet traffic to the secondary network interface. The application realizes business user granularity aggregation of external network bandwidth control.

Description

Technical Field

[0001] This application relates to the fields of cloud computing, container networking, and software-defined networking, specifically to a method and system for controlling user-level shared internet bandwidth for multi-tenant Pods. Background Technology

[0002] Kubernetes (K8s, a container orchestration platform) is widely used in multi-tenant cloud computing scenarios, where a large number of business container groups (Pods) run for multiple business users within the same cluster. In multi-tenant scenarios, it is necessary to manage the external network traffic accessed by different business users independently. That is, under the same business user name, regardless of which node the Pod is scheduled to in the cluster, all Pods share the same total internet bandwidth, and the bandwidth quotas of different business users are isolated from each other.

[0003] To achieve the above objectives, existing Kubernetes networking solutions have many insurmountable drawbacks.

[0004] First, the rate limiting granularity of the native Container Network Interface (CNI) network component is per Pod, meaning that external network bandwidth limits can only be set for each Pod. If the same business user starts multiple Pods, the external network bandwidth of the multiple Pods will be aggregated, making it impossible to achieve total quota control on a business user basis. For example, if each Pod is allocated 10Mbps of bandwidth, the aggregate bandwidth when two Pods access the external network simultaneously can reach 20Mbps, exceeding the preset total bandwidth limit for that business user.

[0005] Second, if bandwidth limiting is set on the physical network card of the cluster node or the node's global network address translation (NAT) gateway, a single node will carry multiple Pods of different business users. All business user traffic will be mixed together, making it impossible to distinguish the traffic ownership and achieve independent external network bandwidth quotas for each business user.

[0006] Third, to provide business users with independent external network access, the existing solution configures a second network interface card (NIC) for each Pod, switching all traffic to this secondary NIC and routing it through a NAT gateway to access the internet. This one-size-fits-all routing modification forwards internal network traffic from Pods to the Core DNS, cluster services, and internal storage segments to the external NAT gateway. This can lead to abnormal access to internal cluster services, incorrect rate limiting of internal network traffic by external bandwidth, and in severe cases, routing loops, compromising the availability of cluster services.

[0007] Fourth, to achieve a complete set of egress resources for each business user, including an independent Virtual Private Cloud (VPC), NAT gateway, Elastic Internet Protocol (EIP) address, and Network Attachment Definition (NAD), existing solutions require operations administrators to manually maintain a large number of configuration files. These network resources have strong dependencies and must be created in a specific order; manual operation is prone to IP address conflicts and incorrect resource creation order. If the deletion order is reversed, residual Source Network Address Translation (SNAT) rules and Quality of Service (QoS) flow control rules will remain, leading to resource leaks. In large-scale multi-tenant scenarios, manual maintenance is costly and unreliable.

[0008] Fifth, some existing solutions modify the routing table by executing scripts inside the initialization container (initContainer) or the business container to modify the routing table in order to change the internal routing of the Pod. This approach requires modifying the business image, modifying the Pod configuration definition, and granting high-risk network management permissions to the container, which is highly intrusive to the business; different business images have different environments, some images lack network tools, and the solution has poor universality.

[0009] Furthermore, while the Multus CNI (Multi-NIC Container Networking Component) in the Kubernetes ecosystem supports mounting multiple NICs to a Pod, the Multus meta plugin only serves to chain lower-level CNI plugins and complete the mounting of multiple NICs; the plugin itself does not handle routing policies within the Pod's network namespace. To achieve separation of internal and external network traffic, container scripts are still required. The meta plugin lacks mechanisms for route snapshots, transaction rollback, and configuration consistency checks. Errors during route writing can leave incomplete routes, directly causing Pod network failures. Multiple executions of the CNI ADD command lack idempotency protection, easily leading to conflicting and redundant routes. The CNI DEL command cannot automatically restore the original route when deleting a NIC. Relying solely on the native Multus component is insufficient for securely and non-intrusive selective traffic splitting between the cluster and external networks without business disruption.

[0010] In summary, existing technologies lack a complete closed-loop solution and cannot simultaneously meet multiple requirements such as: granular aggregation of external network bandwidth control at the business user level, unrestricted internal traffic within the cluster, non-intrusive modification of business Pods, multi-tenant isolation of shared namespaces, automatic network resource orchestration, fault tolerance of routing operations, and secure resource reclamation. Summary of the Invention

[0011] This application aims to provide a user-level shared internet bandwidth control method and system for multi-tenant Pods, which achieves aggregated external network bandwidth control at the business user level, unrestricted internal cluster traffic, non-intrusive modification of business Pods, and multi-tenant isolation of shared namespaces.

[0012] In a first aspect, embodiments of this application provide a user-level shared internet bandwidth control method for multi-tenant Pods, including:

[0013] In response to receiving a user's egress statement, the system performs user egress resource chain orchestration operations to generate the user's egress resources. The egress resources include a network address translation gateway (NAT gateway), an elastic public IP address (EIP), source network address translation (SNAT) rules, a user-defined virtual private network (VPC), and a quality of service (QoS) bandwidth policy bound to the elastic public IP address.

[0014] In response to the creation of a Pod, the user identifier carried by the Pod is read. Using the Pod's business namespace and the user identifier as joint query conditions, a unique and ready egress resource is found. The managed network attachment definition (NAD) corresponding to the egress resource is injected into the Pod's network annotation, and the Pod's original main container network interface (CNI) network attachment is retained.

[0015] The permission verification logic is executed to prevent Pods without user IDs from mounting managed network attachment definitions (NADs), so that Pods with user IDs are only allowed to mount managed network attachment definitions belonging to themselves, and the modification of user IDs and managed network annotations is prohibited during the Pod runtime update phase.

[0016] Configure the primary and secondary network interfaces for the Pod, and pre-configure the Cluster Common Classless Inter-Domain Routes (CIDR) set on the node side. Keep the network segment routes corresponding to the Cluster Common Classless Inter-Domain Routes set on the primary network interface, and switch the default routes for other Internet traffic to the secondary network interface.

[0017] Secondly, embodiments of this application also provide a user-level shared internet bandwidth control system for multi-tenant Pods, including:

[0018] Controller component, variant webhook component, verification webhook component, chained container network interface (CNI) meta-plugin, network address translation (NAT) gateway component, and quality of service (QoS) execution module;

[0019] The controller component is used to receive user exit declarations and exit address pool objects, perform exit resource chain orchestration and cluster-level exit address pool management; it is also used to perform exit resource reclamation according to the preset dependency order after receiving a deletion instruction, set resource protection flags when reclamation fails, re-queue the reclamation coordination task into the processing queue for retry, and clear resource protection flags after all resources have been reclaimed.

[0020] The variant Webhook component is used to respond to Pod creation events, read the user identifier carried by the Pod, query the unique and ready exit resources in combination with the Pod's business namespace, inject the corresponding managed network attachment definition (NAD) into the Pod's network annotation, and retain the Pod's original main container network interface network attachment.

[0021] The Webhook component is used to execute permission verification logic to prevent Pods without user identifiers from mounting managed network attachment definitions. Pods with user identifiers are only allowed to mount managed network attachment definitions belonging to themselves, and the user identifier and managed network annotations of Pods are prohibited from being modified during the Pod runtime update phase.

[0022] Different users have independent VPCs, and the address spaces of each VPC are isolated. The controller does not perform cross-VPC global CIDR conflict verification, allowing different users to reuse the same internal subnet CIDR, subnet logical gateway, and NAT gateway internal addresses.

[0023] The chained container network interface meta-plugin is used to configure the primary and secondary network interfaces for Pods; and based on the cluster public classless inter-domain route set pre-configured on the node side, it retains the network segment route corresponding to the cluster public classless inter-domain route set on the primary network interface, and switches the default route of other Internet traffic to the secondary network interface.

[0024] A network address translation gateway component is used to implement source network address translation;

[0025] The Quality of Service (QoS) execution module, bound to the Elastic IP address, is used to implement bandwidth control over internet traffic flowing through the Elastic IP address.

[0026] In summary, this application proposes a user-level shared internet bandwidth control method for multi-tenant Pods. By responding to user egress declarations, it automatically executes egress resource chain orchestration, generating dedicated isolated egress resources for each business user. This achieves automated and orderly deployment of the entire set of network resources, avoiding the address conflicts, high error rates, and low efficiency issues associated with manual operation and maintenance. During container group creation, it matches egress resources based on the business namespace and user identifier, injects managed network attachment definitions into the container group while retaining the original main container network attachments. This automatically associates the container group with the external network egress resources of its respective business user, allowing the container group to continuously reuse the same set of egress resources during cross-node scheduling. Finally, by configuring permission verification logic, it restricts containers from mounting non-authorized data without authorization. By locking user identifiers and network annotations during the runtime phase, the network resources belonging to each user are strictly isolated, and unauthorized bandwidth preemption is prevented between multiple service users. By configuring primary and secondary network interfaces for container groups and implementing traffic splitting in conjunction with the pre-configured cluster public classless inter-domain routing set, internal cluster traffic is maintained on the primary network interface, while only Internet traffic is switched to the secondary network interface for forwarding. This prevents internal network traffic from being mistakenly forwarded to the external network gateway, prevents internal network traffic from being subject to external network bandwidth throttling, and avoids routing loops. Without making intrusive modifications to the service containers, the unified aggregation of external network traffic at the service user level is achieved, providing a technical implementation for all container groups under the same service user to share a single aggregated external network bandwidth. Attached Figure Description

[0027] Figure 1 This is a flowchart of a user-level shared internet bandwidth control method for a multi-tenant Pod provided in an embodiment of this application;

[0028] Figure 2 This is a staged orchestration flowchart of a user exit resource chain provided in an embodiment of this application;

[0029] Figure 3 This is a schematic diagram of user exit deletion and dependency order reclamation provided in an embodiment of this application;

[0030] Figure 4 This is a schematic diagram of an internal routing selection within a Pod provided in an embodiment of this application;

[0031] Figure 5 This is a hierarchical architecture diagram of a Pod-level user egress bandwidth management system provided in an embodiment of this application;

[0032] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0034] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] Example 1

[0036] Figure 1 This is a flowchart illustrating a user-level shared internet bandwidth control method for a multi-tenant Pod provided in an embodiment of this application. See also... Figure 1 The user-level shared internet bandwidth control method for this multi-tenant Pod specifically includes the following steps:

[0037] S110. In response to receiving a user exit statement, perform user exit resource chain orchestration operation to generate the user's exit resources.

[0038] Egress resources include NAT gateways, EIPs, SNAT rules, user-specific VPCs, and QoS bandwidth policies bound to EIPs.

[0039] The user exit declaration is a Kubernetes custom resource, UserEgress. This resource can be manually created and committed by the cluster administrator using the kubectl client, or it can be automatically generated and committed by the upper-layer multi-tenant business platform by calling the Kubernetes API. The user exit declaration needs to be created before the business Pod is created.

[0040] The user egress controller listens to the Kubernetes API. When it receives a user egress declaration, it triggers the resource orchestration process to generate a complete set of egress resources for that user. Only after the resource status is marked as ready can the business Pods created subsequently use the set of egress resources.

[0041] For example, performing a user egress resource chain orchestration operation to generate the user's egress resources includes:

[0042] Perform parameter validation on the user's export declaration. If the validation passes, proceed with the following steps:

[0043] Create a user-independent VPC and a user-internal subnet, and create a two-way QoS policy that binds to the EIP;

[0044] First, create a startup NAD containing only the secondary network plugin, then create the NAT gateway;

[0045] EIPs are allocated by the underlying IPAM, and SNAT rules are created after the EIPs are polled and waited for to reach the ready state.

[0046] The NAD will be upgraded to a full chained NAD that includes a sub-network plugin and a routing plugin, and the EIP information and readiness status will be written back to the user's exit statement.

[0047] Create user-specific VPCs and user intranets, including:

[0048] Configure static internet routes pointing to the network address translation gateway within the user's independent VPC;

[0049] Add the internal addresses of the subnet logical gateway and NAT gateway to the IP exclusion list of the user's internal subnet; write the dynamic parameters of the secondary network gateway and allowed secondary network CIDR users into the complete chained NAD.

[0050] When performing user egress resource chain orchestration operations, cross-VPC global conflict checks are not performed on the internal addresses of user-internal subnet CIDRs, subnet logical gateways, and NAT gateways for different users. This allows different users to configure the same internal addresses of user-internal subnet CIDRs, subnet logical gateways, and NAT gateways in their respective independent VPCs. Different users each have independent virtual private networks, and different users can reuse the same internal addresses of user-internal subnet CIDRs, subnet logical gateways, and NAT gateways. This addresses the problem of large-scale multi-tenant IP planning pressure.

[0051] Specifically, the user egress declaration must include at least the following parameters: user identifier, service namespace, user internal subnet CIDR, subnet logical gateway, NAT gateway internal address, egress address pool reference, download bandwidth, upload bandwidth, and burst parameters. Optionally, a NAT gateway scheduling node can also be specified.

[0052] The user exit controller uses deterministic names to generate resources and performs idempotent coordination in the following phases, such as... Figure 2 The flowchart of the phased orchestration of the user exit resource chain is shown. (1) First, the parameters of the user exit declaration are checked, such as whether the subnet logical gateway and NAT gateway addresses are both located in the internal subnet and are different; check whether the service namespace and address pool exist and whether the address pool is ready; after the verification is passed, the subsequent operations are continued.

[0053] The subnet logical gateway is a Layer 3 gateway within the user's internal subnet segment, serving as the first-hop forwarding node for traffic from a Pod's secondary network interface to the outside of the subnet. The network address translation gateway (NAT gateway) is an independent component deployed within the same subnet, receiving internet traffic forwarded by the subnet and performing source network address translation. During the subnet configuration phase, it verifies that both the subnet logical gateway address and the NAT gateway's internal network address are within the user's internal subnet segment and includes these two addresses in the subnet IP exclusion list to avoid address conflicts when the address management module allocates IPs.

[0054] (2) Create a user-independent VPC and configure a static route in the VPC with a destination address (e.g., 0.0.0.0 / 0) and a next hop to the internal address of the NAT gateway.

[0055] The secondary network interface of the Pod accessing this user's egress is located within the user's internal subnet of this VPC. When a Pod initiates a packet to access the Internet, after the packet reaches the subnet logical gateway, it relies on this VPC-level static route to forward all traffic destined for the address (0.0.0.0 / 0) to the NAT gateway's internal network address. This achieves the convergence of all external network traffic from all Pods under the same business user to the same NAT gateway. After all traffic is converged to the same NAT gateway, external network traffic is uniformly rate-limited through the Quality of Service (QoS) bandwidth policy bound to the user's dedicated elastic public IP address.

[0056] This static route is deployed only within the current business user's independent VPC and belongs to the secondary network plane route. The cluster's primary network interface relies on the cluster's public classless inter-domain route set policy, and internal network traffic still goes through the primary network interface and is not affected by this default route. The primary and secondary network selective traffic splitting mechanism is as follows: internal network traffic goes through the primary network interface; internet traffic goes through the secondary network interface → VPC → this static route → NAT gateway.

[0057] Once traffic reaches the NAT gateway, Source Network Address Translation (SNAT) is performed to convert the Pod's internal network IP address into a user-specific elastic public IP address for internet access; external network return packets are also routed back through the NAT gateway to ensure bidirectional communication.

[0058] In other words, a static route is configured within the user's independent virtual private network (VPN), with the destination address and the next hop being an internal address of the network address translation gateway (NAT). This static route is used to redirect all internet access traffic within the VPN to the NAT, ensuring that all external network access traffic from all container groups under the same business user converges to the same NAT. This leverages the Quality of Service (QoS) bandwidth policy bound to the elastic public IP address to achieve user-level aggregated traffic control. Simultaneously, this route only operates within the VPN plane of the secondary network and does not affect the cluster intranet communication of the container group's primary network interface. Combined with the primary and secondary network traffic splitting strategy, it achieves separate scheduling of internal and external network traffic.

[0059] (3) Create a user internal subnet and add the internal addresses of the subnet logical gateway and NAT gateway to the exclusion list.

[0060] Among them, the user's independent Virtual Private Network (VPC) is a completely isolated logical private network address space; the user's internal subnet (i.e., Subnet) is an independent IP network segment (i.e., CIDR address segment) allocated within the VPC and specifically used to carry secondary network traffic.

[0061] Through automated resource chain orchestration, a complete set of isolated egress resources is generated in an orderly manner according to dependencies, without the need for manual configuration; an independent virtual private network, an independent network address translation gateway, and an elastic public IP address are built for each business user, realizing the isolation of external network resources between business users and providing the hardware foundation for the aggregation of traffic from all container groups under the same business user.

[0062] Create a user-internal subnet, which provides a pool of IP addresses for the secondary network interface of the Virtual Private Network (VPN). Add the internal addresses of the subnet's logical gateway and network address translation gateway to the subnet's IP exclusion list. The internal addresses of the subnet's logical gateway and network address translation gateway are pre-specified fixed IP addresses, serving as the subnet's segment gateway and the next hop for external network forwarding, respectively. The exclusion list constrains the underlying address management module (IPAM) to not allocate these two addresses, preventing IP address conflicts caused by service container groups preempting them and ensuring the stable availability of static routes and forwarding links within the VPN.

[0063] (4) Create a two-way QoS policy bound to the EIP.

[0064] This involves creating a bidirectional Quality of Service (QoS) bandwidth policy bound to the Elastic Public Address (IPA). This bandwidth policy is not bound to individual container groups, but rather to the user's IPA. Internet access traffic from all container groups under the same business user is aggregated to this IPA and uniformly managed by the QoS bandwidth policy, thus achieving aggregated external network bandwidth control at the business user level. The bidirectional policy separately imposes quota constraints on internet download and upload traffic. Traffic between container groups within a Virtual Private Network (VPN) does not pass through the IPA and is therefore not subject to the bandwidth policy. This step pre-creates the QoS bandwidth policy object, and the actual binding is completed once the IPA is ready, consistent with the logic of unbinding before deleting resources during the resource reclamation phase.

[0065] (5) First create a startup NAD containing only the bottom-level network plugins, then create the VPC NAT Gateway to avoid the NATGateway referencing a plugin chain that is not yet available when it starts up.

[0066] During resource orchestration, a startup network attachment definition configuring only the lower-level network plugins is created first, followed by the creation of a Network Address Translation (NAT) gateway instance. The NAT gateway runs as a container group, and its startup requires referencing the network attachment definition to create the secondary network interface. If the gateway is created first but the network attachment definition has not yet been generated, container group sandbox initialization will fail. Simultaneously, the routing meta-plugins within the complete chained network attachment definition rely on the NAT gateway's internal network address to generate static internet routes within the virtual private network. If the gateway is not ready, the complete chained plugin chain will fail to validate. To address this, this embodiment employs a two-stage network attachment definition upgrade strategy. The startup network attachment definition contains only the secondary network plugins and does not involve routing configuration logic, allowing the NAT gateway to start successfully and obtain an internal network address. Once the gateway instance is ready, the startup network attachment definition is upgraded to a complete chained network attachment definition including routing meta-plugins. This resolves the circular resource dependency between the gateway and the chained plugin chain, preventing the NAT gateway from referencing an unavailable plugin chain during startup and causing startup anomalies.

[0067] (6) Create an EIP object without a specified fixed IP address, and assign an address to it by the shared external subnet IPAM.

[0068] Elastic public IP address objects without specified fixed IPs are created, and the addresses are automatically allocated by the underlying address management module (IPAM) of the shared external subnet. This design eliminates the need for users to manually fill in elastic public IP addresses before making egress declarations, avoiding address conflicts and configuration errors caused by manual IP selection. Elastic public IP addresses are automatically requested from the unified shared external subnet address pool of the cluster, facilitating centralized statistics on the total capacity of the address pool, the number of allocated and available addresses. The controller polls the elastic public IP address objects, and only creates source network address translation rules after the address allocation is complete and the status is ready, ensuring that the source network address translation rules reference valid public IP addresses. During resource reclamation, the address management module automatically releases IPs and returns them to the address pool. Combined with the address pool's expansion and contraction conflict verification mechanism, this improves the management reliability of public IP resources in large-scale multi-tenant scenarios.

[0069] (7) Poll the EIP status and do not create SNAT until the address is obtained and the status is ready.

[0070] After the user egress controller creates an Elastic Public Address (EPS) object, it polls the EPS's running status. Before the EPS obtains an assigned public address and its status is "ready," the creation of source network address translation (NIP) rules is not executed. The IP allocation of EPS is asynchronously completed by the underlying address management module (IPAM). If a NIP rule is created before the address is ready, the rule may reference an empty or invalid public IP, causing external network source address translation to fail. The polling waiting mechanism ensures that the source NIP rule references a real and valid EPS when it is created; it also adapts to the asynchronous allocation model of the address management module, avoiding the generation of invalid resources. This creation sequence corresponds to the logic of prioritizing the deletion of source NIP rules before deleting EPS during the resource reclamation phase, forming a two-way dependency constraint between creation and deletion, reducing the risk of network rule residue leakage.

[0071] (8) Create SNAT rules from the user’s internal subnet to the EIP.

[0072] SNAT rules are used to translate the source address of private network packets from a user's internal subnet into the public IP address corresponding to the Elastic Public Address (ESP), enabling Pod network traffic to access the Internet normally. This step is executed after the ESP is ready, ensuring that the translation rule uses a real and valid public IP address; at the same time, all external network traffic of Pods under the same business user is uniformly sent out through this ESP, providing a network foundation for user-granular aggregated bandwidth management.

[0073] (9) Update the startup NAD to a complete chained NAD that includes the low-level network plugin and the routing meta-plugin.

[0074] The startup network attachment definition will be updated to a complete chained network attachment definition that includes both the low-level network plugin and the routing meta-plugin. Initially, the startup network attachment definition is only used to ensure the normal startup of the network address translation gateway and its acquisition of internal network addresses. After the gateway, elastic public IP address, and source network address translation rules are deployed, the upgrade will allow the routing meta-plugin to obtain complete gateway network parameters. Once the service container group mounts this complete chained network attachment definition, the chained container network interface meta-plugin will be invoked to perform selective traffic splitting between the primary and secondary network interfaces. This update is an in-place resource update and will not interfere with the network connectivity of already running network address translation gateway instances.

[0075] (10) Write the EIP, resource name and readiness conditions into the user exit status.

[0076] The elastic public IP address, sub-resource name, and readiness conditions are written into the status field of the user egress declaration. On the one hand, this feeds back the resource orchestration results to the upper layer, making it easier for the business side to query the allocated public IP addresses and resource readiness status; on the other hand, it provides a basis for determining the mutation webhook in the admission plane, allowing the injection of the managed network attachment definition corresponding to the egress into the container group only when the readiness conditions are met; at the same time, the saved status information serves as the basis for controller idempotent coordination, fault recovery, and subsequent resource reclamation, allowing the controller to continue resource processing based on the status after restarting.

[0077] Multiple users can reside in the same service namespace. Each user still has its own independent VPC, Subnet, NAD, NATGateway, EIP, and QoS. Since the logical address spaces of each VPC are isolated from each other, different users can reuse the same internal subnet CIDR, subnet logical gateway, and NAT gateway internal addresses, reducing the pressure of internal address planning in large-scale user scenarios.

[0078] S120. In response to the creation of a Pod, read the user identifier carried by the Pod, use the Pod's business namespace and user identifier as joint query conditions to find a unique and ready egress resource, inject the managed NAD corresponding to the egress resource into the Pod's network annotation, and retain the Pod's original main CNI network attachments.

[0079] S130. Execute permission verification logic to prevent Pods without user identifiers from mounting managed NADs, ensuring that Pods with user identifiers are only allowed to mount managed NADs belonging to themselves, and prohibiting modification of user identifiers and managed network annotations during Pod runtime update phases.

[0080] When a Pod is created, the `sys_user_name` tag is read, and the user exit is found using the Pod namespace and user identifier as a combined condition. Only when there is exactly one match and the user's status is ready is the managed NAD of that user appended to the Pod's Multus network annotation; existing unmanaged network attachments are preserved.

[0081] The system establishes a "managed NAD-user" mapping and executes the following permission verification logic: Pods without user identifiers are not allowed to mount managed NADs; Pods with user identifiers must and can only mount their own managed NADs once; Pods cannot mount managed NADs of other users; User tags and managed network annotations cannot be modified when a Pod is updated. Therefore, network selection no longer relies on the business side manually writing error-prone NAD names.

[0082] "Managed NAD" refers to a NetworkAttachmentDefinition resource automatically created by the UserEgress user egress controller, managed throughout its entire lifecycle, and bound to a specific user. This differs from ordinary NADs created manually by the user and not managed by the system. Key attributes: "Managed NAD" is not created manually by the administrator using YAML; it is automatically generated by the UserEgress controller in stages (first generating a startup NAD containing only the secondary network plugin, then upgrading to a full chained NAD after NAT and EIP are ready); the NAD is strongly bound to a specific sys_user_name user and business namespace, exclusively belonging to that user's egress; the NAD internally encapsulates a complete plugin chain: secondary network plugins and the user-egress-route chained route meta-plugin; it is protected by Webhook verification: only Pods with the corresponding user tag are allowed to mount it, other user Pods are prohibited from mounting it, and the annotation cannot be modified while the Pod is running. Its lifecycle completely follows the UserEgress object: when the user egress is deleted, the controller automatically reclaims and deletes the managed NAD.

[0083] By using an admission webhook, a managed NAD bound to a user is injected into the Multus network annotation of the Pod, while preserving the Pod's original primary CNI network attachments. The managed NAD is a network attachment definition resource generated by the user's egress controller and bound one-to-one with a specific user identifier. This step does not require modifying the original definition of the business Pod, enabling the Pod to automatically access the user's dedicated secondary network while maintaining the cluster's primary network communication capabilities. At the same time, it establishes the binding relationship between the Pod, user identifier, and managed NAD, providing a configuration foundation for subsequent selective traffic splitting inside and outside the cluster, user-level public network bandwidth aggregation and rate limiting, and cross-user network access protection.

[0084] For example, the QoS bandwidth policy uses a hierarchical token bucket queue algorithm to implement traffic control; for Internet download traffic, bandwidth control is implemented on the inbound direction of the external communication interface of the NAT gateway.

[0085] For internet upload traffic, bandwidth control is implemented on the outbound direction of the external communication interface of the NAT gateway.

[0086] Specifically, the QoS bandwidth policy is not directly bound to a single Pod, but rather to the user's EIP. Download traffic is redirected to the intermediate function block interface after entering through the NAT Gateway's external interface ingress, and is controlled by a hierarchical token bucket queue; upload traffic is controlled on the external interface egress. Since all internet connections of the same user's Pods pass through the same NAT Gateway and the same EIP, their byte streams enter the same QoS class, thus forming a cross-node, cross-Pod aggregation limit.

[0087] For example, if a user's egress is configured with both download and upload speeds of 10 Mbps: a single Pod download speed is approximately 10 Mbps; when two Pods from the same user download simultaneously, the aggregated value is still approximately 10 Mbps; when different users each have independent EIPs and QoS, the aggregated value for simultaneous downloads is approximately 20 Mbps. When two Pods from the same user communicate within the user's VPC via net1, they do not go through external EIPs and QoS, therefore internal transmission can reach the bandwidth allowed by the node network itself.

[0088] Data plane isolation for different users includes: each user uses an independent VPC and provider, even if their internal CIDRs are the same, they belong to different logical networks; service Pods only mount their own user's NAD; the Operator only creates SNAT, and does not create DNAT, Floating IP, or EIP port mappings to service Pods; the main CNI NetworkPolicy can prohibit service Pods from accessing physical EIP network segments via eth0. The external interface of the NAT Gateway can communicate with each other within the physical network segment without affecting service Pod isolation, because ordinary service Pods do not have this external interface, and there are no rules to map inbound EIP traffic to service Pods.

[0089] S140. Configure the primary and secondary network interfaces for the Pod, and pre-configure the cluster common CIDR set on the node side. Keep the network segment route corresponding to the cluster common CIDR set on the primary network interface, and switch the default route of other Internet traffic to the secondary network interface.

[0090] By differentiating between primary and secondary network interfaces to distribute traffic, and relying on the pre-configured cluster public classless inter-domain route set on the node side for routing, traffic within the cluster continues to communicate through the primary network interface, while only internet access traffic is switched to the secondary network interface for forwarding. This prevents intranet traffic from being incorrectly forwarded to the external network address translation gateway, avoids internal network traffic from being erroneously rate-limited by external network bandwidth control policies, eliminates the risk of routing loops caused by unreasonable routing configurations, and ensures normal access to services within the cluster. At the same time, external network traffic is uniformly forwarded through the secondary network interface to the dedicated egress resources of business users, realizing the aggregation of external network traffic of all container groups under the same business user, and providing a traffic foundation for subsequent implementation of business user-level aggregated bandwidth control at the network address translation gateway.

[0091] Example 2

[0092] Furthermore, it also includes:

[0093] In response to the user's exit deletion command, resource reclamation is performed on the entire set of exit resources according to the preset dependency order. During the resource reclamation process, if any reclamation step fails, a resource protection flag is set to prevent the corresponding exit resource object from being directly deleted, and the resource reclamation task is re-queued for processing, waiting to be re-executed. When all exit resources have been reclaimed, the resource protection flag is cleared.

[0094] Resource recovery is performed on the entire set of exported resources according to the preset dependency order, including:

[0095] Prioritize deleting SNAT rules;

[0096] Remove the binding relationship between EIP and QoS bandwidth policy, and wait for the underlying components to complete the state clearing of the binding relationship;

[0097] Then delete EIP, QoS bandwidth policy, NAT gateway, user internal subnet, managed NAD and user independent VPC in sequence;

[0098] Once all the above-mentioned export resources have been cleared, the resource protection markers will be removed, and the occupied address capacity corresponding to the export address pool will be released.

[0099] like Figure 3 The diagram illustrates a user egress deletion and dependency recycling sequence. When deleting a user egress, the controller first deletes SNAT, then clears the QoS binding of the EIP. After the underlying state is cleared, the EIP, QoS, NAT Gateway, internal Subnet, user NAD, and VPC are deleted. Finally, the finalizer is removed. This order avoids EIPs still being referenced by SNAT, residual flow control classes in the NATGateway, or network resources that cannot be deleted due to dependencies.

[0100] Furthermore, it also includes: receiving an exit address pool object, which is configured with at least a physical network CIDR, a physical gateway, one or more allowed address ranges, an external subnet name, an external network attachment name, a host physical interface, and network address translation gateway scheduling conditions.

[0101] Verify the physical network CIDR, physical gateway, and each allowed address range to confirm that the start and end addresses of each address range are within the physical network CIDR range and that the order of the address ranges is valid.

[0102] Multiple allowed address ranges are merged, and the address complement is obtained by subtracting the merged allowed address ranges from the physical network CIDR. The address complement is configured as the excluded address of the external subnet so that the underlying IPAM can allocate EIPs only within the allowed address ranges.

[0103] Create a shared external subnet and define external network attachments.

[0104] Based on EIP resources carrying address pool labels, the total capacity of the egress address pool, the number of allocated EIPs, and the number of available EIPs are calculated.

[0105] When the address pool is expanded, update the excluded address range of the external subnet;

[0106] When the address pool performs a shrinking operation, if there is an already allocated EIP in the address range to be moved out, the address pool convergence will be refused and a conflict warning will be output in the address pool's status condition.

[0107] When a user exit reference to the exit address pool object is detected, a resource protection flag is set to prevent the exit address pool object from being directly deleted.

[0108] Specifically, the cluster administrator creates an egress address pool object, which at least declares the physical network CIDR, physical gateway, one or more allowed address ranges, external subnet name, external network attachment name, host physical interface, and NAT gateway scheduling conditions.

[0109] The address pool controller performs the following steps: (1) Verify the physical CIDR, physical gateway and each allowed interval to confirm that the start and end addresses are in the physical CIDR and the order is valid; (2) Merge multiple allowed intervals and calculate the address complement after subtracting the allowed interval from the physical CIDR; (3) Encode the complement into the excluded address of the external subnet so that the underlying IPAM can only allocate EIPs from the allowed intervals; (4) Create a shared External Subnet and External NAD; (5) Calculate the capacity, allocated amount and available amount of EIP resources with address pool labels; (6) Update the exclusion range when the address pool is expanded; if the address removed from the range is still in the allocated state when the pool is reduced, convergence is rejected and a conflict is reported in the status condition; (7) If there are still user exits referencing the pool, the deletion of the address pool is prevented through the finalizer.

[0110] This method automatically selects the address using the underlying IPAM, eliminating the need for users to pre-fill EIPs for their egress objects and avoiding manual checks for IP address occupancy.

[0111] Furthermore, it also includes: during the CNI ADD processing phase (i.e., the container network interface add instruction processing phase), collecting the original routing snapshot of the Pod network namespace, generating a routing configuration plan, and performing a validity check on the routing configuration plan; if the routing configuration plan fails the check, then performing route restoration based on the original routing snapshot; and saving the routing configuration summary after the check passes.

[0112] When CNI ADD processing is executed repeatedly, the route configuration summary is compared. If the route configuration summary has not changed and the actual route verification is qualified, the processing result is returned directly. If the route configuration summary has changed or the actual route verification is unqualified, route restoration is first performed based on the original route snapshot, and then the new route configuration plan is applied.

[0113] During the CNI CHECK processing phase (i.e., the container network interface verification instruction processing phase), the saved route configuration summary is compared with the actual routes in the Pod to complete the route consistency verification.

[0114] During the CNI DEL processing phase (i.e., the container network interface deletion instruction processing phase), the original route of the Pod's main network is restored based on the original route snapshot.

[0115] Specifically, the complete NAD first calls the secondary network plugin to create net1 and assign user internal addresses, and then calls the routing meta-plugin. Node-level public configurations store relatively stable cluster information, including the primary network interface, primary network gateway, Pod CIDR, Service CIDR, physical network CIDR, routing table number, and security switches; user dynamic parameters (secondary network gateway and allowed secondary network CIDRs) are written to the NAD by the user egress controller.

[0116] The plugin executes the following steps during the CNI ADD phase: (1) Parse the previous CNI results and confirm that it is running in a chained plugin manner; (2) Verify that net1 already has an IPv4 address and that the address belongs to the allowed user subnet CIDR; (3) Generate a static routing plan for the main network based on the node common CIDR set; (4) Generate an IPv4 default routing plan through net1 and the user subnet logical gateway; (5) Collect original route snapshots for the target network segment and default route that will be affected; (6) Use RouteReplace to write the main network reserved route, delete the competing default route and write the subnet default route; (7) Verify each reserved route and the unique default route; (8) If the application or verification fails, restore immediately according to the snapshot; save the state with configuration summary after success; (9) When repeating ADD, if the configuration summary is consistent and the current route is verified, return the previous result directly; if inconsistent, restore the old snapshot first and then apply the new plan; (10) CNI CHECK verifies the persistent configuration summary and the actual route, and CNI DEL restores the original route of the main network and deletes the state according to the policy.

[0117] For reference Figure 4 The diagram illustrates an internal routing scheme within a Pod, showcasing the automatic traffic splitting logic within a container group: packets are categorized and forwarded based on the network segment to which the destination address belongs; internal cluster traffic is transmitted via the eth0 primary network interface and the primary CNI; traffic between users is directly connected via the net1 secondary network interface; internet traffic enters the user's VPC via the net1 secondary network interface, and then accesses the internet through the NAT gateway and elastic public network bandwidth policy. Business containers require no modification; internal network communication for the cluster control plane, services, and storage still relies on the existing primary network, with only external network traffic entering the user's secondary network plane.

[0118] This means that business containers can automatically achieve traffic separation without any code modification. After a data packet enters the Pod network namespace, the routing meta-plugin classifies the destination address of the packet based on the pre-configured public CIDR set and user secondary network parameters, and selects the forwarding network interface according to three scenarios. Internal cluster traffic: If the destination address belongs to the cluster's public CIDR range (Pod network segment, Service network segment, node physical network segment, etc.), the traffic is forwarded by the primary network interface eth0, using the primary CNI path. Cluster control plane communication, CoreDNS domain name resolution, Service access, and storage and node intranet interaction all use the primary network plane. Internal user communication traffic: If the destination address belongs to the CIDR of the same user's independent VPC subnet, the traffic is directly forwarded through the secondary network interface net1, enabling direct secondary network communication between multiple Pods under the same user without going through an external gateway. Internet traffic: If the destination address does not belong to either of the above two network segments, it is determined to be internet access traffic. It is forwarded to the user's subnet gateway via the secondary network interface net1. The traffic enters the user's VPC, and after source address translation and rate limiting by the QoS policy of binding the elastic public IP address, it accesses the internet.

[0119] This solution achieves automatic traffic splitting through a hierarchical routing strategy within Pods. Business applications do not need to be aware of the existence of multiple network cards or modify their programs. Business traffic within the cluster remains in the original cluster main network, while only internet egress traffic enters the user's dedicated secondary network plane. Tenant-level external network bandwidth management is achieved by relying on the user's VPC, NAT gateway, and EIP bandwidth policies, while ensuring that internal cluster communication is not interfered with by egress network policies.

[0120] For example, in a Kubernetes test cluster, the Pod CIDR of the main CNI is 10.244.0.0 / 18, and the Service CIDR is 10.244.64.0 / 18.

[0121] The physical network is 192.168.251.0 / 24, and the next hop of the main network is 169.254.1.1. The egress pool is reserved from 192.168.251.221 to 192.168.251.225.

[0122] Create a user exit point for userC, with an internal network address of 10.203.0.0 / 24, a subnet logical gateway of 10.203.0.1, and a NAT Gateway internal address of 10.203.0.254. Both download and upload bandwidth are 10 Mbps. The controller automatically assigns an EIP of 192.168.251.223.

[0123] Two Pods with `sys_user_name=userC` are located on different nodes and each has the `net1` address. The test results are as follows:

[0124] A single Pod downloading 50,000,000 bytes took approximately 42 seconds, averaging about 9.52 Mbps. Two Pods from the same user each downloading 50,000,000 bytes in parallel took approximately 83 seconds, averaging about 9.64 Mbps. When userC and userD downloaded in parallel, the total aggregate speed was about 19.51 Mbps, indicating that the different user quotas are independent of each other.

[0125] 256 MiB of data was transmitted between Pods of the same user via net1, with a measured speed of approximately 1.8 to 2.0 Gbps, which was not subject to the 10 Mbps Internet QoS limit.

[0126] After deleting userD's user exit, its VPC, Subnet, NAT Gateway, EIP, SNAT, QoS, and NAD are all reclaimed, the number of allocated addresses in the address pool decreases, while userC's exit remains normal.

[0127] The above data is used to illustrate the feasibility of this application and does not constitute a numerical limitation.

[0128] The technical solution presented in this application enables user-level quotas. Rate limiting is located at the user's unified EIP, unaffected by changes in the number of Pods or scheduling nodes. Internal and external traffic are decoupled; internet traffic is rate-limited, while CoreDNS, Services, storage, nodes, and traffic within the same user's Pods are not mistakenly rate-limited. Pods operate with zero intrusion, without modifying the business image, command, or initContainer, and without requiring network management permissions for the business container. Shared namespaces are supported, with network injection automatically based on user identity; multiple users can share the business namespace while maintaining independent egress and bandwidth. Internal address planning is saved; independent VPCs allow different users to reuse the same internal CIDR, subnet logical gateway, and NAT Gateway addresses. Configuration errors are reduced; a single user egress declaration automatically generates a complete resource chain, and underlying objects use deterministic names, owner references, state conditions, and idempotent coordination. Fault recovery is possible; snapshots are saved before route modifications, and rollbacks are performed in case of failure; resource deletion is performed in dependency order, reducing the risks of network interruptions, EIP conflicts, and residual flow control rules. It is compatible with existing main CNIs. Without replacing the main network such as Calico, the existing NetworkPolicy, Service routing, and Pod lifecycle remain unchanged. It is scalable and can replace the underlying VPC / NAT implementation, flow control algorithm, identity label source, or address pool backend without altering the overall closed loop of "user claims—secure access—flow splitting—aggregation rate limiting—recycling".

[0129] This application describes a method for achieving user-level shared internet bandwidth in a Kubernetes multi-tenant cluster by leveraging primary CNI, secondary networks, virtual private networks, network address translation, elastic egress addresses, and traffic control. It uses users, rather than individual Pods, as the rate-limiting object. Through user egress declarations, multiple Pods of the same user, distributed across different nodes, are aggregated to that user's independent virtual private network, NAT gateway, and elastic public egress address. Aggregated QoS is implemented at the egress address, allowing Pods of the same user to share a single internet bandwidth quota, while communication between Pods within the same user does not go through this QoS.

[0130] The system automatically orchestrates the complete egress resource chain based on a single declaration. The controller customizes resources according to the user's egress declaration, and creates or updates virtual private networks, internal subnets, network attachments, NAT gateways, elastic egress addresses, source address translation rules, and bandwidth policies according to dependencies. It also gradually converges based on the resource status, eliminating the need for administrators to manually maintain multiple strongly related YAML files.

[0131] Identity-driven secure access in a shared namespace: Admission Webhook resolves a unique and ready user exit within the same namespace based on the pod's sys_user_name tag and automatically injects the corresponding network attachment; the verification logic prevents the pod from mounting other users' managed networks and prohibits changes to user identity or managed network annotations during runtime.

[0132] Selective traffic splitting and transactional rollback between primary and secondary networks; the chained CNI meta-plugin explicitly preserves the destination network segments within the cluster, such as Pod CIDR, Service CIDR, and physical network CIDR, on the primary CNI interface, and switches the default route for other IPv4 traffic to the secondary network; saves snapshots of affected routes before application, rolls back in case of failure, and restores according to configuration when deleted, thus eliminating the need to modify the business image, startup command, or grant Pod privileges.

[0133] Secure allocation and dependency-ordered reclamation of the shared EIP pool. The address pool controller calculates the complement of the address pool relative to the physical CIDR based on the allowed allocation range and converts the complement into external subnet excluded addresses. Before scaling down, the controller checks the allocated addresses and reclaims them in the following order when deleting user egress addresses: SNAT, QoS unbinding, EIP, NAT gateway, and internal network dependencies, reducing the risk of address conflicts and residual flow control rules.

[0134] The above elements form a closed loop: the user's identity determines the managed network, the managed network directs Internet traffic to the user's independent exit, the exit address carries user-level aggregated QoS, and the controller is responsible for the entire lifecycle of resources and fault convergence.

[0135] Example 3

[0136] Based on the same inventive concept, and corresponding to any of the methods in the above embodiments, this application also provides a user-level shared internet bandwidth control system for multi-tenant Pods. This system includes: a controller component, a variant Webhook component, a verification Webhook component, a chained container network interface meta-plugin, a network address translation gateway component, and a quality of service execution module.

[0137] The controller component is used to receive user exit declarations and exit address pool objects, perform exit resource chain orchestration and cluster-level exit address pool management; it is also used to perform exit resource reclamation according to the preset dependency order after receiving a deletion instruction, set resource protection flags when reclamation fails, re-queue the reclamation coordination task into the processing queue for retry, and clear resource protection flags after all resources have been reclaimed.

[0138] The variant Webhook component is used to respond to Pod creation events, read the user identifier carried by the Pod, query the unique and ready exit resources in combination with the Pod's business namespace, inject the corresponding managed network attachment definition into the Pod's network annotation, and retain the Pod's original main container network interface network attachments.

[0139] The Webhook component is used to perform permission verification logic to prevent Pods without user identifiers from mounting managed network attachment definitions. Pods with user identifiers are only allowed to mount managed network attachment definitions belonging to themselves, and modification of the Pod's user identifier and managed network annotations is prohibited during the Pod runtime update phase.

[0140] The chained container network interface meta-plugin is used to configure the primary and secondary network interfaces for Pods; and based on the cluster public classless inter-domain route set pre-configured on the node side, it retains the network segment routes corresponding to the cluster public classless inter-domain route set on the primary network interface, and switches the default routes of other Internet traffic to the secondary network interface.

[0141] The Network Address Translation (NAT) gateway component is used to implement source network address translation. The Quality of Service (QoS) enforcement module, bound to the Elastic Public Address (IPA), is used to implement bandwidth control on internet traffic flowing through the IPA.

[0142] More specifically, the controller components include a cluster-level egress address pool controller and a user egress controller. For example, in a concrete implementation, the above declarations and plugins are respectively named EgressIPPool, UserEgress, and user-egress-route.

[0143] refer to Figure 5The diagram shows the overall layered architecture of a Pod-level user egress bandwidth management system. The system is divided into a control plane, an admission plane, an access and routing layer, and a Pod data plane from top to bottom. It also includes two levels of resource pools: shared egress resources and independent egress resources for each user.

[0144] Resource input sources: The cluster administrator creates an egress address pool declaration (EgressIPPool) to define a globally shared external network address resource pool; the business platform creates a user egress declaration (UserEgress) to submit user egress bandwidth, network segment, namespace and other business requirements; the business Pod carries the user identifier (sys_user_name) to participate in scheduling access.

[0145] Control plane: Consists of address pool controller and user egress controller. Address pool controller uniformly manages shared egress resources, including external subnets, network attachment definitions, and elastic public address pools; user egress controller listens to user egress declarations and automatically creates independent resources for each user according to a phased orchestration process: virtual private networks, user internal subnets, network attachment definitions, NAT gateways, elastic public addresses, source network address translation rules, and bidirectional quality of service policies, constructing a dedicated external network egress link for each user.

[0146] The access plane consists of a variant webhook and a verification webhook. During the business Pod creation phase, it reads the user identifier carried by the Pod, matches it with the corresponding ready user egress resource, injects the configuration required for secondary network access into the Pod, and enables the business Pod to securely access the matching user's independent network, ensuring the correct association between the Pod and the user egress resource.

[0147] Access and Routing Layer: Relying on the Multus multi-NIC component, Kube-OVN network plugin, and user egress routing meta plugin, primary and secondary NICs are generated for Pods, and traffic splitting routing rules are dynamically deployed within the Pod network namespace to achieve traffic classification and scheduling.

[0148] Pod Data Plane: Internally, each Pod uses two network interface cards (NICs) for traffic separation. eth0 is the primary NIC, handling internal cluster communication traffic; net1 is the secondary NIC, handling internet access traffic. External network traffic enters the user's independent egress resource via the secondary NIC, ultimately accessing the internet. Aggregated bandwidth limiting is implemented for all external network traffic from all Pods of the same user at the user's dedicated elastic public IP address.

[0149] This architecture enables declarative resource orchestration, non-intrusive Pod access, and isolation of primary and secondary network traffic. It relies on users' independent egress resources to achieve tenant-level external network bandwidth aggregation and control, and the cluster's internal network traffic is not constrained by egress bandwidth policies.

[0150] correspond Figure 5Specifically, the cluster administrator configures a shared egress address pool, and the business platform issues user egress declarations; the controller in the control plane automatically orchestrates the declared resources to generate independent egress network resources for each user; the admission plane completes user network matching and access during the Pod creation phase through Webhook; the access and routing layers implement multi-NIC and traffic splitting routing based on Multus and routing meta-plugins; the Pod data plane relies on the primary NIC to carry cluster traffic and the secondary NIC to carry internet traffic, and external network traffic enters the user's dedicated egress link, completing the aggregation and rate limiting of traffic from multiple Pods of the same user on the elastic public IP address.

[0151] Example 4

[0152] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 6 As shown, the electronic device 500 includes one or more processors 501 and memory 502.

[0153] The processor 501 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 500 to perform desired functions.

[0154] Memory 502 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and processor 501 may execute the program instructions to implement the user-level shared internet bandwidth control method for multi-tenant Pods in any embodiment of this application described above, and / or other desired functions. Various contents such as initial extrinsic parameters and thresholds may also be stored in the computer-readable storage medium.

[0155] In one example, the electronic device 500 may further include an input device 503 and an output device 504, these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 503 may include, for example, a keyboard, a mouse, etc. The output device 504 may output various information to the outside. The output device 504 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0156] Of course, for the sake of simplicity, Figure 6Only some of the components of the electronic device 500 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 500 may include any other suitable components depending on the specific application.

[0157] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the user-level shared internet bandwidth control method for multi-tenant Pods provided in any embodiment of this application.

[0158] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0159] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the user-level shared internet bandwidth control method for multi-tenant Pods provided in any embodiment of this application.

[0160] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0161] It should be noted that the terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitations, an element defined by the phrase "comprising an..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element.

[0162] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. The above descriptions are only preferred embodiments of this application. It should be noted that due to the limitations of written expression, while there are objectively infinite specific structures, those skilled in the art can make several improvements, modifications, or changes without departing from the principles of this invention, and can also combine the above technical features in an appropriate manner. These improvements, modifications, changes, or combinations, or the direct application of the inventive concept and technical solution to other situations without modification, should all be considered within the scope of protection of this application.

Claims

1. A method for controlling user-level shared internet bandwidth in a multi-tenant Pod, characterized in that, include: Upon receiving a user's exit statement, perform user exit resource chain orchestration operations to generate the user's exit resources; The outbound resources include network address translation gateways, elastic public IP addresses, source network address translation rules, user-defined virtual private networks, and quality of service bandwidth policies bound to the elastic public IP addresses; In response to the creation of a Pod, the user identifier carried by the Pod is read. Using the business namespace of the Pod and the user identifier as a joint query condition, a unique and ready exit resource is found. The managed network attachment definition corresponding to the exit resource is injected into the network annotation of the Pod, and the original main container network interface network attachment of the Pod is retained. The execution permission verification logic is used to prevent Pods without user identifiers from mounting managed network attachment definitions, so that Pods with user identifiers are only allowed to mount managed network attachment definitions belonging to themselves, and the modification of user identifiers and managed network annotations is prohibited during the Pod runtime update phase. Configure the primary and secondary network interfaces for the Pod, and pre-configure the cluster common classless inter-domain route set on the node side. Keep the network segment routes corresponding to the cluster common classless inter-domain route set on the primary network interface, and switch the default routes of other Internet traffic to the secondary network interface.

2. The user-level shared internet bandwidth control method for multi-tenant Pods according to claim 1, characterized in that, Also includes: In response to a user's exit deletion command, resource reclamation is performed on the entire set of exit resources according to a preset dependency order. During the resource reclamation process, if any reclamation step fails, a resource protection flag is set to prevent the corresponding exit resource object from being directly deleted, and the resource reclamation task is re-queued for processing, waiting to be re-executed. When all exit resources have been reclaimed, the resource protection flag is cleared.

3. The user-level shared internet bandwidth control method for multi-tenant Pods according to claim 2, characterized in that, The process of recycling the entire set of exported resources according to a preset dependency order includes: Prioritize deleting the source network address translation rule; Remove the binding relationship between the elastic public IP address and the quality of service bandwidth policy, and wait for the underlying components to complete the state clearing of the binding relationship; Next, delete the Elastic Public IP Address, Quality of Service Bandwidth Policy, Network Address Translation Gateway, User Internal Subnet, Managed Network Attachment Definition, and User Independent Virtual Private Network in sequence; Once all the above-mentioned export resources have been cleared, the resource protection markers will be removed, and the occupied address capacity corresponding to the export address pool will be released.

4. The user-level shared internet bandwidth control method for multi-tenant Pods according to claim 1, characterized in that, Also includes: Receive an exit address pool object, which is configured with at least the following: physical network classless inter-domain route, physical gateway, one or more allowed address ranges, external subnet name, external network attachment name, host physical interface, and network address translation gateway scheduling conditions. Verify the Classless Inter-Domain Routes, Physical Gateways, and each allowed address range of the physical network to confirm that the start and end addresses of each address range are within the range of the Classless Inter-Domain Routes of the physical network and that the order of the address ranges is valid. Multiple allowed address ranges are merged, and the address complement is obtained by subtracting the merged allowed address ranges from the classless inter-domain routes of the physical network. The address complement is configured as the excluded address of the external subnet so that the underlying address management module can allocate elastic public network addresses only within the allowed address ranges. Create a shared external subnet and define external network attachments.

5. The user-level shared internet bandwidth control method for multi-tenant Pods according to claim 4, characterized in that, Also includes: Based on the elastic public IP address resources carrying address pool labels, the total capacity of the egress address pool, the number of allocated elastic public IP addresses, and the number of available elastic public IP addresses are statistically analyzed. When the address pool is expanded, update the excluded address range of the external subnet; When the address pool performs a shrinking operation, if there are already allocated elastic public network addresses in the address range to be moved out, the address pool convergence will be refused and a conflict prompt will be output in the address pool status condition. When a user exit reference to the exit address pool object is detected, a resource protection flag is set to prevent the exit address pool object from being directly deleted.

6. The user-level shared internet bandwidth control method for multi-tenant Pods according to claim 1, characterized in that, The step of performing user exit resource chain orchestration operations to generate user exit resources includes: Perform parameter validation on the user's export declaration. If the validation passes, proceed with the following steps: Create user-independent virtual private networks and user-internal subnets, and create bidirectional quality of service policies that bind elastic public IP addresses; First, create a startup network attachment definition that contains only the secondary network plugin, then create a network address translation gateway; Elastic public IP addresses are allocated by the underlying address management module. After the elastic public IP addresses reach the ready state through polling, source network address translation rules are created. The startup network attachment definition is upgraded to a complete chained network attachment definition that includes sub-network plugins and routing plugins, and the elastic public IP address information and readiness status are written back to the user egress statement.

7. The user-level shared internet bandwidth control method for multi-tenant Pods according to claim 6, characterized in that, The creation of user-independent virtual private networks and user-internal subnets includes: Configure static internet routes pointing to the network address translation gateway within the user's independent virtual private network; Add the internal addresses of the subnet logical gateway and network address translation gateway to the IP exclusion list of the user's internal subnet; Write the sub-network gateway, which is used as a user dynamic parameter, and the allowed sub-network classless inter-domain routes into the complete chained network attachment definition.

8. The user-level shared internet bandwidth control method for multi-tenant Pods according to claim 1, characterized in that, Also includes: During the instruction processing phase of the container network interface, the original route snapshot of the Pod network namespace is collected, a route configuration plan is generated, and the validity of the route configuration plan is verified. If the route configuration plan fails the verification, route restoration is performed based on the original route snapshot. After the verification passes, the route configuration summary is saved. When repeatedly executing the container network interface add command processing, compare the route configuration summary. If the route configuration summary has not changed and the actual route verification is qualified, return the processing result directly. If the route configuration summary changes or the actual route verification fails, route restoration is first performed based on the original route snapshot, and then the new route configuration plan is applied. During the container network interface verification instruction processing phase, the saved route configuration summary is compared with the actual routes in the Pod to complete the route consistency verification. During the container network interface deletion instruction processing phase, the original route of the Pod's main network is restored based on the original route snapshot.

9. The user-level shared internet bandwidth control method for multi-tenant Pods according to claim 1, characterized in that, The Quality of Service (QoS) bandwidth strategy employs a hierarchical token bucket queue algorithm to implement traffic control. For internet download traffic, bandwidth control is implemented on the inbound direction of the external communication interface of the network address translation gateway; For internet upload traffic, bandwidth control is implemented on the outbound direction of the external communication interface of the network address translation gateway.

10. A user-level shared internet bandwidth control system for multi-tenant Pods, characterized in that, The method for performing any one of claims 1-9 includes: a controller component, a variant Webhook component, a verification Webhook component, a chained container network interface meta-plugin, a network address translation gateway component, and a quality of service execution module; The controller component is used to receive user exit declarations and exit address pool objects, perform exit resource chain orchestration and cluster-level exit address pool management; it is also used to perform exit resource reclamation according to a preset dependency order after receiving a deletion instruction, set a resource protection flag when reclamation fails, re-arrange the reclamation coordination task into the processing queue for retry, and clear the resource protection flag after all resources have been reclaimed. The modified Webhook component is used to respond to Pod creation events, read the user identifier carried by the Pod, query the unique and ready exit resources in combination with the Pod's business namespace, inject the corresponding managed network attachment definition into the Pod's network annotation, and retain the Pod's original main container network interface network attachments. The verification Webhook component is used to execute permission verification logic to prevent Pods without user identifiers from mounting managed network attachment definitions. Pods with user identifiers are only allowed to mount managed network attachment definitions belonging to themselves. Furthermore, during the Pod runtime update phase, it is prohibited to modify the Pod's user identifier and managed network annotations. The chained container network interface meta-plugin is used to configure the primary network interface and secondary network interface for Pods; and based on the cluster public classless inter-domain route set pre-configured on the node side, to retain the network segment route corresponding to the cluster public classless inter-domain route set on the primary network interface, and to switch the default route of other Internet traffic to the secondary network interface. The network address translation gateway component is used to implement source network address translation; The Quality of Service (QoS) execution module is bound to the Elastic Public Address (IPA) and is used to implement bandwidth control on Internet traffic flowing through the IPA.

Images