Multi-tenant container and virtual machine integrated scheduling management system based on K8s

Abstract

The invention discloses a multi-tenant container and virtual machine integrated scheduling management system based on K8s. The system comprises a unified access layer; and a core control plane which is constructed on the basis of a Kubernetes API Server, and is used for receiving the data from the core control plane, the system comprises a multi-tenant management module used for realizing multi-tenant resource isolation and dynamic management and control, an integrated scheduling module used for realizing unified scheduling of a container and a virtual machine, a virtual resource management module used for managing the life cycle of the virtual machine, and a network management and control module used for realizing tenant network isolation and intercommunication, the storage management and control module is used for realizing tenant storage isolation and cross-protocol sharing; the operation and maintenance closed-loop module is used for realizing tenant-level monitoring and self-healing; a resource abstraction layer; and the infrastructure layer comprises a container node for running a container load, a virtualization node for running a virtual machine load, a network plug-in and a storage system. The system has the characteristics of thorough isolation, intelligent scheduling and high operation and maintenance efficiency.

Description

Technical Field

[0001] This invention belongs to the field of computer technology and relates to a scheduling and management system, specifically a multi-tenant container and virtual machine integrated scheduling and management system based on K8s (Kubernetes). Background Technology

[0002] With the deepening of enterprise digital transformation, business scenarios are becoming increasingly complex; in the current multi-tenant cloud computing scenario, the demand for mixed management of containers and virtual machines is growing. However, the existing K8s-based management solutions have significant limitations: (1) Multi-tenant management focuses on quota control of single resources (such as containers or virtual machines only), and basic isolation is achieved through K8s Namespace or virtual machine resource pools, without forming a unified isolation system across resource types of "computing-network-storage"; (2) The scheduling process does not associate tenant priority with resource usage thresholds, which can easily lead to high-priority tenant resources being preempted; the operation interface is mostly a separate console for containers and virtual machines, and tenants need to switch interfaces to manage different types of resources, resulting in low operation and maintenance efficiency.

[0003] Therefore, there is an urgent need for a system that can achieve unified abstraction of containers and virtual machines, deeply integrate multi-tenant isolation and priority scheduling, and provide an integrated operation and maintenance experience. Summary of the Invention

[0004] To address the aforementioned shortcomings, this invention provides an integrated scheduling and management system for multi-tenant containers and virtual machines based on Kubernetes.

[0005] To achieve the above objectives, the present invention provides a multi-tenant container and virtual machine integrated scheduling and management system based on Kubernetes, comprising: A unified access layer is provided to offer an integrated operation portal, enabling a unified view, hybrid deployment, and operation and maintenance management of container and virtual machine resources in multi-tenant scenarios. The core control plane, built on the Kubernetes API Server, includes a multi-tenant management module for implementing multi-tenant resource isolation and dynamic management, an integrated scheduling module for implementing unified scheduling of containers and virtual machines, a virtualization resource management module for managing the lifecycle of virtual machines, a network management module for implementing tenant network isolation and interconnection, a storage management module for implementing tenant storage isolation and cross-protocol sharing, and an operation and maintenance closed-loop module for implementing tenant-level monitoring and self-healing. The resource abstraction layer abstracts tenants, services, virtual machines, and storage resources through a series of custom resource definition CRDs, constructing a three-dimensional model of "service-resource-tenant". The infrastructure layer includes container nodes for running container workloads, virtualization nodes for running virtual machine workloads, network plug-ins, and storage systems.

[0006] Optimally, the multi-tenant management module includes a Tenant-Controller and a TenantResource CRD. The TenantResource CRD is used to define the tenant's unique identifier, priority, total resource quota, compute pool resource ratio, network policy, and storage policy. The Tenant-Controller is used to monitor the TenantResource CRD instance and execute dynamic adjustment logic for tenant resources. The dynamic adjustment logic includes triggering idle-time resource sharing to the global shared resource pool based on a resource utilization threshold, and triggering automatic resource expansion during busy hours based on a resource utilization warning threshold.

[0007] Ideally, the integrated scheduling module implements a four-dimensional scheduling process by extending the Kubernetes scheduler. The four-dimensional scheduling process is executed sequentially: Tenant priority filtering sorts and filters according to the priority of the tenant to which the load to be scheduled belongs; Tenant quota pre-verification filters out nodes whose remaining resources are insufficient to meet the current request or that may cause the tenant's total resource usage to exceed the quota; Business needs matching: Candidate nodes are scored based on business type and service quality requirements; Resource status binding, combined with real-time node load and affinity strategy, selects the final node.

[0008] Furthermore, the integrated scheduling module also executes a resource balancing strategy within a tenant to ensure that the actual resource occupancy ratio of containers and virtual machines conforms to the computing pool ratio defined in the TenantResource CRD.

[0009] Ideally, the network management module is implemented by extending the Calico CNI plugin: Assign and bind a dedicated IP address pool to each tenant; By default, all cross-tenant network communication is prohibited, and cross-tenant access rules that require approval are defined through TenantNetworkPolicy CRD; Based on the tenant priority and bandwidth quota defined in TenantResource CRD, differentiated QoS bandwidth guarantees are provided for different tenants and services with different priorities within the same tenant. The Service-Bridge component automatically registers and discovers tenant content providers and virtual machine services, and provides internal domain name resolution based on service names.

[0010] Ideally, the storage management module is implemented via a Multi-Protocol-CSI plugin: By binding persistent storage volumes to specific tenants through tags, a tenant-specific storage pool is formed, and it is ensured that the tenant can only access the storage volume bound to itself; It supports containers mounting the same unified storage volume simultaneously via the CSI protocol and virtual machines via the iSCSI or NFS protocol, and automatically performs protocol conversion; By using a tenant-level distributed lock mechanism and read / write permissions defined in the workload annotations, concurrent access to the shared storage volume by containers and virtual machines is controlled, ensuring data consistency.

[0011] Ideally, the operation and maintenance closed-loop module includes: The Metric-Fusion component is used to automatically add tenant identification tags to monitoring data collected from Prometheus, enabling tenant-level isolation and aggregation of monitoring data; TenantFaultCorrelate-Controller has a built-in cross-type fault correlation rule library, which is used to analyze fault data of tenant containers, virtual machines, networks and storage, and locate the root cause; The Heal-Controller is used to automatically execute predefined self-healing strategies based on the type of failure. These strategies include performing hot migration of virtual machines, restarting and rescheduling of containers, and switching of storage volumes.

[0012] Ideally, the "Business-Resource-Tenant" 3D model is implemented using the following CRD: TenantResource CRD is used to define tenants and their resource blueprints; The WorkloadProfile CRD is used to define the business template and is associated with TenantResource through the tenant identifier field; The VMInstance CRD is used to define a virtual machine and is associated with a TenantResource through the tenant identifier field; UnifiedPV CRD is used to define a unified storage volume and includes a tenant identification tag; TenantNetworkPolicy CRD is used to define cross-tenant network access policies.

[0013] Ideally, the integrated operation portal is used for: Provides a unified resource view at the tenant level, which graphically displays the resource status and quota usage of tenant containers, virtual machines, networks, and storage. It provides a visual hybrid deployment wizard to guide users in defining hybrid applications that include container components and virtual machine components through a graphical interface, and automatically generates the corresponding Kubernetes resource description files; Provides a fine-grained resource management interface, allowing tenant administrators to adjust the computing pool resource ratio, apply for quota expansion, and manage off-peak resource sharing settings; It provides a closed-loop interface for cross-mode operation and maintenance, integrating tenant-level monitoring, log query, and one-click fault self-healing functions.

[0014] This invention is a multi-tenant container and virtual machine integrated scheduling and management system based on Kubernetes, which features thorough isolation, intelligent scheduling, high operation and maintenance efficiency, and flexible and precise management and control. Thorough isolation: Through a three-tier tenant isolation system of "computing-network-storage", cross-tenant interference and data leakage risks are fundamentally eliminated, meeting the high security requirements of scenarios such as finance and government affairs; Intelligent scheduling: The innovative four-dimensional scheduling model deeply integrates tenant priority, resource quota and business awareness, which not only ensures the SLA of core tenants, but also improves the overall resource utilization of the cluster to more than 85%. High operational efficiency: The integrated operation interface reduces the deployment time of hybrid services from hours to minutes, and the cross-mode fault location time from 30 minutes to 5 minutes, improving operational efficiency by more than 70%. Flexible and precise management: It supports customizable ratio of tenant server / virtual machine resources, resource sharing during idle hours, and dynamic expansion during peak hours, making resource management highly flexible and perfectly adapting to changes in business peaks. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the structure of the K8s-based multi-tenant container and virtual machine integrated scheduling and management system of the present invention; Figure 2 This is the overall architecture diagram of the multi-tenant container and virtual machine integrated scheduling and management system based on K8s of this invention; Figure 3 This is a four-dimensional scheduling flowchart of the integrated scheduling module of the present invention; Figure 4 This is a schematic diagram of the core functions of the integrated user interface of the present invention; Figure 5 This is a schematic diagram of the "business-resource-tenant" three-dimensional abstract model of the present invention; Figure 6 This is a schematic diagram of multi-tenant cross-protocol storage sharing according to the present invention; Figure 7 This is a schematic diagram of the multi-tenant integrated operation and maintenance closed loop of the present invention. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art under the premise of equivalent changes and modifications should fall within the protection scope of the present invention.

[0017] like Figure 1 and Figure 2 The Kubernetes-based multi-tenant container and virtual machine integrated scheduling and management system shown mainly includes a unified access layer, a core control plane, a resource abstraction layer, and an infrastructure layer.

[0018] This invention is a multi-tenant container and virtual machine integrated scheduling and management system based on Kubernetes, featuring thorough isolation, intelligent scheduling, high operational efficiency, and flexible and precise control. Thorough isolation: Through a three-layer tenant isolation system of "computing-network-storage," it fundamentally eliminates cross-tenant interference and data leakage risks, meeting the high security requirements of scenarios such as finance and government. Intelligent scheduling: The innovative four-dimensional scheduling model deeply integrates tenant priority, resource quotas, and business awareness, ensuring the SLA of core tenants while increasing the overall cluster resource utilization rate to over 85%. High operational efficiency: The integrated operation interface reduces the deployment time for mixed services from hours to minutes, and the cross-type fault location time from 30 minutes to 5 minutes, improving operational efficiency by over 70%. Flexible and precise control: It supports customizable container / virtual machine resource ratios within tenants, resource sharing during idle periods, and dynamic scaling during peak periods, making resource management highly flexible and perfectly adaptable to changes in business peaks.

[0019] The unified access layer provides an integrated operation portal, enabling a unified view, hybrid deployment, and operation and maintenance management of container and virtual machine resources in multi-tenant scenarios. Specifically, the unified access layer provides an integrated operation portal, offering a unified Web UI and API entry point for system administrators, tenant administrators, and ordinary users. This portal provides a tenant-level unified resource view, a visual hybrid deployment wizard, fine-grained resource management, and cross-mode operation and maintenance monitoring capabilities, enabling single sign-on and management of all resources.

[0020] The core control plane is built on the Kubernetes API Server and includes a multi-tenant management module for multi-tenant resource isolation and dynamic management, an integrated scheduling module for unified scheduling of containers and virtual machines, a virtualization resource management module for managing the lifecycle of virtual machines, a network management module for tenant network isolation and interconnection, a storage management module for tenant storage isolation and cross-protocol sharing, and an operation and maintenance closed-loop module for tenant-level monitoring and self-healing.

[0021] The multi-tenant management module includes Tenant-Controller and TenantResource CRD. TenantResource CRD defines a tenant's unique identifier, priority, total resource quota, compute pool resource allocation, network policies, and storage policies. Tenant-Controller monitors TenantResource CRD instances and executes dynamic adjustment logic for tenant resources. This dynamic adjustment logic includes triggering idle-time resource sharing to the global shared resource pool based on resource utilization thresholds, and triggering automatic resource expansion during busy periods based on resource utilization warning thresholds. Specifically, the multi-tenant management module uses Tenant-Controller as its core, defining tenants and their resource quotas (CPU, memory, storage, bandwidth), compute pool allocation (container and virtual machine resource allocation ratio), and resource sharing policies through custom TenantResource components, thus achieving dynamic management of tenant resources throughout their entire lifecycle. For example, the core logic for achieving dynamic management of tenant resources throughout their entire lifecycle through the Tenant-Controller component includes: Idle Resource Sharing Release: Real-time monitoring of tenant resource utilization. When the utilization of a certain resource type is below the threshold (e.g., CPU utilization ≤ 30%) and the "Resource Sharing Switch" is on, Tenant-Controller automatically marks idle resources exceeding the threshold (e.g., 10 CPU cores) as "shareable resources", adds them to the global shared resource pool, and records "shared resource owner tenant", "sharing duration", and "maximum borrowing amount". When the tenant's resource utilization rises back above the threshold (e.g., CPU utilization ≥ 50%), the shared resources are automatically reclaimed.

[0022] Resource scaling during peak hours: When a tenant's utilization of a certain type of resource exceeds the warning threshold (e.g., CPU utilization ≥ 80%), Tenant-Controller triggers the following process: Push expansion reminders to tenant administrators (via the integrated operation interface or email), displaying "current resource usage", "suggested expansion quota", and "sources of available expansion resources" (such as the global shared resource pool or newly added nodes in the cluster). After the tenant administrator confirms the expansion, Tenant-Controller prioritizes allocating resources from the global shared resource pool (such as borrowing 5 CPU cores from P2 tenant). If the shared resources are insufficient, the cluster node expansion process is triggered (such as automatically creating new nodes and adding them to the tenant subnet). After the expansion is completed, the TenantResource quota will be automatically updated (e.g., the CPU quota will be temporarily increased from 100 cores to 105 cores), and an "expansion validity period" (e.g., 72 hours) will be set. After the validity period expires, the temporary expansion resources will be automatically reclaimed (if the tenant still needs to use them, they need to apply again).

[0023] Excess Quota Warning and Restriction: If a tenant's business attempts to request resources exceeding the quota (e.g., a virtual machine requests 10 CPU cores, but the tenant only has 5 cores remaining), the scheduler will directly refuse to deploy and push a "Resource Insufficiency Warning" to the tenant administrator through the Tenant-Controller. At the same time, "Resource Optimization Suggestions" (such as reducing low-priority business resources or requesting quota adjustments) will be displayed on the integrated interface.

[0024] The integrated scheduling module implements a four-dimensional scheduling process by extending the Kubernetes scheduler, such as... Figure 3 As shown, the four-dimensional scheduling process is executed sequentially: Tenant priority filtering sorts and filters tenants based on their priorities. The scheduler first reads the TenantResource priorities associated with the service to be deployed, prioritizing high-priority tenant services in the order of "P0→P1→P2". If high-priority tenants lack sufficient resources (e.g., P0 tenant has 5 CPU cores remaining, but the service requires 8 cores), the "idle resource borrowing" logic is triggered—the global shared resource pool is searched, and idle resources of low-priority tenants (e.g., P2) are borrowed first, with the borrowing amount not exceeding 50% of the low-priority tenants' idle resources (to avoid affecting low-priority services).

[0025] Tenant quota pre-verification filters out nodes whose remaining resources are insufficient to meet the current request or that would cause the tenant's total resource usage to exceed the quota. For the selected candidate nodes, the following checks are performed: "Whether the node's remaining resources are within the tenant's quota": Computational resource verification: If the tenant currently uses 90 CPU cores (quota of 100 cores), and the service to be deployed requires 8 cores, then only nodes with ≥8 remaining CPU cores and the tenant's total usage does not exceed 100 cores are selected; Network resource verification: Check whether the node belongs to the tenant's dedicated subnet or whether cross-subnet access permissions have been configured; Storage resource verification: Confirm that the storage volume requested by the service belongs to the tenant's dedicated storage pool and that IOPS usage does not exceed the quota.

[0026] Business needs matching: Candidate nodes are scored based on business type (such as online transactions, offline computing, etc.) and service quality requirements; Resource status binding, combined with real-time node load and affinity strategy, selects the final node.

[0027] The integrated scheduling module also executes intra-tenant resource balancing strategies to ensure that the actual resource utilization ratio of containers and virtual machines conforms to the compute pool ratio defined in the TenantResource CRD. Its core functional diagram is shown below. Figure 4 As shown. If both container and virtual machine services exist within a tenant, the scheduler automatically allocates resources according to the TenantResource "Compute Pool Ratio": For example, if the tenant's compute pool is configured as "60% for containers and 40% for virtual machines," and the tenant's total CPU quota is 100 cores, then the container service can use a maximum of 60 cores, and the virtual machine service can use a maximum of 40 cores; if the container service is currently using only 30 cores (not reaching the limit), and the virtual machine service needs to add 15 cores (not reaching the 40-core limit), then the scheduler will allocate resources normally; if the virtual machine service needs to add 20 cores (exceeding the 40-core limit), then deployment will be rejected and a message will be displayed: "The virtual machine resources within the tenant have reached the quota limit. You can adjust the compute pool ratio or apply for quota expansion."

[0028] The virtualization resource management module is based on the KubeVirt project. It uses VMInstance to customize the lifecycle of virtual machines and encapsulates virtual machines as K8s Pods, thereby incorporating them into a unified scheduling and management system.

[0029] The network management module extends the Calico CNI plugin to achieve the following: allocating and binding a dedicated IP address pool to each tenant; prohibiting all cross-tenant network communication by default, and defining cross-tenant access rules requiring approval through the TenantNetworkPolicy CRD; providing differentiated QoS bandwidth guarantees for different tenants and services with different priorities within a tenant based on the tenant priority and bandwidth quota defined in the TenantResource CRD; and automatically registering and discovering tenant container and virtual machine services through the Service-Bridge component, providing internal domain name resolution based on service names.

[0030] The storage management module is implemented through the Multi-Protocol-CSI plugin: persistent storage volumes are bound to specific tenants via tags, forming tenant-specific storage pools and ensuring that tenants can only access the storage volumes they are bound to; it supports containers mounting the same unified storage volume simultaneously via the CSI protocol and virtual machines via iSCSI or NFS protocols, with automatic protocol conversion; and it controls concurrent access to shared storage volumes by containers and virtual machines through a tenant-level distributed locking mechanism and read / write permissions defined in the workload annotations, ensuring data consistency (e.g., ...). Figure 6 (As shown).

[0031] This module extends the CSI plugin and adds a tenant storage permission controller to achieve a storage system of "tenant-specific storage pool - cross-mode sharing - fine-grained permissions". The core optimizations are as follows: Tenant-specific storage pool construction: An extension based on the Multi-Protocol-CSI plugin has been added, introducing a "tenant storage pool filtering" function to ensure that tenants can only access their own dedicated storage resources: Storage pool and tenant binding: When creating a UnifiedPV (Unified Storage Volume), it is bound to a specific tenant via the "tenant-id" tag (e.g., "tenant-id: tenant-finance-001"). The Multi-Protocol-CSI plugin only shows the UnifiedPV with this tag to the tenant. Tenant storage volume classification: In the "Storage Pool" field of TenantResource, dedicated storage volumes are classified by "Purpose" (such as "Database Storage Volume", "Log Storage Volume", "Backup Storage Volume"). Tenant administrators can allocate the corresponding type of storage volume to the business as needed to avoid the abuse of storage resources; Storage resource isolation verification: When a tenant requests to mount a storage volume, the Multi-Protocol-CSI plugin first verifies whether the "tenant-id label of the storage volume is consistent with the tenant ID". If they are inconsistent, the mounting is rejected to ensure cross-tenant storage access isolation.

[0032] Cross-format storage sharing control: The protocol conversion logic of Multi-Protocol-CSI has been optimized to support tenant servers and virtual machines securely sharing the same storage volume. The core mechanism is as follows: Multi-protocol adaptation: The Multi-Protocol-CSI plugin supports containers mounting the same UnifiedPV simultaneously via the CSI protocol and virtual machines via the iSCSI / NFS / CEPH protocol. The plugin automatically completes the protocol conversion (such as converting CSI requests to iSCSI commands) without requiring manual configuration by the tenant. Distributed lock mechanism: To avoid data inconsistency caused by cross-mode access, a new "tenant-level distributed lock" has been added: When a container and a virtual machine are mounted on the same storage volume at the same time, the Multi-Protocol-CSI plugin automatically adds a distributed lock to the storage volume and controls read and write operations according to the "permission type". For example, if a container is configured with "read-only permission" and a virtual machine is configured with "read-write permission", then the distributed lock allows the virtual machine to write data, while the container can only read the latest data; if both are configured with "read-write permission", then the locking mechanism ensures that only one resource can be written to at the same time, avoiding data conflicts. Shared status monitoring: The Multi-Protocol-CSI plugin monitors the mounting status of storage volumes in real time (such as "Current mounted resource type: 1 container + 1 virtual machine" and "Permission configuration: container read-only, virtual machine read-write") and synchronizes it to the TenantResource "Storage Pool" status field. Tenant administrators can view the sharing status in the integrated interface.

[0033] Fine-grained storage access control: A new TenantStorageAuth-Controller has been added to implement three-level access control based on "tenant-business-resource type": Tenant-level permissions: By default, all tenants only have "access rights" to their own dedicated storage volumes, and do not have "delete / modify storage volume configuration" permissions (this permission is only available to system administrators); Business-level permissions: Configure the "Storage permissions" field in the WorkloadProfile (e.g., "Database business: read and write permissions; Log business: read-only permissions"), and TenantStorageAuth-Controller will automatically assign corresponding permissions to the containers and virtual machines associated with the business. Resource-level permissions: Supports configuring different permissions for containers and virtual machines under the same business (e.g., "container: read-only; virtual machine: read-write"). Permission configuration is implemented through the "storage-auth" annotation of VMInstance and Pod. Permission Changes and Auditing: Tenant administrators can modify storage permissions through an integrated interface. TenantStorageAuth-Controller records "permission change records" (change time, changer, original permissions, new permissions) to support subsequent auditing; permission changes take effect in real time without restarting the container or virtual machine.

[0034] Tenant Storage QoS and Security: To address the performance and security requirements of tenant storage, the following mechanisms have been added: Storage QoS Guarantee: Configure "IOPS Quota" and "Bandwidth Quota" in the "Storage Pool" field of TenantResource. The Multi-Protocol-CSI plugin ensures that tenant storage IOPS and bandwidth do not exceed the quota through flow control, avoiding excessive storage resources occupied by a single tenant. Data encryption: Provides "storage volume encryption" for tenants with high security levels (such as financial tenants). TenantResource configures a tenant-specific encryption key, and the Multi-Protocol-CSI plugin automatically uses this key to encrypt data when storing it and decrypts it when reading it, ensuring data security. Backup and recovery: Supports tenant administrators to trigger "storage volume backup" in the integrated interface. Backup data is stored in the tenant's dedicated backup storage volume and supports point-in-time recovery (such as restoring to the state 24 hours ago) to avoid data loss.

[0035] The operation and maintenance closed-loop module includes: the Metric-Fusion component, which automatically adds tenant identification tags to monitoring data collected from Prometheus to achieve tenant-level isolation and aggregation of monitoring data; the TenantFaultCorrelate-Controller, which has a built-in cross-morphological fault correlation rule library to analyze fault data of tenant containers, virtual machines, networks and storage, and locate the root cause; and the Heal-Controller, which automatically executes predefined self-healing strategies according to the fault type. These strategies include performing hot migration of virtual machines, restarting and rescheduling of containers, and switching of storage volumes.

[0036] This module, supported by underlying technology modules and the user interface, implements an operation and maintenance system of "tenant-level monitoring - fault correlation analysis - automatic self-healing". The core optimizations are as follows: Tenant-level monitoring data isolation and aggregation: Extending based on Prometheus and Metric-Fusion components, and connecting to the monitoring dashboard of the operation interface, it achieves isolation and aggregation of tenant operation and maintenance data: Monitoring data isolation: When Prometheus collects resource monitoring metrics, it automatically adds a "tenant-id" tag to the metrics (such as "container_cpu_usage_seconds_total {tenant-id="tenant-finance-001"}). The Metric-Fusion component only returns metrics with this tag to tenants, ensuring cross-tenant monitoring data isolation. Tenant Metric Aggregation: Metric-Fusion supports aggregating and displaying similar metrics of tenant containers and virtual machines (such as "Total Tenant CPU Utilization = (Total Container CPU Utilization + Total Virtual Machine CPU Utilization) / Tenant CPU Quota"), generating "Global Tenant Metrics", and synchronizing them to the operation interface for tenant administrators to view; Custom monitoring rules: Tenant administrators can create "custom monitoring rules" (such as "trigger an alarm if container CPU utilization exceeds 90% for 5 consecutive minutes") through the operation interface. The rules only apply to the current tenant, and the alarm information is pushed to the "Resource Alarm Summary" module of the operation interface in real time.

[0037] Cross-morphological fault correlation analysis: A new TenantFaultCorrelate-Controller has been added, which, combined with the fault information in the user interface, enables correlation analysis of cross-type faults within a tenant: Fault data acquisition: The controller collects fault information of tenant servers, virtual machines, storage, and networks in real time, including "fault time, fault type, fault resource ID, fault description, and associated service ID", and synchronizes it to the operation interface; Association rule library: Built-in "cross-morphology fault association rules", for example: Rule 1: If a virtual machine storage volume fails and the container is mounted on the same storage volume, then it is determined that "the container service abnormality is caused by a virtual machine storage failure"; Rule 2: If container network access times out and the corresponding virtual machine network port is not open, then it is determined that "the container access error is caused by the virtual machine network configuration"; Rule 3: If multiple containers and virtual machines within a tenant simultaneously exceed the CPU utilization threshold, it is determined that "insufficient tenant resource quotas have led to excessive global load". Association analysis results display: The controller pushes the association analysis results to the operation interface, displaying "root cause failure resources", "list of affected resources", "fault propagation path" and providing "handling suggestions".

[0038] Automatic self-healing within the tenant: Optimize the Heal-Controller component to integrate with the "one-click fault self-healing" function in the user interface, enabling automatic self-healing of cross-type faults within a tenant: Self-healing permission control: Heal-Controller only performs self-healing operations on resources within a tenant, and the scope of the operation does not exceed the tenant quota (e.g., virtual machine migration can only be migrated to nodes within the tenant). Categorized self-healing strategy: Develop in-tenant self-healing strategies for different fault types. Container failure: Core business containers trigger "immediate restart + reschedule to an idle node within the tenant", while non-core business containers trigger "delayed restart (e.g., after 5 minutes)"; Virtual machine failure: Core business virtual machines trigger "hot migration to other nodes within the tenant", while non-core business virtual machines trigger "reboot + remount storage volume"; Storage failure: Triggers "Switch to standby storage volume within the tenant" and automatically synchronizes the most recent backup data; Network failure: Triggers "Reconfigure intra-tenant network policy" to restore cross-mode communication; Self-healing result feedback: After the self-healing operation is completed, Heal-Controller will synchronize the "self-healing result" and "operation details" to the operation interface and notify the tenant administrator; if self-healing fails, "manual handling guidance" will be provided.

[0039] Tenant log management and auditing: Extend the Log-Bridge component to connect to the "Log Query" function in the user interface, enabling the isolation, correlation, and auditing of tenant logs: Log isolation collection: Log-Bridge only collects container and virtual machine logs with the "tenant-id" tag and stores them in a tenant-specific log storage to avoid cross-tenant log leakage; Cross-format log association: Supports associating container and virtual machine logs by "business ID" or "request ID". After entering the query conditions in the operation interface, the associated logs are returned in real time. Log auditing: Tenant administrators can configure "log auditing rules" through the operation interface. Log-Bridge automatically retains audit logs and supports exporting them by time range and log type to meet compliance requirements.

[0040] The resource abstraction layer uses a series of custom resource definitions (CRDs) to abstract and isolate tenants, services, virtual machines, and storage resources, constructing a three-dimensional "service-resource-tenant" model. This "service-resource-tenant" model is implemented using the following CRDs (e.g., ...). Figure 5As shown): TenantResource CRD, used to define tenants and their resource blueprints; WorkloadProfile CRD, used to define business templates and associated with TenantResource through the tenant identifier field; VMInstance CRD, used to define virtual machines and associated with TenantResource through the tenant identifier field; UnifiedPV CRD, used to define unified storage volumes and with tenant identifier tags; TenantNetworkPolicy CRD, used to define cross-tenant network access policies.

[0041] Through the collaboration of Profile-Controller and Tenant-Controller, real-time synchronization of the "business-resource-tenant" status is achieved. The core process is as follows: Resource status synchronization: Profile-Controller collects the running status of VMInstances and container Pods in real time (such as CPU utilization and storage mount status) and synchronizes it to the "status" field of the corresponding WorkloadProfile; Tenant quota synchronization: Tenant-Controller periodically (every 30 seconds) calculates the total resource usage of all VMInstances and container Pods within a tenant and updates the "status.usedQuota" field of TenantResource to ensure that the tenant quota view is accurate in real time; Business status feedback: When the business associated with the WorkloadProfile has an anomaly (such as response time exceeding the threshold), Profile-Controller automatically checks whether it is caused by "insufficient tenant resources" (such as the tenant CPU quota being full, and the business being unable to scale). If it is a quota issue, it pushes an "insufficient resources causing business anomaly" alarm to the tenant administrator. The 3D model provides foundational data support for multi-tenant scheduling, ensuring that the scheduler can obtain full information on "tenant priority, tenant quota, business requirements, and resource status": The scheduler quickly locates the tenant by using the "tenant-id" tag of the VMInstance and WorkloadProfile; it obtains the tenant priority and resource quota through TenantResource for "tenant priority filtering" and "quota pre-verification"; and it completes "business requirement matching" by matching the "business type and latency threshold" of the WorkloadProfile with the "hardware requirements" of the VMInstance. The state synchronization of the 3D model ensures that the resource data obtained by the scheduler is real-time and valid, avoiding incorrect scheduling decisions based on outdated data.

[0042] Isolation design includes tenant-level isolation, resource-type-level isolation, and business-level isolation.

[0043] Tenant-level isolation: Each tenant corresponds to a unique TenantResource instance. Core fields include: Basic identifier: Tenant ID (e.g., "tenant-finance-001"), Tenant name ("Financial Core Tenant"), Priority (P0 / P1 / P2, P0 is the highest); Total resource quota: CPU (e.g., 100 cores), Memory (e.g., 200 Gi), Storage capacity (e.g., 1TB), Network bandwidth (e.g., 1Gbps); Resource sharing switch: Whether to allow idle resources to be added to the "Global Shared Resource Pool", and the maximum sharing ratio (e.g., not exceeding 50% of idle resources); Security configuration: Tenant-specific data encryption key (e.g., RSA key pair), cross-tenant access permissions (disabled by default, requires administrator approval to enable).

[0044] Resource type layer isolation: Under TenantResource, subfields such as "Compute Pool," "Network Pool," and "Storage Pool" are defined to achieve type-based resource management. Compute Pool: Configure the resource ratio of tenant container Pods and virtual machines (VMInstances) (e.g., 60% for containers and 40% for virtual machines), and the maximum resource limit for a single service (e.g., a maximum of 8 CPU cores per virtual machine). Network pool: Allocate tenant-specific subnet segments (e.g., "10.244.1.0 / 24"), network policies (e.g., prohibiting virtual machines within the tenant from accessing the external public network), and QoS baselines (e.g., a minimum guaranteed bandwidth of 100Mbps for services within the tenant). Storage pool: Bind tenant-specific unified storage volumes (a subset of UnifiedPV, such as “tenant-finance-pv-01” and “tenant-finance-pv-02”), storage IOPS quota (such as 5000 IOPS), and data backup policies (such as full backup every day at midnight).

[0045] Business layer isolation: Add a "tenant identifier" field to the original WorkloadProfile CRD to force association with the TenantResource of the tenant to ensure that business resource requests do not exceed the tenant quota; at the same time, add "business priority within the tenant" (such as "core transaction" and "non-core query") to realize secondary scheduling of resources within the tenant.

[0046] The infrastructure layer comprises container nodes for running container workloads, virtualization nodes for running virtual machine workloads, network plugins, and storage systems. It includes standard Kubernetes container nodes, virtualization nodes with KubeVirt installed, CNI network plugins (such as Calico) and CSI storage plugins supporting multi-tenant isolation, and backend storage systems (such as Ceph), providing stable and scalable physical resources for the upper layers.

[0047] System implementation prerequisites: Deploy a Kubernetes 1.26 cluster and configure the RBAC permission system (add the roles “SystemAdmin”, “TenantAdmin”, and “TenantUser”). Deploy core components: Tenant-Controller, Profile-Controller, TenantStorageAuth-Controller, TenantFaultCorrelate-Controller, extended Calico CNI, Multi-Protocol-CSI, and a unified web interface (developed based on Vue + Element UI). Initialize 2 tenants: Tenant A (tenant-finance-001): P0 priority, CPU quota of 100 cores, memory of 200 Gi, storage quota of 1TB, bandwidth quota of 1Gbps, with off-peak resource sharing enabled (maximum share of 50%). Tenant B (tenant-test-002): P2 priority, CPU quota of 50 cores, memory quota of 100Gi, storage quota of 500GB, bandwidth quota of 500Mbps, with off-peak resource sharing enabled (maximum share of 50%).

[0048] The above system implementation steps are as follows: Step 1: Create tenant resource CRD and dedicated resource pool Deploy a TenantResource instance (tenant A): By submitting the application using `kubectl apply -f tenant-finance.yaml`, Tenant-Controller automatically creates a dedicated subnet and storage pool for tenant A.

[0049] Create a dedicated UnifiedPV for tenant A: Submitting via `kubectl apply -f unifiedpv-finance-01.yaml` will mark the file as a dedicated storage volume for tenant A using the Multi-Protocol-CSI plugin.

[0050] Step 2: Deploy multi-tenant hybrid services (depends on the 4.5 user interface) Tenant A's administrator logs into the integrated operation interface, enters the "Hybrid Deployment Wizard," and configures the "Backup Transportation Scheduling Service." Business Information: Name: "Equipment Transportation Dispatch Service", Type: "Online Transaction", Tenant Priority: "Core"; Resource components: Add container (Tomcat: 9.0, 2 cores 4Gi), add virtual machine (CentOS 7.9, 4 cores 8Gi); Storage configuration: Select "tenant-finance-pv-01" and set "container read-only, virtual machine read-write"; Network configuration: Select subnet "10.244.1.0 / 24", QoS bandwidth 500Mbps; After submitting the deployment, the interface displays "Resource usage: CPU 6 cores (6%), memory 12Gi (6%), sufficient quota".

[0051] The scheduler executes a four-dimensional scheduling process: Tenant priority filtering: Prioritize processing business from tenant A (P0); Quota pre-verification: The node belongs to tenant A subnet, the storage volume is a dedicated resource, and the resource usage has not exceeded the quota; Intra-tenant resource balancing: The container / virtual machine ratio conforms to the 6:4 configuration, and the operation interface displays "Running" after deployment.

[0052] Step 3: Tenant resource adjustment and operation and maintenance verification (connecting to the 4.6 operation and maintenance module) Tenant B's idle resource sharing: CPU utilization 20% (10 idle cores). Tenant-Controller adds 5 cores to the shared pool, which can be viewed in the "Resource Sharing Records" section of the operation interface.

[0053] Tenant A Expansion: The service requires an additional 8 CPU cores. The scheduler prioritizes allocating the remaining resources of Tenant A. If the quota is insufficient, it will automatically borrow shared resources from Tenant B. The "Quota Management" interface will display "Temporary Expansion of 5 Cores".

[0054] Self-healing: Tenant A's virtual machine storage volume failed, and the operation interface issued an alert "VM1 storage failure affects Pod2". Click "One-click self-healing": The Heal-Controller triggers a hot migration of the virtual machine to tenant A node and switches to the standby storage volume; Five minutes later, the interface displays "Self-healing successful", and the migration details can be viewed in the log query.

[0055] Log auditing: Tenant A administrator queries the "Equipment Transportation Scheduling Service" request logs through the operation interface to obtain the container and virtual machine association logs and locate latency issues.

[0056] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. At the same time, those skilled in the art should understand and implement the above description. Therefore, any equivalent changes or modifications made without departing from the concept disclosed in the present invention should be covered within the scope of protection of the present invention.

Claims

1. A multi-tenant container and virtual machine integrated scheduling and management system based on Kubernetes, characterized in that it include: A unified access layer is provided to offer an integrated operation portal, enabling a unified view, hybrid deployment, and operation and maintenance management of container and virtual machine resources in multi-tenant scenarios. The core control plane, built on the Kubernetes API Server, includes a multi-tenant management module for implementing multi-tenant resource isolation and dynamic management, an integrated scheduling module for implementing unified scheduling of containers and virtual machines, a virtualization resource management module for managing the lifecycle of virtual machines, a network management module for implementing tenant network isolation and interconnection, a storage management module for implementing tenant storage isolation and cross-protocol sharing, and an operation and maintenance closed-loop module for implementing tenant-level monitoring and self-healing. The resource abstraction layer abstracts tenants, services, virtual machines, and storage resources through a series of custom resource definition CRDs, constructing a three-dimensional model of "service-resource-tenant". The infrastructure layer includes container nodes for running container workloads, virtualization nodes for running virtual machine workloads, network plug-ins, and storage systems.

2. The Kubernetes-based multi-tenant container and virtual machine integrated scheduling and management system according to claim 1, characterized in that: The multi-tenant management module includes a Tenant-Controller and a TenantResource CRD. The TenantResource CRD is used to define the tenant's unique identifier, priority, total resource quota, compute pool resource ratio, network policy, and storage policy. The Tenant-Controller is used to monitor the TenantResource CRD instance and execute dynamic adjustment logic for tenant resources. The dynamic adjustment logic includes triggering the sharing of idle resources to the global shared resource pool based on a resource utilization threshold, and triggering automatic expansion of resources during busy periods based on a resource utilization warning threshold.

3. The Kubernetes-based multi-tenant container and virtual machine integrated scheduling and management system according to claim 1, characterized in that: The integrated scheduling module implements a four-dimensional scheduling process by extending the Kubernetes scheduler. The four-dimensional scheduling process is executed sequentially: Tenant priority filtering sorts and filters according to the priority of the tenant to which the load to be scheduled belongs; Tenant quota pre-verification filters out nodes whose remaining resources are insufficient to meet the current request or that may cause the tenant's total resource usage to exceed the quota; Business needs matching: Candidate nodes are scored based on business type and service quality requirements; Resource status binding, combined with real-time node load and affinity strategy, selects the final node.

4. The Kubernetes-based multi-tenant container and virtual machine integrated scheduling and management system according to claim 3, characterized in that: The integrated scheduling module also executes a resource balancing strategy within a tenant to ensure that the actual resource usage ratio of containers and virtual machines conforms to the computing pool ratio defined in the TenantResource CRD.

5. The Kubernetes-based multi-tenant container and virtual machine integrated scheduling and management system according to claim 1, characterized in that: The network management module achieves this by extending the Calico CNI plugin: Assign and bind a dedicated IP address pool to each tenant; By default, all cross-tenant network communication is prohibited, and cross-tenant access rules that require approval are defined through TenantNetworkPolicy CRD; Based on the tenant priority and bandwidth quota defined in TenantResource CRD, differentiated QoS bandwidth guarantees are provided for different tenants and services with different priorities within the same tenant. The Service-Bridge component automatically registers and discovers tenant content providers and virtual machine services, and provides internal domain name resolution based on service names.

6. The Kubernetes-based multi-tenant container and virtual machine integrated scheduling and management system according to claim 1, characterized in that: The storage management module is implemented through the Multi-Protocol-CSI plugin: By binding persistent storage volumes to specific tenants through tags, a tenant-specific storage pool is formed, and it is ensured that the tenant can only access the storage volume bound to itself; It supports containers mounting the same unified storage volume simultaneously via the CSI protocol and virtual machines via the iSCSI or NFS protocol, and automatically performs protocol conversion; By using a tenant-level distributed lock mechanism and read / write permissions defined in the workload annotations, concurrent access to the shared storage volume by containers and virtual machines is controlled, ensuring data consistency.

7. The Kubernetes-based multi-tenant container and virtual machine integrated scheduling and management system according to claim 1, characterized in that, The operation and maintenance closed-loop module includes: The Metric-Fusion component is used to automatically add tenant identification tags to monitoring data collected from Prometheus, enabling tenant-level isolation and aggregation of monitoring data; TenantFaultCorrelate-Controller has a built-in cross-type fault correlation rule library, which is used to analyze fault data of tenant containers, virtual machines, networks and storage, and locate the root cause; The Heal-Controller is used to automatically execute predefined self-healing strategies based on the type of failure. These strategies include performing hot migration of virtual machines, restarting and rescheduling of containers, and switching of storage volumes.

8. The Kubernetes-based multi-tenant container and virtual machine integrated scheduling and management system according to claim 1, characterized in that, The "Business-Resource-Tenant" three-dimensional model is implemented through the following CRD: TenantResource CRD is used to define tenants and their resource blueprints; The WorkloadProfile CRD is used to define a business template and is associated with TenantResource through the tenant identifier field. The VMInstance CRD is used to define a virtual machine and is associated with a TenantResource through the tenant identifier field; UnifiedPV CRD is used to define a unified storage volume and includes a tenant identification tag; TenantNetworkPolicy CRD is used to define cross-tenant network access policies.

9. The Kubernetes-based multi-tenant container and virtual machine integrated scheduling and management system according to claim 1, characterized in that, The integrated operation portal is used for: Provides a unified resource view at the tenant level, which graphically displays the resource status and quota usage of tenant containers, virtual machines, networks, and storage. It provides a visual hybrid deployment wizard to guide users in defining hybrid applications that include container components and virtual machine components through a graphical interface, and automatically generates the corresponding Kubernetes resource description files; Provides a fine-grained resource management interface, allowing tenant administrators to adjust the computing pool resource ratio, apply for quota expansion, and manage off-peak resource sharing settings; It provides a closed-loop interface for cross-mode operation and maintenance, integrating tenant-level monitoring, log query, and one-click fault self-healing functions.

Images