Cluster computing resource isolation method and device, electronic equipment and storage medium

Abstract

The invention provides a cluster computing resource isolation method and device, electronic equipment and a storage medium, and relates to the technical field of cluster computing resource management. And scheduling the containerized task to an adaptive physical graphics processor node based on the information, intercepting a resource access request during task operation, allocating an isolated video memory space and a computing context according to an allocated virtual graphics processor resource limit, and dynamically maintaining an isolated state of multiple tasks, so that the task access efficiency is improved. The problems of inter-task interference, limited hardware compatibility, insufficient resource allocation flexibility and low utilization rate caused by lack of a flexible virtualization scheme for decoupling from hardware and imperfect resource scheduling and isolation mechanisms in the prior art can be solved.

Description

Technical Field

[0001] This disclosure relates to the field of cluster computing resource management technology, and in particular to a cluster computing resource isolation method and apparatus, electronic device and storage medium. Background Technology

[0002] With the continuous expansion of AI model scale and the rapid growth of multi-tenant concurrent computing demands, data centers are placing higher demands on the efficient reuse, flexible scheduling, and stable operation of GPU resources. However, the systemic bottlenecks in core performance and application adaptability of existing GPU virtualization technologies are insufficient to meet the application needs of real-world scenarios. On the one hand, existing solutions either adopt a shared mode, leading to mutual interference between tasks, or rely on hardware partitioning, resulting in insufficient adaptability and flexibility, failing to simultaneously address the needs for isolation, compatibility, and fine-grained allocation. On the other hand, due to the defects in resource allocation mechanisms, most scenarios still require a deployment method where a single task exclusively occupies a single GPU, leaving a large number of GPU resources idle or operating inefficiently for extended periods. This not only causes a serious waste of computing resources but also significantly increases the deployment and maintenance costs of AI applications, hindering the large-scale implementation and popularization of AI technology across various industries. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for isolating cluster computing resources. Its main purpose is to at least partially solve one of the technical problems in related technologies.

[0004] According to a first aspect of this disclosure, a method for isolating cluster computing resources is provided, comprising: In a containerized cluster, physical graphics processor resources are virtualized and abstracted to generate and report virtual graphics processor resource information. Based on the virtual graphics processor resource information, the containerized tasks that request virtual graphics processor resources are scheduled, and the containerized tasks are allocated to physical graphics processor nodes with corresponding remaining resources. During the execution of the containerized task, the access requests of the containerized task to the graphics processor resources are intercepted, and according to the virtual graphics processor resource limit allocated to the containerized task, video memory space and computing context are allocated and isolated on the physical graphics processor. The isolation status of each containerized task's memory space and computing context on the physical graphics processor is dynamically maintained to ensure resource partitioning and isolation when multiple containerized tasks share the same physical graphics processor.

[0005] Optionally, the virtualization abstraction of physical graphics processor resources includes: The video memory resources of the physical graphics processor are quantized into multiple independent virtual video memory resource units according to a preset granularity. Different models of physical graphics processor hardware are managed through a unified abstract interface, generating virtual graphics processor resource information with a unified description format.

[0006] Optionally, scheduling containerized tasks requesting virtual graphics processor resources based on the reported virtual graphics processor resource information includes: Receive resource request descriptions submitted by one or more containerized tasks, wherein each resource request description specifies the amount of virtual graphics processor resources requested by the corresponding task; Based on the resource request description and the virtual resource description information, the one or more containerized tasks are scheduled to one or more computing nodes that have sufficient corresponding remaining resources.

[0007] Optionally, intercepting the containerized task's access requests to graphics processor resources includes: Intercept the graphics processor memory operation instructions issued by the containerized task; Based on the virtual memory limit set for the containerized task, a corresponding independent memory area is allocated from the physical graphics processor's memory space for the containerized task to use, and the memory operation instructions are redirected to the independent memory area.

[0008] Optionally, the dynamic maintenance of the isolation state of each containerized task's resource space on the physical graphics processor includes: Monitor the actual usage of physical graphics processor resources by each containerized task; When any task is detected to be using more resources than its allocated limit, a resource reclamation operation is performed to restore the isolation state.

[0009] Optionally, allocating and isolating video memory and computing context on the physical graphics processor according to the virtual graphics processor resource quota allocated for the containerized task includes: By loading the virtualization driver library into the process space of the containerized task, the graphics processor resource operation instructions issued by the containerized task are intercepted. Based on the resource limit allocated to the containerized task, a private memory address range belonging to the containerized task is mapped in the global memory address space of the physical graphics processor. The intercepted memory operation instructions are redirected to the private memory address range for execution, and an independent computing context is created for the containerized task, thereby achieving dual isolation between the memory space and the computing context.

[0010] According to a second aspect of this disclosure, a cluster computing resource isolation device is provided, comprising: The virtualization unit is used to virtualize and abstract physical graphics processor resources in a containerized cluster, and to generate and report virtual graphics processor resource information. The scheduling unit is used to schedule containerized tasks that request virtual graphics processor resources based on the virtual graphics processor resource information, and allocate the containerized tasks to physical graphics processor nodes with corresponding remaining resources. The allocation unit is used to intercept the containerized task's access request to the graphics processor resources during the execution of the containerized task, and allocate and isolate video memory space and computing context on the physical graphics processor according to the virtual graphics processor resource limit allocated to the containerized task. The maintenance unit is used to dynamically maintain the isolation status of the video memory space and computing context of each containerized task on the physical graphics processor, ensuring resource partitioning and isolation when multiple containerized tasks share the same physical graphics processor.

[0011] Optionally, the virtualization unit is also used for: The video memory resources of the physical graphics processor are quantized into multiple independent virtual video memory resource units according to a preset granularity. Different models of physical graphics processor hardware are managed through a unified abstract interface, generating virtual graphics processor resource information with a unified description format.

[0012] Optionally, the scheduling unit is also used for: Receive resource request descriptions submitted by one or more containerized tasks, wherein each resource request description specifies the amount of virtual graphics processor resources requested by the corresponding task; Based on the resource request description and the virtual resource description information, the one or more containerized tasks are scheduled to one or more computing nodes that have sufficient corresponding remaining resources.

[0013] Optionally, the allocation unit is also used for: Intercept the graphics processor memory operation instructions issued by the containerized task; Based on the virtual memory limit set for the containerized task, a corresponding independent memory area is allocated from the physical graphics processor's memory space for the containerized task to use, and the memory operation instructions are redirected to the independent memory area.

[0014] Optionally, the maintenance unit is also used for: Monitor the actual usage of physical graphics processor resources by each containerized task; When any task is detected to be using more resources than its allocated limit, a resource reclamation operation is performed to restore the isolation state.

[0015] Optionally, the allocation unit is also used for: By loading the virtualization driver library into the process space of the containerized task, the graphics processor resource operation instructions issued by the containerized task are intercepted. Based on the resource limit allocated to the containerized task, a private memory address range belonging to the containerized task is mapped in the global memory address space of the physical graphics processor. The intercepted memory operation instructions are redirected to the private memory address range for execution, and an independent computing context is created for the containerized task, thereby achieving dual isolation between the memory space and the computing context.

[0016] According to a third aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect above.

[0017] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first aspect above.

[0018] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0019] The cluster computing resource isolation method, apparatus, electronic device, and storage medium disclosed herein virtualize and abstract physical graphics processor resources in a containerized cluster and report relevant information. Based on this information, containerized tasks are scheduled to suitable physical graphics processor nodes. During task execution, resource access requests are intercepted, and isolated video memory space and computing context are allocated according to the allocated virtual graphics processor resource quota. At the same time, the isolation status of multiple tasks is dynamically maintained. Therefore, it can solve the problems of inter-task interference, limited hardware compatibility, insufficient resource allocation flexibility, and low utilization caused by the lack of flexible virtualization schemes decoupled from hardware and imperfect resource scheduling and isolation mechanisms in the prior art. It achieves fine-grained segmentation and strong isolation of physical graphics processor resources in a containerized environment, supports multiple containerized tasks to share the same physical graphics processor, and improves the technical effects of resource utilization and task operation stability.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 A flowchart illustrating a cluster computing resource isolation method provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of the structure of a cluster computing resource isolation device provided in an embodiment of the present disclosure; Figure 3 A schematic block diagram of an example electronic device provided for embodiments of this disclosure. Detailed Implementation

[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] The following description, with reference to the accompanying drawings, outlines a method and apparatus for isolating cluster computing resources, an electronic device, and a storage medium according to embodiments of the present disclosure.

[0024] Figure 1 This is a flowchart illustrating a cluster computing resource isolation method provided in an embodiment of the present disclosure.

[0025] like Figure 1 As shown, the method includes the following steps: Step 101: In the containerized cluster, the physical graphics processor resources are virtualized and abstracted to generate and report virtual graphics processor resource information.

[0026] In the embodiments of this disclosure, within a containerized cluster environment, a preset virtualization abstraction mechanism abstracts the hardware resources of the physical graphics processing unit (GPU) away from its specific hardware architecture. This transforms the core resources of the GPU, such as computing power and video memory, into virtual graphics processing unit (vGPU) resource information that is recognizable and uniformly managed by the cluster. Then, through a resource reporting mechanism adapted to the containerized cluster, the generated vGPU resource information is synchronized to the cluster's resource management center, providing basic data support for subsequent resource request and scheduling decisions for containerized tasks. As one implementation method, a device detection component integrated into the containerized cluster scheduling system can abstract and quantify various resources of the physical GPU, generating standardized vGPU resource description information and reporting it to the cluster control plane.

[0027] Existing technologies suffer from the problem of physical graphics processor resources being strongly bound to hardware architecture, making it difficult to be uniformly managed by containerized clusters. The embodiments disclosed in this disclosure can realize the virtualized unified representation and reporting of physical graphics processor resources, laying the foundation for subsequent flexible scheduling and fine-grained resource allocation, and improving the compatibility of containerized clusters with different models of physical graphics processors and the uniformity of resource management.

[0028] Step 102: Based on the virtual graphics processor resource information, schedule the containerized tasks that request virtual graphics processor resources and allocate the containerized tasks to physical graphics processor nodes with corresponding remaining resources.

[0029] In the embodiments of this disclosure, based on the virtual graphics processor (vGPU) resource information reported to the containerized cluster resource management center, after receiving a virtual GPU resource request from a containerized task, the cluster scheduling mechanism performs real-time verification and supply-demand matching assessment of the remaining virtual GPU resources on each physical GPU node. Based on resource adaptation principles, it executes scheduling decisions, accurately allocating the containerized task to a physical GPU node whose remaining resources can meet its request, ensuring the task receives stable virtualization resource support. As one implementation method, efficient scheduling and allocation of containerized tasks can be achieved by extending the scheduler functionality of the containerized cluster and combining it with the global resource status information maintained by the cluster control plane.

[0030] Existing technologies suffer from problems such as a lack of effective adaptation of scheduling mechanisms to virtualized resources and low accuracy in matching tasks with physical graphics processor resources. The embodiments disclosed in this disclosure can achieve precise docking between containerized tasks and physical graphics processor nodes, improve the scientificity and efficiency of scheduling decisions, provide scheduling-level guarantees for multiple containerized tasks to share physical graphics processor resources, and help improve the rationality of overall resource utilization.

[0031] Step 103: During the execution of the containerized task, intercept the containerized task's access request to the graphics processor resources, and allocate and isolate video memory space and computing context on the physical graphics processor according to the virtual graphics processor resource limit allocated to the containerized task.

[0032] In the embodiments of this disclosure, when a containerized task is running, a preset resource access interception mechanism captures various access requests from the task to the physical graphics processing unit (GPU) resources. Simultaneously, it retrieves the virtual graphics processing unit (vGPU) resource quota parameters pre-allocated to the containerized task. Based on these quota parameters, it allocates dedicated video memory space for the task on the corresponding physical GPU and constructs an independent computing context. This ensures that the resource usage of the task is strictly limited to the allocated quota and is separated from the resource usage space and computing logic of other tasks. As one implementation method, a virtualization driver component can be injected into the runtime environment of the containerized task to intercept GPU-related application programming interface (API) calls initiated by the task. Combined with resource quotas, this allows for precise allocation of video memory space and independent configuration of the computing context.

[0033] Existing technologies suffer from poor resource isolation and insufficient allocation precision when multiple tasks share a physical graphics processor. The embodiments disclosed in this disclosure can achieve dual isolation and precise control of video memory space and computing context, effectively avoiding resource competition and performance interference between tasks, and providing core technical support for the stable concurrent operation of multiple containerized tasks on the same physical graphics processor.

[0034] Step 104: Dynamically maintain the isolation status of the video memory space and computing context of each containerized task on the physical graphics processor to ensure resource partitioning and isolation when multiple containerized tasks share the same physical graphics processor.

[0035] In the embodiments of this disclosure, a dynamic isolation maintenance mechanism is established to monitor and regulate the isolation status of the allocated video memory space and computing context of each containerized task on the physical graphics processing unit (GPU) in real time. This mechanism responds promptly to changes in resource usage during task execution, proactively avoids anomalies that may compromise isolation boundaries, and ensures that even when multiple containerized tasks share the same physical graphics processing unit resource, their respective resource usage scopes remain strictly defined and independent. As one implementation method, a virtualization driver component can continuously track the resource access behavior and status changes of each task. Once a tendency to exceed the isolation boundary is detected, a regulation mechanism is immediately activated to maintain the isolation status, ensuring the independence of multi-task parallel execution.

[0036] Existing technologies suffer from problems such as easy failure of isolation state and insufficient stability when multiple tasks share a physical graphics processor. The embodiments disclosed in this disclosure can achieve dynamic stability of the isolation state, ensuring the resource independence and clear boundary of multiple containerized tasks running concurrently for a long time, and further improving the reliability of physical graphics processor resource sharing and the overall stability of system operation.

[0037] The cluster computing resource isolation method disclosed herein virtualizes and abstracts physical graphics processor resources in a containerized cluster and reports relevant information. Based on this information, it schedules containerized tasks to suitable physical graphics processor nodes, intercepts resource access requests during task execution, and allocates isolated video memory space and computing context according to the allocated virtual graphics processor resource quota. At the same time, it dynamically maintains the isolation status of multiple tasks. Therefore, it can solve the problems of inter-task interference, limited hardware compatibility, insufficient resource allocation flexibility, and low utilization caused by the lack of flexible virtualization schemes decoupled from hardware and imperfect resource scheduling and isolation mechanisms in the prior art. It achieves fine-grained segmentation and strong isolation of physical graphics processor resources in a containerized environment, supports multiple containerized tasks to share the same physical graphics processor, and improves the technical effects of resource utilization and task running stability.

[0038] As a specific implementation of this disclosure, based on the basic scheme, the virtualization abstraction of physical graphics processor resources is further defined as follows: the video memory resources of the physical graphics processor are quantized into multiple independent virtual video memory resource units according to a preset division granularity; different models of physical graphics processor hardware are managed through a unified abstraction interface to generate virtual graphics processor resource information with a unified description format.

[0039] Specifically, when virtualizing and abstracting the video memory resources of a physical graphics processing unit (GPU), a preset partitioning granularity (such as 1MB or 1GB as the smallest quantization unit) is first set. Based on this granularity, the physical video memory is divided and quantized into multiple independent virtual video memory resource units that do not overlap. Each unit has a unique identifier and a clear capacity attribute. At the same time, a unified hardware abstraction interface is constructed. This interface is pre-adapted to the hardware characteristics and resource acquisition protocols of different models of physical graphics processing units (such as T4, A100, and domestic chips). Through this interface, core information such as the total video memory and computing power parameters of each model of physical graphics processing unit is collected in batches. Then, it is standardized and packaged according to a preset unified description specification (such as the key-value pair format of "nvidia.com / vgpu: XGi"). Finally, a unified format of virtual graphics processing unit resource information that can be directly recognized by containerized clusters is generated.

[0040] By quantizing virtual memory resource units, fine-grained management of memory resources is achieved. By using a unified abstract interface and description format, the hardware barriers between different models of physical graphics processors are broken down. This not only improves the flexibility of virtual graphics processor resource allocation, but also ensures the compatibility of containerized clusters with heterogeneous GPU hardware, laying the foundation for unified scheduling and resource reuse across GPU models in the future.

[0041] As a specific implementation of this disclosure, based on the basic scheme, the scheduling of containerized tasks requesting virtual graphics processor resources based on the reported virtual graphics processor resource information is further defined as follows: receiving resource request descriptions submitted by one or more containerized tasks, wherein each resource request description specifies the amount of virtual graphics processor resources requested by the corresponding task; and scheduling the one or more containerized tasks to one or more computing nodes with sufficient corresponding remaining resources according to the resource request descriptions and the virtual resource description information.

[0042] Specifically, containerized tasks submit resource requests using Pods (the smallest deployment unit in Kubernetes). The resource request description for each containerized task is specified through the `resources.limits` field in a configuration file (such as a YAML file), explicitly indicating the requested amount of virtual graphics processing unit (vGPU) resources. The description format is "nvidia.com / vgpu: "Xm" (where Xm is a resource quantification value, for example, 1500m corresponds to 1.5GB of video memory resources); the vGPU scheduler (hami-scheduler), which is extended from the Kubernetes scheduler, continuously receives these resource request descriptions and synchronously obtains the virtual graphics processor resource information (including core data such as the remaining vGPU video memory and computing power of the node) reported by each compute node in the cluster. It compares and evaluates the request amount of each task with the remaining resources of each node in real time through a preset resource matching algorithm. If the remaining virtual graphics processor resources of a single compute node can meet the request requirements of multiple containerized tasks, these tasks are scheduled to that node in batches. If the resources of a single node are insufficient to support the total request amount of a task or multiple tasks, the relevant tasks are scheduled to multiple different compute nodes with sufficient remaining resources to ensure that each containerized task can be allocated to a physical graphics processor node with suitable resources.

[0043] By clearly defining the standardized format of resource request descriptions and the precise comparison logic during scheduling, efficient adaptation of containerized tasks and computing nodes is achieved. This not only supports the concurrent scheduling of multiple tasks to the same node to achieve physical graphics processor resource sharing, but also allows for flexible allocation to different nodes based on resource availability. This improves the rationality of scheduling decisions and the accuracy of resource matching, laying the scheduling foundation for subsequent multi-task sharing of physical graphics processor resources and further enhancing the overall resource utilization efficiency of the cluster.

[0044] As a specific implementation of this disclosure, based on the basic solution, the interception of the containerized task's access request to the graphics processor resources is further limited to: intercepting the graphics processor memory operation instructions issued by the containerized task; allocating a corresponding independent memory area from the physical graphics processor's memory space for the containerized task to use, according to the virtual memory limit set for the containerized task, and redirecting the memory operation instructions to the independent memory area.

[0045] Specifically, the vCUDA driver library (libvgpu.so) injected into the runtime environment of the containerized task (i.e., Pod, the smallest deployment unit in Kubernetes) intercepts resource access requests. It focuses on capturing various graphics processing unit (GPU) memory operation instructions initiated by the containerized task, including memory allocation instructions (such as cudaMalloc), memory release instructions (such as cudaFree), and memory data read / write instructions. After intercepting the above instructions, the vCUDA driver library immediately retrieves the virtual memory limit parameters pre-set for the containerized task. Based on these parameters, it allocates an independent memory region in the overall memory space of the physical graphics processor through an address mapping mechanism. This region has a unique address identifier and does not overlap with the memory regions of other containerized tasks. Then, the target address of the intercepted memory operation instructions is redirected so that the instructions can only access this independent memory region and cannot touch other memory spaces of the physical graphics processor.

[0046] By precisely intercepting memory operation instructions and combining them with address redirection technology, a rigid constraint on the use of memory by containerized tasks is achieved through virtual memory quotas. This not only avoids a single task from over-consuming memory resources, but also completely eliminates memory interference between multiple tasks through the physical isolation of independent memory areas. This significantly enhances the isolation and control precision of memory resources, providing core technical support for multiple containerized tasks to securely share the memory resources of the same physical graphics processor, while also improving the level of refined utilization of physical memory.

[0047] As a specific implementation of this disclosure, based on the basic scheme, the dynamic maintenance of the isolation state of the resource space of each containerized task on the physical graphics processor is further defined as follows: monitoring the actual occupation of the physical graphics processor resources by each containerized task; when it is detected that the resource occupation of any task exceeds its allocation limit, a resource reclamation operation is performed to restore the isolation state.

[0048] Specifically, during dynamic maintenance of the isolation state, the vCUDA driver library (libvgpu.so) injected into containerized tasks (i.e., Pods, the smallest deployment unit in Kubernetes) works in collaboration with the heterogeneous computing virtualization middleware in the cluster. The vCUDA driver library collects real-time data on the actual resource usage of physical graphics processors (GPUs) by each containerized task, including core metrics such as memory usage and computing unit usage time, and reports the data to the heterogeneous computing virtualization middleware in real time. The heterogeneous computing virtualization middleware presets a resource usage monitoring threshold (this threshold is bound to the virtual graphics processor (vGPU) resource limit allocated to each task), and continuously compares the actual usage data of each task with the threshold. When it detects that the resource usage of any containerized task exceeds its allocated limit, it immediately triggers a resource reclamation mechanism, which performs reclamation operations through the vCUDA driver library, such as suspending the task's new resource requests, releasing its excess memory space, adjusting the scheduling priority of its computing context, and forcibly pulling its resource usage back within the limit range, while ensuring that the resource space of other containerized tasks is not affected, thereby restoring and stabilizing the isolation state between tasks.

[0049] Through a closed-loop mechanism of real-time monitoring and dynamic recycling, the problem of a single containerized task over-consuming resources and disrupting the isolation state is effectively prevented. This further strengthens the resource boundary control when multiple tasks share a physical graphics processor, ensuring the compliance and stability of resource usage for each task. At the same time, it avoids the waste of overall computing power caused by abnormal resource consumption by a single task, and improves the reliability and efficiency of physical graphics processor resource sharing.

[0050] As a specific implementation of this disclosure, based on the basic scheme, the allocation and isolation of video memory space and computing context on the physical graphics processor according to the virtual graphics processor resource limit allocated to the containerized task includes: intercepting graphics processor resource operation instructions issued by the containerized task through a virtualization driver library loaded into the process space of the containerized task; mapping a private video memory address range belonging to the containerized task in the global video memory address space of the physical graphics processor according to the resource limit allocated to the containerized task; redirecting the intercepted video memory operation instructions to the private video memory address range for execution, and creating an independent computing context for the containerized task, thereby achieving dual isolation of video memory space and computing context.

[0051] Specifically, when a containerized task (i.e., a Pod, the smallest deployment unit in Kubernetes) starts, the virtualization driver library (i.e., the vCUDA driver library, libvgpu.so) is loaded into its process space. This driver library acts as an interception layer, listening in real time and intercepting all graphics processing unit (GPU) resource operation instructions issued by the containerized task, including CUDA APIs (Application Programming Interfaces) related to memory allocation, data read / write, and computation scheduling. The application programming interface (API) is called; subsequently, the virtualization driver library retrieves the virtual graphics processor (vGPU) resource quota (including parameters such as video memory capacity and computing power ratio) pre-allocated for the containerized task. Based on the global video memory address space layout of the physical graphics processor, a private video memory address range dedicated to this task is defined through address mapping technology. This range has a unique start address and end address and does not overlap with the private video memory address ranges of other containerized tasks. At the same time, the virtualization driver library creates an independent computing context for the containerized task, establishes a dedicated binding relationship between it and the streaming multiprocessor (SM) of the physical graphics processor, and then redirects all intercepted video memory operation instructions to the aforementioned private video memory address range for execution. The computing instructions are scheduled and executed in an independent computing context, thus forming a dual isolation mechanism between video memory space and computing context.

[0052] By intercepting instructions, mapping addresses, and creating independent computing contexts through the virtualization driver library, precise isolation between video memory space and computing resources is achieved, completely avoiding resource contention and operational interference between multiple tasks. At the same time, this implementation does not rely on a specific hardware architecture, but achieves dual isolation solely through software-level technology. This not only improves the stability and security of containerized task execution, but also ensures the compatibility of the technical solution with different models of physical graphics processors, laying a core foundation for the efficient sharing of the same physical graphics processor by multiple tasks.

[0053] It should be noted that the embodiments of this disclosure may include multiple steps. For ease of description, these steps are numbered, but these numbers are not a limitation on the execution time slots or execution order between the steps; these steps can be implemented in any order, and the embodiments of this disclosure do not limit this.

[0054] Corresponding to the aforementioned cluster computing resource isolation method, this disclosure also proposes a cluster computing resource isolation device. Since the device embodiments of this disclosure correspond to the aforementioned method embodiments, details not disclosed in the device embodiments can be referred to the aforementioned method embodiments, and will not be repeated here.

[0055] Figure 2This is a schematic diagram of the structure of a cluster computing resource isolation device provided in an embodiment of this disclosure, as shown below. Figure 2 As shown, it includes: The virtualization unit 21 is used to perform virtualization abstraction of physical graphics processor resources in a containerized cluster, and generate and report virtual graphics processor resource information. Scheduling unit 22 is used to schedule containerized tasks that request virtual graphics processor resources based on the virtual graphics processor resource information, and allocate the containerized tasks to physical graphics processor nodes with corresponding remaining resources. The allocation unit 23 is used to intercept the containerized task's access request to the graphics processor resources during the execution of the containerized task, and allocate and isolate video memory space and computing context on the physical graphics processor according to the virtual graphics processor resource limit allocated to the containerized task. The maintenance unit 24 is used to dynamically maintain the isolation status of the video memory space and computing context of each containerized task on the physical graphics processor, ensuring resource partitioning and isolation when multiple containerized tasks share the same physical graphics processor.

[0056] The cluster computing resource isolation device disclosed herein virtualizes and abstracts physical graphics processor resources in a containerized cluster and reports relevant information. Based on this information, it schedules containerized tasks to suitable physical graphics processor nodes, intercepts resource access requests during task execution, and allocates isolated video memory space and computing context according to the allocated virtual graphics processor resource quota. At the same time, it dynamically maintains the isolation status of multiple tasks. Therefore, it can solve the problems of inter-task interference, limited hardware compatibility, insufficient resource allocation flexibility, and low utilization caused by the lack of flexible virtualization schemes decoupled from hardware and imperfect resource scheduling and isolation mechanisms in the prior art. It achieves fine-grained segmentation and strong isolation of physical graphics processor resources in a containerized environment, supports multiple containerized tasks to share the same physical graphics processor, and improves the technical effects of resource utilization and task running stability.

[0057] Furthermore, in one possible implementation of this embodiment, the virtualization unit 21 is also used for: The video memory resources of the physical graphics processor are quantized into multiple independent virtual video memory resource units according to a preset granularity. Different models of physical graphics processor hardware are managed through a unified abstract interface, generating virtual graphics processor resource information with a unified description format.

[0058] Furthermore, in one possible implementation of this embodiment, the scheduling unit 22 is also used for: Receive resource request descriptions submitted by one or more containerized tasks, wherein each resource request description specifies the amount of virtual graphics processor resources requested by the corresponding task; Based on the resource request description and the virtual resource description information, the one or more containerized tasks are scheduled to one or more computing nodes that have sufficient corresponding remaining resources.

[0059] Furthermore, in one possible implementation of this embodiment, the allocation unit 23 is also used for: Intercept the graphics processor memory operation instructions issued by the containerized task; Based on the virtual memory limit set for the containerized task, a corresponding independent memory area is allocated from the physical graphics processor's memory space for the containerized task to use, and the memory operation instructions are redirected to the independent memory area.

[0060] Furthermore, in one possible implementation of this embodiment, the maintenance unit 24 is also used for: Monitor the actual usage of physical graphics processor resources by each containerized task; When any task is detected to be using more resources than its allocated limit, a resource reclamation operation is performed to restore the isolation state.

[0061] Furthermore, in one possible implementation of this embodiment, the allocation unit 23 is also used for: By loading the virtualization driver library into the process space of the containerized task, the graphics processor resource operation instructions issued by the containerized task are intercepted. Based on the resource limit allocated to the containerized task, a private memory address range belonging to the containerized task is mapped in the global memory address space of the physical graphics processor. The intercepted memory operation instructions are redirected to the private memory address range for execution, and an independent computing context is created for the containerized task, thereby achieving dual isolation between the memory space and the computing context.

[0062] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of this embodiment, and the principle is the same, so it is not limited in this embodiment.

[0063] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0064] Figure 3A schematic block diagram of an example electronic device 300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0065] like Figure 3 As shown, the electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 302 or a computer program loaded from storage unit 308 into RAM (Random Access Memory) 303. The RAM 303 may also store various programs and data required for the operation of the electronic device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An I / O (Input / Output) interface 305 is also connected to the bus 304.

[0066] Multiple components in electronic device 300 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of displays, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0067] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as cluster computing resource isolation methods. For example, in some embodiments, the cluster computing resource isolation method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, computing unit 301 may be configured to perform the aforementioned cluster computing resource isolation method by any other suitable means (e.g., by means of firmware).

[0068] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0069] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0070] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0071] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0072] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0073] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0074] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0075] The various numerical designations such as "first," "second," etc., used in this disclosure are merely for ease of description and are not intended to limit the scope of the embodiments of this disclosure, nor do they indicate a sequential order.

[0076] At least one of the features described in this disclosure can also be described as one or more, and multiple features can be two, three, four or more, and this disclosure does not impose any limitations. In the embodiments of this disclosure, for a technical feature, the technical features in that technical feature are distinguished by "first", "second", "third", "A", "B", "C" and "D", etc., and there is no sequential order or size order among the technical features described by "first", "second", "third", "A", "B", "C" and "D".

[0077] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0078] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for isolating cluster computing resources, characterized in that, include: In a containerized cluster, physical graphics processor resources are virtualized and abstracted to generate and report virtual graphics processor resource information. Based on the virtual graphics processor resource information, the containerized tasks that request virtual graphics processor resources are scheduled, and the containerized tasks are allocated to physical graphics processor nodes with corresponding remaining resources. During the execution of the containerized task, the access requests of the containerized task to the graphics processor resources are intercepted, and according to the virtual graphics processor resource limit allocated to the containerized task, video memory space and computing context are allocated and isolated on the physical graphics processor. The isolation status of each containerized task's memory space and computing context on the physical graphics processor is dynamically maintained to ensure resource partitioning and isolation when multiple containerized tasks share the same physical graphics processor.

2. The method according to claim 1, characterized in that, The virtualization abstraction of physical graphics processor resources includes: The video memory resources of the physical graphics processor are quantized into multiple independent virtual video memory resource units according to a preset granularity. Different models of physical graphics processor hardware are managed through a unified abstract interface, generating virtual graphics processor resource information with a unified description format.

3. The method according to claim 1, characterized in that, The scheduling of containerized tasks requesting virtual graphics processor resources based on the reported virtual graphics processor resource information includes: Receive resource request descriptions submitted by one or more containerized tasks, wherein each resource request description specifies the amount of virtual graphics processor resources requested by the corresponding task; Based on the resource request description and the virtual resource description information, the one or more containerized tasks are scheduled to one or more computing nodes that have sufficient corresponding remaining resources.

4. The method according to claim 1, characterized in that, The interception of the containerized task's access requests to graphics processor resources includes: Intercept the graphics processor memory operation instructions issued by the containerized task; Based on the virtual memory limit set for the containerized task, a corresponding independent memory area is allocated from the physical graphics processor's memory space for the containerized task to use, and the memory operation instructions are redirected to the independent memory area.

5. The method according to claim 1, characterized in that, The dynamic maintenance of the isolation status of each containerized task's resource space on the physical graphics processor includes: Monitor the actual usage of physical graphics processor resources by each containerized task; When any task is detected to be using more resources than its allocated limit, a resource reclamation operation is performed to restore the isolation state.

6. The method according to claim 1, characterized in that, The step of allocating and isolating video memory and computing context on the physical graphics processor according to the virtual graphics processor resource quota allocated for the containerized task includes: By loading the virtualization driver library into the process space of the containerized task, the graphics processor resource operation instructions issued by the containerized task are intercepted. Based on the resource limit allocated to the containerized task, a private memory address range belonging to the containerized task is mapped in the global memory address space of the physical graphics processor. The intercepted video memory operation instructions are redirected to the private video memory address range for execution, and an independent computing context is created for the containerized task, thereby achieving dual isolation between video memory space and computing context.

7. A cluster computing resource isolation device, characterized in that, include: The virtualization unit is used to virtualize and abstract physical graphics processor resources in a containerized cluster, and to generate and report virtual graphics processor resource information. The scheduling unit is used to schedule containerized tasks that request virtual graphics processor resources based on the virtual graphics processor resource information, and allocate the containerized tasks to physical graphics processor nodes with corresponding remaining resources. The allocation unit is used to intercept the containerized task's access request to the graphics processor resources during the execution of the containerized task, and allocate and isolate video memory space and computing context on the physical graphics processor according to the virtual graphics processor resource limit allocated to the containerized task. The maintenance unit is used to dynamically maintain the isolation status of the video memory space and computing context of each containerized task on the physical graphics processor, ensuring resource partitioning and isolation when multiple containerized tasks share the same physical graphics processor.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images