Heterogeneous accelerator card sharing and isolating method and system based on API hijacking

Abstract

The invention discloses a heterogeneous accelerator card sharing and isolating method and system based on API hijacking, and relates to the technical field of resource allocation, the method comprises the following steps: identifying and managing heterogeneous computing equipment mounted on a GPU node, and reporting resource information of the heterogeneous computing equipment; task workloads are distributed to appropriate GPU nodes through an extension scheduler; the GPU node mounts a heterogeneous device API interceptor for the task workload; intercepting API (Application Program Interface) calling of various heterogeneous computing devices; and judging whether the allocated workload exceeds a resource limiting quantity or a video memory limiting quantity, carrying out resource limitation on the exceeded heterogeneous computing equipment, and calling the native API of the heterogeneous computing equipment with sufficient computing resources to execute the task workload. By means of the technical scheme, isolation and sharing of fine grit are achieved, the resource limiting quantity and the video memory limiting quantity of all the devices can be scheduled and managed in a unified mode, and the resource utilization rate of the heterogeneous computing devices is increased.

Description

Technical Field

[0001] This invention relates to the field of resource allocation technology, and in particular to a method for sharing and isolating heterogeneous accelerator cards based on API hijacking, and a system for sharing and isolating heterogeneous accelerator cards based on API hijacking. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, the application of large-scale language models is becoming increasingly widespread. Against this backdrop, graphics processing units (GPUs) are playing an increasingly important role in the AI ​​field due to their significant performance in accelerating the training and inference processes of AI applications. Currently, mainstream internet technology companies widely adopt Kubernetes as the platform for running their internet services. Kubernetes' clustering capabilities provide powerful support for distributed training and inference tasks in AI applications.

[0003] Kubernetes supports workloads using third-party devices, including a device plugin from a specific brand, through its device plugin mechanism, allowing workloads to utilize that brand's GPUs. However, with the explosive growth of AI applications, the demand for GPU resources has also increased. Currently, many AI workloads have low GPU utilization, leading to resource waste. Furthermore, existing device plugins from that brand primarily support a single workload using a single GPU card, limiting the optimization and utilization of resources.

[0004] To address this issue, the industry has begun exploring GPU sharing technologies. These technologies allow multiple applications to share the same GPU card by limiting application resource usage on the GPU device, effectively improving GPU utilization. Features such as Virtual GPU (vGPU) and Multi-Instance GPU (MIG) proposed by a certain brand are crucial for improving resource utilization and workload isolation.

[0005] By introducing GPU sharing technology, we can not only optimize resource allocation for AI applications and improve the utilization efficiency of GPU resources, but also reduce operating costs and accelerate the training and inference processes of AI models. The practical application of these technologies is of great significance for promoting the further development of the AI ​​field.

[0006] With the development of artificial intelligence technology, in addition to GPUs from a certain brand, various heterogeneous accelerator cards are gradually being widely used, such as NPUs and intelligent processors from a certain brand. These heterogeneous accelerator devices face the following challenges in a Kubernetes environment: (1) The device discovery and resource reporting mechanisms are not unified, and each device requires an independent Device Plugin implementation.

[0007] (2) There is a lack of a unified scheduling strategy across devices, making it impossible to perform intelligent scheduling based on application characteristics and device features.

[0008] (3) The different resource isolation and sharing mechanisms increase the complexity of operation and maintenance.

[0009] (4) Inconsistent equipment monitoring and fault handling methods affect the reliability of the system. Summary of the Invention

[0010] To address the aforementioned issues, this invention provides a method and system for sharing and isolating heterogeneous accelerator cards based on API hijacking. By intercepting API calls from various heterogeneous computing devices through API hijacking technology, fine-grained isolation and sharing are achieved. Furthermore, unified scheduling and management are implemented based on the resource and memory limits of each device, thereby improving the resource utilization of heterogeneous computing devices.

[0011] To achieve the above objectives, this invention provides a method for sharing and isolating heterogeneous accelerator cards based on API hijacking, comprising: The Device Plugin identifies and manages heterogeneous computing devices mounted on GPU nodes within containers and reports the resource information of these heterogeneous computing devices to the Kubernetes cluster. Based on the task workload submitted by the user, the extended scheduler allocates the task workload to the appropriate GPU node according to the preset resource allocation strategy; The GPU node attaches a heterogeneous device API interceptor to the assigned task workload through the Device Plugin; The heterogeneous device API interceptor intercepts API calls from various heterogeneous computing devices. Based on the preset resource limits and video memory limits of the heterogeneous computing devices, determine whether the workload allocated to each of the heterogeneous computing devices exceeds the resource limits or the video memory limits; The heterogeneous device API interceptor limits the resources of heterogeneous computing devices that exceed the resource or memory limits, and calls the native API of heterogeneous computing devices with sufficient computing resources to execute the task workload.

[0012] In the above technical solution, preferably, the method of identifying and managing heterogeneous computing devices mounted on GPU nodes within the container through the Device Plugin, and reporting the resource information of the heterogeneous computing devices to the Kubernetes cluster, specifically includes: The Device Plugin automatically discovers and identifies heterogeneous computing devices mounted on the GPU nodes within the container; Obtain the computing resources of the heterogeneous computing device and report the resource information to the Kubernetes cluster; Monitor the status of the heterogeneous computing devices within the container using the Device Plugin, and identify and isolate faulty heterogeneous computing devices; The heterogeneous computing device is initialized using the Device Plugin, and its operating environment is configured.

[0013] In the above technical solution, preferably, the step of allocating the task workload submitted by the user to suitable GPU nodes through an extended scheduler based on a preset resource allocation strategy includes: Receive the task workload submitted by the user and determine the computing resources required for the task workload; Based on the device requirements declared by the application, the extended scheduler prioritizes allocating the task workload to GPU nodes with corresponding types of heterogeneous computing devices. Based on a preset device selection strategy, the optimal device is selected according to the load of the available heterogeneous computing devices, and the task workload is allocated to the optimal device through the extended scheduler. The extended scheduler configures corresponding resource allocation strategies for different types of heterogeneous computing devices.

[0014] In the above technical solution, preferably, based on the preset resource limits and memory limits of the heterogeneous computing devices, it is determined whether the workload allocated to each of the heterogeneous computing devices exceeds the resource limits or the memory limits. The specific method includes: Based on the computing resources of the heterogeneous computing devices, pre-configure the resource limits and video memory limits for each of the heterogeneous computing devices; Based on the task workload allocated to the heterogeneous computing device by the Device Plugin, determine whether the workload exceeds the resource limits and video memory limits of the heterogeneous computing device.

[0015] In the above technical solution, preferably, the heterogeneous computing devices whose computing resources exceed the resource limit or the video memory limit are restricted by the heterogeneous device API interceptor, and the task workload is executed by calling the native API of the heterogeneous computing device with sufficient computing resources. The specific method includes: If the computing resources of the workload exceed the resource limit of the heterogeneous computing device, then the heterogeneous computing device is controlled to enter a program hibernation state; If the computing resources of the workload exceed the video memory limit of the heterogeneous computing device, then the control will send an OOM error instruction indicating insufficient memory. If the computing resources or video memory of the heterogeneous computing device meet the workload requirements, then the native API of the heterogeneous computing device is invoked to execute the task workload.

[0016] This invention also proposes a heterogeneous accelerator card sharing and isolation system based on API hijacking, applying the heterogeneous accelerator card sharing and isolation method based on API hijacking disclosed in any of the above technical solutions, including: The device resource identification module is used to identify and manage heterogeneous computing devices mounted on GPU nodes within containers through the Device Plugin, and to report the resource information of the heterogeneous computing devices to the Kubernetes cluster. The task load allocation module is used to allocate the task workload submitted by the user to the appropriate GPU node through the extended scheduler according to the preset resource allocation strategy. The API interception mounting module is used to mount heterogeneous device API interceptors to the task workload allocated to the GPU node through the Device Plugin; The device API hijacking module is used to intercept API calls from various heterogeneous computing devices through the heterogeneous device API interceptor. The resource limit determination module is used to determine whether the workload allocated to each of the heterogeneous computing devices exceeds the resource limit or the video memory limit based on the preset resource limit and video memory limit of the heterogeneous computing devices. The resource sharing scheduling module is used to limit the resources of heterogeneous computing devices whose computing resources exceed the resource limit or the video memory limit through the heterogeneous device API interceptor, and to call the native API of heterogeneous computing devices with sufficient computing resources to execute the task workload.

[0017] In the above technical solution, preferably, the device resource identification module is specifically used for: The Device Plugin automatically discovers and identifies heterogeneous computing devices mounted on the GPU nodes within the container; Obtain the computing resources of the heterogeneous computing device and report the resource information to the Kubernetes cluster; Monitor the status of the heterogeneous computing devices within the container using the Device Plugin, and identify and isolate faulty heterogeneous computing devices; The heterogeneous computing device is initialized using the Device Plugin, and its operating environment is configured.

[0018] In the above technical solution, preferably, the task load allocation module is specifically used for: Receive the task workload submitted by the user and determine the computing resources required for the task workload; Based on the device requirements declared by the application, the extended scheduler prioritizes allocating the task workload to GPU nodes with corresponding types of heterogeneous computing devices. Based on a preset device selection strategy, the optimal device is selected according to the load of the available heterogeneous computing devices, and the task workload is allocated to the optimal device through the extended scheduler. The extended scheduler configures corresponding resource allocation strategies for different types of heterogeneous computing devices.

[0019] In the above technical solution, preferably, the resource limitation judgment module is specifically used for: Based on the computing resources of the heterogeneous computing devices, pre-configure the resource limits and video memory limits for each of the heterogeneous computing devices; Based on the task workload allocated to the heterogeneous computing device by the Device Plugin, determine whether the workload exceeds the resource limits and video memory limits of the heterogeneous computing device.

[0020] In the above technical solution, preferably, the resource sharing scheduling module is specifically used for: If the computing resources of the workload exceed the resource limit of the heterogeneous computing device, then the heterogeneous computing device is controlled to enter a program hibernation state; If the computing resources of the workload exceed the video memory limit of the heterogeneous computing device, then the control will send an OOM error instruction indicating insufficient memory. If the computing resources or video memory of the heterogeneous computing device meet the workload requirements, then the native API of the heterogeneous computing device is invoked to execute the task workload.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: by intercepting API calls of various heterogeneous computing devices through API hijacking technology, fine-grained isolation and sharing are achieved, and unified scheduling and management are carried out according to the resource limits and memory limits of each device, thereby improving the resource utilization of heterogeneous computing devices. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall architecture of a heterogeneous accelerator card sharing and isolation method based on API hijacking disclosed in an embodiment of the present invention; Figure 2This is a flowchart illustrating a heterogeneous accelerator card sharing and isolation method based on API hijacking, as disclosed in one embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] The present invention will now be described in further detail with reference to the accompanying drawings: like Figure 1 and Figure 2 As shown, a method for sharing and isolating heterogeneous accelerator cards based on API hijacking according to the present invention includes: The Device Plugin identifies and manages heterogeneous computing devices mounted on GPU nodes within containers and reports resource information of these heterogeneous computing devices to the Kubernetes cluster. Based on the task workload submitted by the user, the extended scheduler allocates the task workload to appropriate GPU nodes according to the preset resource allocation strategy. GPU nodes attach heterogeneous device API interceptors to their assigned task workloads via the Device Plugin; Intercept API calls from various heterogeneous computing devices using a heterogeneous device API interceptor; Based on the preset resource and memory limits of heterogeneous computing devices, determine whether the workload allocated to each heterogeneous computing device exceeds the resource or memory limits. By using a heterogeneous device API interceptor, resource limits are imposed on heterogeneous computing devices whose computing resources exceed resource or memory limits, and the native APIs of heterogeneous computing devices with sufficient computing resources are invoked to execute task workloads.

[0025] In this implementation, API calls from various heterogeneous computing devices are intercepted using API hijacking technology, achieving fine-grained isolation and sharing. Furthermore, unified scheduling and management are performed based on the resource and memory limits of each device, thereby improving the resource utilization of heterogeneous computing devices.

[0026] Specifically, the Kubernetes Device Plugin is responsible for reporting heterogeneous computing device resource information to the Kubernetes cluster and managing the mounting process of heterogeneous computing devices within containers. The implementation of this component ensures that containers can correctly identify and use the heterogeneous computing device resources allocated to them.

[0027] The Kubernetes extended scheduler is responsible for coordinating and managing the allocation of heterogeneous computing device resources. It intelligently schedules resources based on the needs of container applications and the available heterogeneous computing device resources in the cluster to achieve optimal allocation of heterogeneous computing device resources.

[0028] The heterogeneous device API interceptor is the core component of this invention, responsible for intercepting all API calls involving computing power and memory management. Through this mechanism, it achieves effective isolation and sharing of computing resources, ensuring the rational allocation and efficient utilization of resources.

[0029] In the above implementation, preferably, the Device Plugin identifies and manages the heterogeneous computing devices mounted on the GPU nodes within the container and reports the resource information of the heterogeneous computing devices to the Kubernetes cluster. The specific method includes: Automatically discover and identify heterogeneous computing devices mounted on GPU nodes within containers using the Device Plugin; Acquire computing resources from heterogeneous computing devices and report the resource information to the Kubernetes cluster; Monitor the status of heterogeneous computing devices within containers through the Device Plugin, and discover and isolate faulty heterogeneous computing devices; The Device Plugin is used to initialize heterogeneous computing devices and configure their operating environment.

[0030] Specifically, the Device Plugin extension supports a variety of heterogeneous computing devices and can realize device discovery and resource reporting, including automatically discovering heterogeneous computing devices on GPU nodes and reporting resource information of various devices.

[0031] The Device Plugin can also initialize heterogeneous computing devices, select the appropriate initialization process based on the device type, and configure the device's operating environment.

[0032] Device Plugin can also be used for health checks of heterogeneous computing devices, monitor the operating status of heterogeneous computing devices, and promptly detect and isolate faulty devices.

[0033] In the above embodiments, preferably, based on the task workload submitted by the user, the extended scheduler allocates the task workload to suitable GPU nodes according to a preset resource allocation strategy. The specific method includes: Receive the task workload submitted by the user and determine the computing resources required for the task workload; Based on the device requirements declared by the application, the extended scheduler prioritizes the allocation of task workloads to GPU nodes with the corresponding type of heterogeneous computing devices. Based on a preset device selection strategy, the optimal device is selected according to the load of available heterogeneous computing devices, and the task workload is distributed to the optimal device through an extended scheduler. The extended scheduler configures corresponding resource allocation strategies for different types of heterogeneous computing devices.

[0034] Specifically, the extended scheduler includes features such as device affinity scheduling, cross-device load balancing, and configurable device resource allocation strategies during the scheduling process of heterogeneous computing devices.

[0035] The extended scheduler can prioritize scheduling to nodes with corresponding device types based on the device requirements declared by the application. When multiple device types are available, the optimal device can be selected based on the device load. The extended scheduler also supports configuring different resource allocation strategies for different device types.

[0036] In the above embodiments, preferably, based on the preset resource limits and memory limits of the heterogeneous computing devices, it is determined whether the workload allocated to each heterogeneous computing device exceeds the resource limits or memory limits. The specific method includes: Based on the computing resources of heterogeneous computing devices, pre-configure the resource limits and video memory limits for each heterogeneous computing device; Based on the task workload allocated to the heterogeneous computing device by the Device Plugin, determine whether the workload exceeds the resource and video memory limits of the heterogeneous computing device.

[0037] In the above implementation, preferably, a heterogeneous computing device API interceptor is used to limit the resources of heterogeneous computing devices that exceed the resource or memory limits, and the native API of the heterogeneous computing device with sufficient computing resources is invoked to execute the task workload. The specific method includes: If the computing resources of the workload exceed the resource limit of the heterogeneous computing device, then control the heterogeneous computing device to enter a program hibernation state; If the computing resources of the workload exceed the video memory limit of the heterogeneous computing device, the control will issue an OutOfMemoryError (OOM) instruction indicating insufficient memory. If the computing resources or video memory of the heterogeneous computing device meet the workload requirements, then the native API of the heterogeneous computing device is called to execute the task workload.

[0038] In the above implementation, the heterogeneous device API interceptor is used to hijack the APIs of various types of heterogeneous computing devices to achieve resource isolation and sharing. The following uses the CUDA API as an example to illustrate the implementation principle. This method is not only applicable to the CUDA API, but the same mechanism can also be extended to the APIs of other acceleration devices, such as the CANN API of a certain brand of NPU and the CNRT API of a certain brand of intelligent processor.

[0039] The CUDA API interceptor enables multi-GPU sharing for containerized applications by hijacking the CUDA API. After a user submits an application workload, it first passes through an extended scheduler, which then selects a suitable GPU node for scheduling. Subsequently, the Device plugin on that GPU node attaches the CUDA API interceptor to this workload, along with initialization tasks such as GPU resource limit information and GPU device mounting.

[0040] The specific steps for the CUDA API interceptor to work are as follows: (1) Identify the CUDA APIs that need to be intercepted.

[0041] The core of a CUDA API interceptor lies in identifying which APIs need to be intercepted. The reason for partially intercepting CUDA APIs is that only APIs related to GPU computation and video memory need to be intercepted. Other CUDA APIs can be passed through, minimizing the impact of CUDA API interception on overall performance and thus improving the efficiency of the interceptor.

[0042] After identifying the APIs that need to be intercepted, the interception layer uses a resource management mechanism to ensure that the resources actually used by the CUDA application do not exceed the resource limit. The resource manager checks whether the current resource usage exceeds the resource limit each time a compute-related API is called. For compute resources, the resource manager performs the following pseudocode check: while True: free_core = limit_core - usage_core if free_core < 0: nanosleep(cycle) continue else: break nanosleep is a Linux system call instruction used to implement high-precision sleep functionality. It allows a program to pause execution for a specified duration. The resource manager checks whether the current computing resource usage exceeds the resource limit. If it does, it calls the nanosleep system call instruction, waits for the kernel function to finish executing and release the computing resources, and then exits the loop, thus achieving the purpose of limiting computing resources.

[0043] For video memory resources, each time the video memory allocation API is called, the resource manager will determine whether the currently used video memory exceeds the video memory limit. If it does, it will return an OOM (Out of Memory) error instruction, thereby achieving the purpose of resource limitation.

[0044] (2) Set LD_LIBRARY_PATH.

[0045] `LD_LIBRARY_PATH` is a Linux environment variable name used to specify paths other than the system default paths when searching for dynamic link libraries during program loading and execution. CUDA API interceptors are compiled into dynamic link libraries. Using the Linux `LD_LIBRARY_PATH` mechanism, the native CUDA APIs called by CUDA applications are replaced by CUDA API interceptors. Within the interceptor, it determines whether a called CUDA API needs to be intercepted. For APIs involving computation and GPU memory, the interceptor intercepts them and limits the GPU resources used by the application. If the current program's resource usage does not exceed the limit, the CUDA API interceptor calls the native CUDA API to access the GPU. If the API does not need to be intercepted, the CUDA API interceptor directly calls the native CUDA API to reduce the performance overhead caused by the interceptor.

[0046] (3) Limit resource usage based on the amount of resources input.

[0047] The CUDA API interceptor internally uses environment variables to import resource limit-related values. These resource limit-related environment variables include: VGPU_CORE_LIMIT (for compute resource limits) and VGPU_MEMORY_LIMIT (for memory).

[0048] VGPU_CORE_LIMIT is a value ranging from 0 to 100. A value of 100 means there is no restriction on the current CUDA process's use of the GPU card's computing resources. The CUDA API interceptor uses this value to set the maximum amount of GPU computing resources that a CUDA application can use.

[0049] The unit of VGPU_MEMORY_LIMIT is MB. The CUDA API interceptor sets the maximum amount of GPU memory that CUDA applications can use based on this value.

[0050] This invention also proposes a heterogeneous accelerator card sharing and isolation system based on API hijacking, applying the heterogeneous accelerator card sharing and isolation method based on API hijacking disclosed in any of the above embodiments, including: The device resource identification module is used to identify and manage heterogeneous computing devices mounted on GPU nodes within containers through the Device Plugin, and to report the resource information of heterogeneous computing devices to the Kubernetes cluster. The task load allocation module is used to allocate the task workload submitted by the user to appropriate GPU nodes through the extended scheduler according to the preset resource allocation strategy. The API interception mounting module is used to mount heterogeneous device API interceptors to the task workloads allocated to GPU nodes via the Device Plugin. The device API hijacking module is used to intercept API calls from various heterogeneous computing devices through a heterogeneous device API interceptor. The resource limit judgment module is used to determine whether the workload allocated to each heterogeneous computing device exceeds the resource limit or the video memory limit based on the preset resource limit and video memory limit of the heterogeneous computing device. The resource sharing scheduling module is used to limit the resources of heterogeneous computing devices that exceed the resource or memory limits through the heterogeneous device API interceptor, and to call the native API of heterogeneous computing devices with sufficient computing resources to execute task workloads.

[0051] In the above embodiments, preferably, the device resource identification module is specifically used for: Automatically discover and identify heterogeneous computing devices mounted on GPU nodes within containers using the Device Plugin; Acquire computing resources from heterogeneous computing devices and report the resource information to the Kubernetes cluster; Monitor the status of heterogeneous computing devices within containers through the Device Plugin, and discover and isolate faulty heterogeneous computing devices; The Device Plugin is used to initialize heterogeneous computing devices and configure their operating environment.

[0052] In the above embodiments, preferably, the task load allocation module is specifically used for: Receive the task workload submitted by the user and determine the computing resources required for the task workload; Based on the device requirements declared by the application, the extended scheduler prioritizes the allocation of task workloads to GPU nodes with the corresponding type of heterogeneous computing devices. Based on a preset device selection strategy, the optimal device is selected according to the load of available heterogeneous computing devices, and the task workload is distributed to the optimal device through an extended scheduler. The extended scheduler configures corresponding resource allocation strategies for different types of heterogeneous computing devices.

[0053] In the above embodiments, preferably, the resource limitation judgment module is specifically used for: Based on the computing resources of heterogeneous computing devices, pre-configure the resource limits and video memory limits for each heterogeneous computing device; Based on the task workload allocated to the heterogeneous computing device by the Device Plugin, determine whether the workload exceeds the resource and video memory limits of the heterogeneous computing device.

[0054] In the above embodiments, preferably, the resource sharing scheduling module is specifically used for: If the computing resources of the workload exceed the resource limit of the heterogeneous computing device, then control the heterogeneous computing device to enter a program hibernation state; If the computing resources of the workload exceed the video memory limit of the heterogeneous computing device, the control will issue an OutOfMemoryError (OOM) instruction indicating insufficient memory. If the computing resources or video memory of the heterogeneous computing device meet the workload requirements, then the native API of the heterogeneous computing device is called to execute the task workload.

[0055] According to the API hijacking-based heterogeneous accelerator card sharing and isolation system disclosed in the above embodiments, the functions to be implemented by each module are consistent with the steps of the API hijacking-based heterogeneous accelerator card sharing and isolation method disclosed in the above embodiments. In the implementation process, refer to the above embodiments for operation, and will not be repeated here.

[0056] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for sharing and isolating heterogeneous accelerator cards based on API hijacking, characterized in that, include: The Device Plugin identifies and manages heterogeneous computing devices mounted on GPU nodes within containers and reports the resource information of these heterogeneous computing devices to the Kubernetes cluster. Based on the task workload submitted by the user, the extended scheduler allocates the task workload to the appropriate GPU node according to the preset resource allocation strategy; The GPU node attaches a heterogeneous device API interceptor to the assigned task workload through the Device Plugin; The heterogeneous device API interceptor intercepts API calls from various heterogeneous computing devices. Based on the preset resource limits and video memory limits of the heterogeneous computing devices, determine whether the workload allocated to each of the heterogeneous computing devices exceeds the resource limits or the video memory limits; The heterogeneous device API interceptor limits the resources of heterogeneous computing devices that exceed the resource or memory limits, and calls the native API of heterogeneous computing devices with sufficient computing resources to execute the task workload.

2. The method for sharing and isolating heterogeneous accelerator cards based on API hijacking according to claim 1, characterized in that, The method for identifying and managing heterogeneous computing devices mounted on GPU nodes within containers through the Device Plugin, and reporting the resource information of the heterogeneous computing devices to the Kubernetes cluster, specifically includes: The Device Plugin automatically discovers and identifies heterogeneous computing devices mounted on the GPU nodes within the container; Obtain the computing resources of the heterogeneous computing device and report the resource information to the Kubernetes cluster; Monitor the status of the heterogeneous computing devices within the container using the Device Plugin, and identify and isolate faulty heterogeneous computing devices; The heterogeneous computing device is initialized using the Device Plugin, and its operating environment is configured.

3. The method for sharing and isolating heterogeneous accelerator cards based on API hijacking according to claim 1, characterized in that, The method of allocating the task workload submitted by the user to appropriate GPU nodes through an extended scheduler based on a preset resource allocation strategy includes: Receive the task workload submitted by the user and determine the computing resources required for the task workload; Based on the device requirements declared by the application, the extended scheduler prioritizes allocating the task workload to GPU nodes with corresponding types of heterogeneous computing devices. Based on a preset device selection strategy, the optimal device is selected according to the load of the available heterogeneous computing devices, and the task workload is allocated to the optimal device through the extended scheduler. The extended scheduler configures corresponding resource allocation strategies for different types of heterogeneous computing devices.

4. The method for sharing and isolating heterogeneous accelerator cards based on API hijacking according to claim 1, characterized in that, Based on the preset resource and video memory limits of the heterogeneous computing devices, it is determined whether the workload allocated to each of the heterogeneous computing devices exceeds the resource or video memory limits. The specific method includes: Based on the computing resources of the heterogeneous computing devices, pre-configure the resource limits and video memory limits for each of the heterogeneous computing devices; Based on the task workload allocated to the heterogeneous computing device by the Device Plugin, determine whether the workload exceeds the resource limits and video memory limits of the heterogeneous computing device.

5. The method for sharing and isolating heterogeneous accelerator cards based on API hijacking according to claim 4, characterized in that, The heterogeneous computing device API interceptor limits the computing resources of heterogeneous computing devices that exceed the resource or memory limits, and then invokes the native API of heterogeneous computing devices with sufficient computing resources to execute the task workload. The specific method includes: If the computing resources of the workload exceed the resource limit of the heterogeneous computing device, then the heterogeneous computing device is controlled to enter a program hibernation state; If the computing resources of the workload exceed the video memory limit of the heterogeneous computing device, then the control will send an OOM error instruction indicating insufficient memory. If the computing resources or video memory of the heterogeneous computing device meet the workload requirements, then the native API of the heterogeneous computing device is invoked to execute the task workload.

6. A heterogeneous accelerator card sharing and isolation system based on API hijacking, characterized in that, The method for sharing and isolating heterogeneous accelerator cards based on API hijacking as described in any one of claims 1 to 5 includes: The device resource identification module is used to identify and manage heterogeneous computing devices mounted on GPU nodes within containers through the Device Plugin, and to report the resource information of the heterogeneous computing devices to the Kubernetes cluster. The task load allocation module is used to allocate the task workload submitted by the user to the appropriate GPU node through the extended scheduler according to the preset resource allocation strategy. The API interception mounting module is used to mount heterogeneous device API interceptors to the task workload allocated to the GPU node through the Device Plugin; The device API hijacking module is used to intercept API calls from various heterogeneous computing devices through the heterogeneous device API interceptor. The resource limit determination module is used to determine whether the workload allocated to each of the heterogeneous computing devices exceeds the resource limit or the video memory limit based on the preset resource limit and video memory limit of the heterogeneous computing devices. The resource sharing scheduling module is used to limit the resources of heterogeneous computing devices whose computing resources exceed the resource limit or the video memory limit through the heterogeneous device API interceptor, and to call the native API of heterogeneous computing devices with sufficient computing resources to execute the task workload.

7. The heterogeneous accelerator card sharing and isolation system based on API hijacking according to claim 6, characterized in that, The device resource identification module is specifically used for: The Device Plugin automatically discovers and identifies heterogeneous computing devices mounted on the GPU nodes within the container; Obtain the computing resources of the heterogeneous computing device and report the resource information to the Kubernetes cluster; Monitor the status of the heterogeneous computing devices within the container using the Device Plugin, and identify and isolate faulty heterogeneous computing devices; The heterogeneous computing device is initialized using the Device Plugin, and its operating environment is configured.

8. The heterogeneous accelerator card sharing and isolation system based on API hijacking according to claim 6, characterized in that, The task load allocation module is specifically used for: Receive the task workload submitted by the user and determine the computing resources required for the task workload; Based on the device requirements declared by the application, the extended scheduler prioritizes allocating the task workload to GPU nodes with corresponding types of heterogeneous computing devices. Based on a preset device selection strategy, the optimal device is selected according to the load of the available heterogeneous computing devices, and the task workload is allocated to the optimal device through the extended scheduler. The extended scheduler configures corresponding resource allocation strategies for different types of heterogeneous computing devices.

9. The heterogeneous accelerator card sharing and isolation system based on API hijacking according to claim 6, characterized in that, The resource limitation judgment module is specifically used for: Based on the computing resources of the heterogeneous computing devices, pre-configure the resource limits and video memory limits for each of the heterogeneous computing devices; Based on the task workload allocated to the heterogeneous computing device by the Device Plugin, determine whether the workload exceeds the resource limits and video memory limits of the heterogeneous computing device.

10. The heterogeneous accelerator card sharing and isolation system based on API hijacking according to claim 9, characterized in that, The resource sharing and scheduling module is specifically used for: If the computing resources of the workload exceed the resource limit of the heterogeneous computing device, then the heterogeneous computing device is controlled to enter a program hibernation state; If the computing resources of the workload exceed the video memory limit of the heterogeneous computing device, then the control will send an OOM error instruction indicating insufficient memory. If the computing resources or video memory of the heterogeneous computing device meet the workload requirements, then the native API of the heterogeneous computing device is invoked to execute the task workload.

Images