Method and apparatus for processing gpu resources

Abstract

Embodiments of the present specification provide a GPU resource processing method and device. The method comprises: virtualizing each GPU into at least two different task types of virtual GPU; based on the GPU usage indicators of the task processes of each task type on the GPU of the work node, performing resource scheduling on the task scheduling units corresponding to each task type, so that the mixed deployment of tasks of multiple different task types, such as AI computing tasks and coding and decoding tasks, on the same physical GPU can be realized, and the GPU resource utilization is significantly improved.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a GPU resource processing method and electronic device. Background Technology

[0002] With the development of artificial intelligence technology, the computational tasks of AI models are increasingly demanding more and more computing power from GPUs (Graphics Processing Units), and how to efficiently utilize GPU hardware has become a focus of attention.

[0003] In related technical solutions, the GPU is spatially partitioned at the hardware level, dividing a single GPU into multiple independent, physically isolated instances, each of which can independently run different applications or workloads. However, this technical solution suffers from poor flexibility due to the use of a fixed GPU partitioning method.

[0004] The information in the background section is merely information known only to the inventor and does not imply that such information had entered the public domain before the filing date of this specification, nor does it imply that it can be considered prior art in this specification. Summary of the Invention

[0005] This specification provides a GPU resource processing method and electronic device that can flexibly partition the GPU according to the task and dynamically adjust the GPU distribution during operation.

[0006] In a first aspect, this specification provides a GPU resource processing method applied to a GPU hybrid deployment system, the GPU hybrid deployment system including at least one worker node, each worker node including at least one GPU, each GPU being virtualized into at least two virtual GPUs of different task types, the task type being the type of task executed by the task unit on the GPU, the method comprising:

[0007] Determine the GPU usage metrics of the task processes corresponding to each task type on the GPU of the worker node, wherein the task processes correspond to the task scheduling units of the worker node; and

[0008] Based on the GPU usage metrics, resource scheduling is performed on the task scheduling units corresponding to each task type on the GPU of the working node.

[0009] In some example embodiments, based on the above scheme, the at least two virtual GPUs of different task types include a computing GPU of computing type and a codec GPU of codec type, the task process includes a computing process or a codec process, and the task scheduling unit includes a computing scheduling unit or a codec scheduling unit.

[0010] In some example embodiments, based on the above scheme, the task type is the computation type, and the step of performing resource scheduling on the task scheduling unit corresponding to each task type on the GPU of the worker node based on the GPU usage metrics includes:

[0011] Based on the GPU usage metrics corresponding to the computing process, the running status of the computing process on the GPU of the worker node is determined, wherein the computing process corresponds to the computing scheduling unit;

[0012] Based on the running status of the computing process, the GPU resources used by the encoding / decoding scheduling unit on the GPU of the working node are adjusted accordingly.

[0013] In some example embodiments, based on the above scheme, the task type is an encoding / decoding type, and the step of scheduling resources for the task scheduling unit corresponding to each task type on the GPU of the working node based on the GPU usage metrics includes:

[0014] Based on the GPU usage metrics corresponding to the encoding / decoding process, the GPU resources used by the encoding / decoding process are determined, wherein the encoding / decoding process corresponds to the encoding / decoding scheduling unit;

[0015] Based on the used GPU resources and the preset encoding / decoding resource threshold, the encoding / decoding scheduling unit corresponding to the encoding / decoding process performs resource control to prevent the used GPU resources from exceeding the preset encoding / decoding resource threshold.

[0016] In some example embodiments, based on the above scheme, the method further includes:

[0017] The interception library intercepts the GPU call requests of the task process to the GPU of the worker node. The interception library is an application programming interface (API) library that is pre-injected into the task scheduling unit corresponding to the worker node.

[0018] In some example embodiments, based on the above scheme, the task type includes a computation type and an encoding / decoding type, the task process includes a computation process, and the method further includes:

[0019] The interception library prevents the computing process from calling the APIs related to the codec type.

[0020] In some example embodiments, based on the above scheme, the step of performing resource scheduling on the task scheduling unit corresponding to each task type on the GPU of the worker node based on the GPU usage metrics includes:

[0021] The system adjusts the GPU call requests intercepted by the interception library based on the GPU usage metrics.

[0022] In some example embodiments, based on the above scheme, the task process further includes an encoding / decoding process, and the step of adjusting the GPU call requests intercepted by the interception library based on the GPU usage metrics includes:

[0023] In the interception library, a counter is initialized for each thread in the encoding / decoding process. The counter is used to count the number of times the thread calls the kernel function.

[0024] If the counter reaches a predetermined threshold, the thread is prohibited from calling the computation kernel function.

[0025] In some example embodiments, based on the above scheme, the method further includes:

[0026] When the thread calls the API related to the task of the codec type, the counter is cleared.

[0027] In some example embodiments, based on the above scheme, determining the GPU usage metrics of the task processes corresponding to each task type on the GPU of the working node includes:

[0028] Collect the metric parameters corresponding to the GPU call requests intercepted by the interception library;

[0029] Based on the metric parameters corresponding to the GPU request, determine the GPU usage metrics of the task processes corresponding to each task type on the GPU of the worker node.

[0030] In some example embodiments, based on the above scheme, the task process includes a computing process or an encoding / decoding process, the task scheduling unit includes a computing scheduling unit or an encoding / decoding scheduling unit, and the method further includes:

[0031] Determine the mixed-process operation metrics on the GPU of each of the aforementioned working nodes, wherein the mixed-process operation metrics include the operation metrics of the computing process and the operation metrics of the encoding / decoding process;

[0032] If the GPU resources occupied by the encoding / decoding process of the working node are continuously less than a preset resource threshold, the encoding / decoding scheduling unit on the working node will be evicted, and the working node will be marked as a non-compete part.

[0033] If the computing process of the working node occupies more video memory than a preset video memory threshold, the encoding / decoding scheduling unit on the working node is evicted, and the working node is marked as a non-compete part.

[0034] In some example embodiments, based on the above scheme, the method further includes:

[0035] If the GPU resources occupied by the encoding / decoding process are not continuously less than the preset resource threshold and the video memory occupied by the computing process is not higher than the preset video memory threshold, then the label of the working node is determined.

[0036] If the label indicates that the working node is a non-mixable part, then the working node is marked as a mixable part.

[0037] Secondly, this specification also provides an electronic device, comprising: at least one storage medium storing at least one instruction set for GPU resource processing; and at least one processor communicatively connected to the at least one storage medium, wherein, when the electronic device is running, the at least one processor reads the at least one instruction set and executes the GPU resource processing method described in the first aspect of this specification according to the instructions of the at least one instruction set.

[0038] As can be seen from the above technical solutions, the GPU resource processing method and device provided in the embodiments of this specification, on the one hand, each GPU is virtualized into at least two virtual GPUs with different task types. The task type is the type of task executed by the task unit on the GPU. Since the GPU itself contains multiple types of task units, such as computing units and encoding / decoding units, it can realize the mixed deployment of multiple different task types, such as AI computing tasks and encoding / decoding tasks, on the same physical GPU. Different task types can be executed in parallel on different task units of the same GPU at the same time, maximizing the utilization of GPU resources and avoiding the problem of idle resources on the GPU, such as encoding / decoding resources. On the other hand, based on the GPU usage indicators of the task processes corresponding to each task type on the GPU of the working node, resource scheduling is performed on the task scheduling units corresponding to each task type. This not only allows for flexible and dynamic adjustment of GPU resources corresponding to various task types according to the task running status, but also ensures the service quality of different task types through software, without requiring specific hardware support. Furthermore, since the GPU resources corresponding to various task types can be flexibly and dynamically adjusted according to the task running status, multiple resources on the GPU, such as computing resources and encoding / decoding resources, can be fully utilized.

[0039] Other functionalities of the GPU resource processing methods and apparatus provided in this specification will be partially listed in the following description. The figures and examples described below will be readily apparent to those skilled in the art. The inventive aspects of the GPU resource processing methods and apparatus provided in this specification can be fully understood through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A schematic diagram of the implementation environment of a GPU resource processing method provided in an embodiment of this specification is shown;

[0042] Figure 2 A hardware structure diagram of an electronic device 200 provided according to an embodiment of this specification is shown;

[0043] Figure 3 A flowchart of a GPU resource processing method provided according to some embodiments of this specification is shown;

[0044] Figure 4 A schematic diagram of the node GPU resource scheduling process provided according to some embodiments of this specification is shown;

[0045] Figure 5 A flowchart illustrating slow node detection according to some embodiments of this specification is shown; and

[0046] Figure 6 A flowchart illustrating a GPU resource processing method provided according to further embodiments of this specification is shown. Detailed Implementation

[0047] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0048] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0049] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0050] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0051] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0052] GPU: An acceleration hardware that can be used for computational acceleration, video encoding / decoding (the focus of this article), and graphics display (not the focus of this article). GPUs are characterized by high-speed parallel computing, making them suitable for tasks that can be processed in parallel, such as typical deep learning tasks. Their performance is often 10+X times higher than that of CPUs, and they can also be used for video image encoding / decoding computations.

[0053] Video memory: High-speed memory medium on GPU cards, with high bandwidth (up to 700GB+ / sec) but small capacity (typically 16 / 32GB), its capacity directly limits some deep learning tasks with large models and large samples.

[0054] SLA (Service-Level Agreement): refers to the level of service that a supplier promises to provide to a customer. In the context of GPUs, it generally refers to the user's requirements for resource quality.

[0055] vGPU (virtual GPU): Virtualization technology virtualizes one or more physical GPUs into multiple smaller GPUs, allowing multiple tasks to share the same physical GPU.

[0056] Hybrid deployment: Hybrid deployment improves hardware utilization by deploying multiple scheduling units, or Pods, on the same physical GPU through vGPU. The applications within these scheduling units may be AI computing or video encoding / decoding.

[0057] Pod: The smallest scheduling unit in the scheduling system, corresponding to an application instance or application process, which can consist of a single container or multiple containers.

[0058] Codec tasks: Encoding and decoding tasks, such as encoding and decoding video using the GPU's hardware codec.

[0059] Compute kernel functions: Asynchronous tasks that run on the GPU. Users can achieve heterogeneous computing by submitting compute kernel functions to the GPU hardware.

[0060] Large-scale GPU clusters are typically used for intensive AI computing tasks, such as model inference or training. However, in addition to computing units, GPU hardware also contains encoding and decoding units. When a GPU cluster is used for AI computing tasks, the GPU's encoding units become idle, resulting in idle GPU encoding and decoding resources.

[0061] Based on the above, embodiments of this specification provide a GPU resource processing method and an electronic device. On one hand, each GPU is virtualized into at least two virtual GPUs with different task types. The task type is the type of task executed by the task unit on the GPU. Since the GPU itself contains multiple types of task units, such as computing units and encoding / decoding units, it can achieve the mixed deployment of multiple different task types, such as AI computing tasks and encoding / decoding tasks, on the same physical GPU. Different task types can be executed in parallel on different task units of the same GPU simultaneously, maximizing the utilization of the GPU's own resources and avoiding the problem of idle resources on the GPU, such as encoding / decoding resources. On the other hand, based on the GPU usage indicators of the task processes corresponding to each task type on the GPU of the working node, resource scheduling is performed on the task scheduling units corresponding to each task type. This not only allows for flexible and dynamic adjustment of GPU resources corresponding to various task types according to the task running status, but also ensures the service quality of different task types through software, without requiring specific hardware support. Furthermore, since the GPU resources corresponding to various task types can be flexibly and dynamically adjusted according to the task running status, the GPU resources of multiple task types, such as computing resources and encoding / decoding resources, can be fully utilized.

[0062] The technical solutions of the embodiments of this specification will now be described in detail with reference to the accompanying drawings.

[0063] Figure 1 A schematic diagram of the implementation environment of a GPU resource processing method provided in an embodiment of this specification is shown.

[0064] See Figure 1 As shown, this implementation environment is a platform that provides automated deployment, scaling, high availability, and application container execution across host clusters, such as the Kubernetes platform. This implementation environment includes a master node 110 and multiple worker nodes 120. The master node 110 manages workloads and directs communication across the entire cluster, while the worker nodes 120 perform specific tasks, such as compute and encoding / decoding tasks. The basic scheduling unit in Kubernetes is called a Pod, which is a higher-level abstraction above containers. A Pod contains one or more containers that reside on the same host and share network, storage, and other resources.

[0065] Taking the Kubernetes platform as an example, the master node includes an API Server, a Scheduler, a Controller Manager, and a database. The API Server is the hub of Kubernetes; both external and internal interfaces must send requests through the API Server. The API Server processes and validates requests and updates the state of API objects in the database, allowing clients to configure workloads and containers across compute nodes. The Scheduler is a pluggable component that selects which nodes unscheduled Pods (task scheduling units) should run on based on resource availability. The Scheduler tracks resource utilization on each node to ensure that workloads do not exceed available resources. The Controller Manager communicates with the API Server to create, update, and delete resources managed by the Controller Manager, such as Pods and server endpoints.

[0066] A worker node, or Node, consists of a node daemon (Kubelet), a GPU device plugin, a local scheduler (Local Scheduler), and at least one GPU. The Kubelet manages the running status of each node, starting, stopping, and maintaining application containers according to the control panel's instructions, and categorizing these containers into specific scheduling units (Pods). The Kubelet detects the status of scheduling unit Pods; if a Pod is not in the required state, it will be redeployed to the same node.

[0067] Kubernetes provides a unified interface and usage method through the GPU Device Plugin programming framework. By deploying GPU Device Plugins provided by these hardware vendors, Kubelet can become aware of and utilize these specific GPU hardware resources. Each GPU Device Plugin is responsible for the following tasks: (1) discovering and defining GPU device resources; (2) making these GPU devices available to containers and securely shared; and (3) checking the operating status of GPU devices. Each GPU Device Plugin provides a Remote Procedure Call (gRPC) service and provides the following two interfaces to Kubelet for invocation: (1) ListAndWatch interface: discovers devices, queries device attributes, and issues notifications when the status changes (such as becoming unavailable); (2) Allocate interface: when Kubelet creates a user container, it executes this function to perform a series of operations such as device initialization and environment variable setting. After the GPU Device Plugin is deployed, Kubernetes will support each GPU as a schedulable resource type. Users can request GPU resources just like they would request CPU or memory resources.

[0068] The GPU local scheduler is used to schedule the GPU resources of at least one GPU on the worker node. After a scheduling unit Pod is scheduled to a worker node, scheduling unit Pods of different task types that are co-located on the same physical card will be controlled and coordinated by the GPU local scheduler.

[0069] After introducing the implementation environment of the embodiments of this specification, the application scenarios of the embodiments of this specification will be described below in conjunction with the above implementation environment. In the following description, the master node is also the master node 110 in the above implementation environment, and the worker node is also the worker node 120 in the above implementation environment. The technical solutions provided by the embodiments of this specification can be applied in scenarios with mixed GPU resource deployment, such as cloud computing or AI training and inference scenarios.

[0070] Taking the technical solution provided in the embodiments of this specification as an example in a cloud computing scenario, the GPU hybrid deployment system includes at least one worker node, each worker node includes at least one GPU, the task type is the type of task executed by the task unit on the GPU, the task type includes computing type and encoding / decoding type, each GPU is virtualized as a computing GPU of computing type and an encoding / decoding GPU of encoding / decoding type, worker node 100 determines the GPU usage index of the task process corresponding to each task type on the worker node's GPU, the task process corresponds to the task scheduling unit of the worker node, i.e., Pod; and based on the GPU usage index of the task process corresponding to each task type, performs resource scheduling on the task scheduling unit corresponding to each task type on the worker node's GPU.

[0071] It should be noted that the above description is based on the application of the technical solutions provided in the embodiments of this specification in a cloud computing scenario. The technical solutions provided in the embodiments of this specification can also be applied to other suitable GPU-mixed deployment scenarios, such as virtual graphics workstations or virtual desktops. The implementation process is the same as the above description and belongs to the same inventive concept, so it will not be repeated here.

[0072] It should be noted that the steps in the GPU resource processing method in the example embodiments of this specification may be partially executed by the worker nodes, partially executed by the master node, or all executed by the master node or all worker nodes. This specification does not impose any special limitations on this.

[0073] based on Figure 1 The implementation environment shown below will be combined with... Figures 2-6 This specification provides a detailed description of the GPU resource processing methods and electronic devices provided in the embodiments. It should be noted that the above-described implementation environments are shown only to facilitate understanding of the spirit and principles of this specification, and the embodiments of this specification are not limited in any way. Rather, the embodiments of this specification can be applied to any applicable scenario.

[0074] Figure 2 This is a schematic diagram of an electronic device 200 provided according to some embodiments of this specification. The electronic device 200 can execute the GPU resource processing method described in this specification. The GPU resource processing method is described in other parts of this specification. The electronic device 200 can be a general-purpose computer or a special-purpose computer. For example, the electronic device 200 can be a server, a personal computer, a portable computer (such as a laptop computer, tablet computer, etc.), or other electronic devices with computing capabilities. Of course, the electronic device can be... Figure 1 The worker node 120 and / or master node 110 can also be terminal devices used by multiple developers to develop programs on the integrated development platform.

[0075] The electronic device described in this specification may include one or more of the following components: processor 210, memory 220, input device 230, output device 240, and bus 250. The processor 210, memory 220, input device 230, and output device 240 may be connected to each other via bus 250.

[0076] Processor 210 may include one or more processing cores. Processor 210 connects to various parts within the electronic device using various interfaces and lines, and executes the GPU resource processing methods described in this specification by running or executing instructions, programs, code sets, or instruction sets stored in memory 220, and by calling data stored in memory 220. Optionally, processor 210 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 210 may integrate one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 210 and may be implemented separately using a communication chip.

[0077] The memory 220 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 220 may include a non-transitory computer-readable storage medium. The memory 220 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 220 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (e.g., touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described below, etc., where the operating system may be a Windows system, a Linux system, or another system.

[0078] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to establish data communication between the third-party applications and the operating system. This would allow the operating system to obtain the current scenario information of the third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0079] The input device 230 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 240 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In one example, the input device 230 and the output device 240 can be combined, and both the input device 230 and the output device 240 can be a touch display screen.

[0080] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, Wireless Fidelity (WiFi) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0081] Figure 3 A flowchart of a GPU resource processing method according to an embodiment of this specification is shown. As previously described, the electronic device 200 can execute the GPU resource processing method of the embodiment of this specification. Specifically, the processor 210 can read the instruction set stored in its local storage medium and then execute the GPU resource processing method of the embodiment of this specification according to the instructions of the instruction set. Hereinafter, steps S310 to S320 of the GPU resource processing method will be described in detail with reference to the accompanying drawings.

[0082] Reference Figure 3 As shown, in step S310, the GPU usage metrics of the task processes corresponding to each task type on the GPU of the worker node are determined.

[0083] In an example embodiment, the GPU hybrid deployment system includes at least one worker node. A worker node refers to a physical server or virtual machine with GPU processing capabilities used to run GPU workloads. For example, in the case where the GPU hybrid deployment system includes one worker node, the worker node can be a server device or a virtual machine. The GPU hybrid deployment system can be a Kubernetes system. Each worker node includes at least one GPU, and each GPU is virtualized into at least two virtual GPUs with different task types. The task type refers to the type of task executed by a task unit on the GPU. A task unit on the GPU is a unit that executes a predetermined task of the GPU, including: a computation unit, an encoding / decoding unit, a rendering unit, etc. Specifically, the computation unit executes computation-type tasks of the GPU; the encoding / decoding unit executes encoding / decoding-type tasks of the GPU; and the rendering unit executes graphics rendering-type tasks of the GPU. Each task unit can access data through video memory.

[0084] In the example embodiment, at least two different task types include a computation type and an encoding / decoding type. The virtual GPUs for at least two different task types include a computation GPU for computation and an encoding / decoding GPU for encoding / decoding. Task processes include either computation processes or encoding / decoding processes, and task scheduling units include either computation scheduling units or encoding / decoding scheduling units. Different task types have different requirements for the GPU (e.g., video memory, computing power, latency), requiring resource allocation based on their characteristics. By virtualizing the GPU into at least two GPUs for different task types, it is possible to achieve mixed deployment of different task types, such as AI computation tasks and encoding / decoding tasks, on the same physical GPU.

[0085] Furthermore, the task process corresponds to the task scheduling unit of the worker node. The task scheduling unit represents the basic scheduling unit corresponding to the virtual GPU, namely the Pod. One task Pod corresponds to one task process or task instance. For example, one compute Pod corresponds to one compute process, and one codec Pod corresponds to one codec process.

[0086] It should be noted that although the example described uses computing or encoding / decoding as the task type, those skilled in the art should understand that the task type can also be other appropriate types, such as graphics rendering or audio processing, which are also within the scope of the embodiments in this specification.

[0087] GPU usage metrics represent the quality of service requirements of GPU resources for users. These metrics specify user requirements regarding GPU performance, availability, resource allocation, and stability. GPU usage metrics can be Service Level Agreements (SLAs). The core objective of setting GPU usage metrics is to ensure that user service quality needs for GPU resources (such as computing power, memory allocation, and task response speed) are effectively met in physical or virtualized environments through measurable indicators, and to trigger remedial mechanisms when these requirements are not met. GPU usage metrics can include one or more of the following: request task type, request response time, hardware resources consumed by the request, and request execution duration.

[0088] In the example embodiment, the electronic device 200 collects GPU usage data of task processes corresponding to various task types on the GPU of the worker node, and determines GPU usage metrics of task processes corresponding to various task types on the GPU of the worker node based on the GPU usage data. Taking a task process as a computing process as an example, the electronic device 200 collects the time to receive the request, the time to start processing the request, and the time to return the processing result for the GPU request corresponding to the computing process on the GPU of the worker node. Based on the time to receive the request and the time to start processing the request, the request response time of the computing process is determined, and based on the time to start processing the request and the time to return the processing result, GPU usage metrics such as the request execution time corresponding to the computing process are determined.

[0089] It should be noted that although the above-mentioned GPU usage metrics have been used as examples in the description, those skilled in the art should understand that GPU usage metrics can also be other appropriate metrics, such as resource utilization metrics and throughput metrics, which are also within the scope of the embodiments in this specification.

[0090] In step S320, resource scheduling is performed on the task scheduling units corresponding to each task type on the GPU of the worker node based on GPU usage metrics.

[0091] In the example embodiment, GPU usage metrics are quantitative indicators that measure GPU resource utilization and provide a basis for resource scheduling. A Task Scheduling Unit (TSU) is the smallest scheduling unit (Pod) in a GPU-deployed system such as Kubernetes, and a Task Pod corresponds to one or more containerized task processes. The electronic device 200, based on GPU usage metrics and corresponding resource scheduling strategies, performs resource scheduling on the Task Scheduling Units corresponding to each task type on at least one GPU of the worker node, coordinating computing power and memory resources among multiple task types to ensure that GPU utilization meets quality requirements.

[0092] For example, resource scheduling strategies may include quota limiting strategies. When electronic device 200 determines, based on GPU usage metrics, that a task process's GPU resource consumption is about to exceed a predetermined resource threshold, it restricts the task process from further consuming GPU resources. For instance, a pre-set memory threshold for a computing task can be configured. When electronic device 200 detects that the memory consumption of a computing task exceeds the memory threshold, it restricts memory allocation for the computing process through the GPU local scheduler.

[0093] In addition, resource scheduling strategies may include priority strategies. If the electronic device 200 determines that a high-priority task process has a longer request duration based on GPU usage metrics, then the high-priority task will be allocated resources first.

[0094] The following section provides a detailed explanation of resource scheduling in a GPU-distributed system using two examples.

[0095] Example 1:

[0096] The task type is computation. The electronic device 200 of the worker node determines the running status of the computation process on the GPU of the worker node through the GPU local scheduler based on the GPU usage index corresponding to the computation process. The computation process corresponds to the computation scheduling unit. Based on the running status of the computation process, the GPU resources used by the encoding and decoding scheduling unit on the GPU of the worker node are adjusted accordingly.

[0097] For example, electronic device 200 collects GPU usage metrics, or SLA metrics, corresponding to the computing process through an interception library, such as request response time and request execution duration. Based on the collected GPU metrics, the local GPU scheduler determines the running status of the computing process on the worker node's GPU. For example, if the request response time exceeds a preset duration, it is determined that the computing process on the worker node's GPU is being interfered with by other tasks, and the degree of interference is determined; the longer the request response time, the greater the degree of interference. Based on the running status of the computing process, the GPU resources used by the encoding / decoding scheduling unit on the worker node's GPU are adjusted accordingly. For example, based on the degree of interference to the computing process, the computing time slices used by the encoding / decoding Pod on the GPU are limited, or the encoding / decoding Pod is evicted.

[0098] Furthermore, a pre-defined correspondence between interference level and request response duration is established. The electronic device 200 determines the interference level of the computing process based on the collected request response duration. If the interference level is medium, the encoding / decoding Pod on the GPU continues to use new computing time slices; if the interference level is high, the encoding / decoding Pod on the GPU is evicted.

[0099] According to the technical solution in the above example embodiment, dynamically adjusting the encoding and decoding scheduling unit based on the actual running status of the computing process can prioritize the service quality of computing tasks and help improve the utilization of GPU resources.

[0100] Example 2:

[0101] The task type is encoding / decoding. The electronic devices of the working nodes 200 determine the GPU resources used by the encoding / decoding process based on the GPU usage indicators corresponding to the encoding / decoding process through the GPU local scheduler. The encoding / decoding process corresponds to the encoding / decoding scheduling unit. Based on the used GPU resources and the preset encoding / decoding resource threshold, the encoding / decoding scheduling unit corresponding to the encoding / decoding process is controlled to prevent the used GPU resources from exceeding the preset encoding / decoding resource threshold.

[0102] Because the GPU resource usage requirements of the encoding / decoding scheduling unit (i.e., the encoding / decoding Pod) are relatively low, the encoding / decoding resources that the encoding / decoding Pod can use are pre-limited. For example, electronic device 200 collects GPU resource requests corresponding to the encoding / decoding process through an interception library, determines the GPU resources already used by the encoding / decoding process through the GPU local scheduler, and combines the used GPU resources with a preset encoding / decoding resource threshold to determine whether the sum of the GPU resources corresponding to the GPU resource request and the used GPU resources will exceed the preset encoding / decoding resource threshold. If it will exceed the preset encoding / decoding resource threshold, the GPU resource request is rejected; if it will not exceed the preset encoding / decoding resource threshold, the GPU resource request is allowed.

[0103] According to the technical solution in the above example embodiment, by setting a preset encoding and decoding resource threshold to constrain the GPU memory usage of the encoding and decoding Pod, it is possible to avoid the encoding and decoding process occupying too much GPU resources during runtime, provide GPU memory preemption capability for high-priority tasks (such as AI inference), and realize dynamic resource reallocation.

[0104] according to Figure 3The technical solution in the example embodiment has several advantages. First, each GPU is virtualized into at least two virtual GPUs of different task types, enabling the mixed deployment of various task types, such as AI computing tasks and encoding / decoding tasks, on the same physical GPU, significantly improving GPU resource utilization. Second, based on the GPU usage metrics of the task processes corresponding to each task type on the GPU of the working node, resource scheduling is performed on the task scheduling units corresponding to each task type. This not only allows for flexible and dynamic adjustment of GPU resources corresponding to various task types according to the task running status, but also ensures the service quality of different task types through software, without requiring specific hardware support. Third, because the GPU resources corresponding to various task types can be flexibly and dynamically adjusted according to the task running status, various resources on the GPU, such as computing resources and encoding / decoding resources, can be fully utilized. Fourth, different types of tasks are unaware of each other's existence, achieving transparency and seamlessness for the user.

[0105] Furthermore, in the example embodiment, the electronic device 200 of the worker node intercepts GPU call requests from task processes targeting the worker node's GPU using an interception library. This interception library is an application programming interface (API) library pre-injected into the task scheduling unit corresponding to the worker node. For example, the interception library libsched.so may include a computing power control API and a memory allocation API. By intercepting GPU call requests corresponding to the computing power allocation API, computing power control of GPU resources can be performed; by intercepting GPU call requests corresponding to the memory allocation API, memory allocation of the GPU can be controlled.

[0106] According to the technical solution in the above example embodiment, by setting up an interception library, a layer of control can be added inside the container of the scheduling unit. By intercepting the underlying system calls of the task process, the resource requests corresponding to the task process can be dynamically adjusted to achieve resource allocation adjustment or resource isolation.

[0107] In some example embodiments, the task type includes both computation type and encoding / decoding type, and the task process includes a computation process. The electronic device 200 uses an interception library to prevent the computation process from calling APIs related to the encoding / decoding type task. For example, the interception library libsched.so prevents the computation process from calling APIs related to the encoding / decoding task, and returns a permission error whenever a computation process is detected calling APIs related to the encoding / decoding task.

[0108] According to the technical solution in the above example embodiment, by blocking the call of encoding and decoding related APIs by the interception library, the misuse of functions and resource conflicts are prevented, and the security, stability and manageability of the system are also improved.

[0109] In some example embodiments, the electronic device 200 collects the metric parameters corresponding to GPU call requests intercepted by the interception library; based on the metric parameters corresponding to the GPU requests, it determines the GPU usage metrics of the task processes corresponding to each task type on the GPU of the worker node. For example, the electronic device 200 collects the GPU metric parameters corresponding to the computing process through the interception library, such as the time of receiving the request, the time of starting to process the request, and the time of returning the processing result for the corresponding GPU request; based on the metric parameters corresponding to the GPU requests, it determines the GPU usage metrics of the task processes corresponding to each task type on the GPU of the worker node. For example, it determines the request response time of the computing process based on the time of receiving the request and the time of starting to process the request, and determines the request execution time of the computing process based on the time of starting to process the request and the time of returning the processing result, etc., which are GPU usage metrics.

[0110] According to the technical solution in the above example embodiment, the GPU usage index is determined by the index parameters corresponding to the GPU call request intercepted by the interception library. This can accurately identify the actual resource consumption of each task process and provide a solid data foundation for GPU resource scheduling and performance optimization.

[0111] Furthermore, in the example embodiment, the electronic device 200 adjusts the GPU call requests intercepted by the interception library based on the GPU usage metrics of the task processes corresponding to each task type. For example, the electronic device 200 runs a feedback adjustment strategy based on the collected GPU usage metrics of the task processes corresponding to each task type, such as threshold comparison and priority comparison, to dynamically adjust the control parameters in the interception library or determine whether to allow or prohibit GPU call requests. Dynamically adjusting the control parameters in the interception library may include: adjusting the computation kernel function call threshold: if the GPU utilization of the AI ​​inference task is low, the upper limit of the computation kernel function call count for the encoding and decoding task can be temporarily increased; dynamically modifying the video memory quota: if it is detected that the actual video memory usage of a certain encoding and decoding task is consistently lower than the upper limit, its quota can be appropriately reduced to release resources for other tasks.

[0112] According to the technical solution in the above example embodiment, by dynamically sensing the GPU usage indicators of processes of different task types, feedback adjustment is performed on GPU call requests that have been intercepted by the interception library (such as libsched.so), thereby achieving fine-grained and adaptive resource control.

[0113] To limit the computational resources used by the encoding / decoding process and prevent it from consuming excessive GPU resources, in an example embodiment, the electronic device 200 initializes a counter for each thread in the encoding / decoding process within an interception library. This counter counts the number of times the thread calls the computation kernel function; if the counter reaches a predetermined threshold, the thread is prohibited from calling the computation kernel function.

[0114] For example, suppose an AI computing task and an encoding / decoding task are running on the GPU of a worker node. The encoding / decoding task typically performs parallel computations (such as motion estimation and DCT transform) by calling kernel functions. When the encoding / decoding process creates a new thread, libsched.so intercepts the thread creation process and allocates and initializes a private counter for that thread. This counter records the number of times the thread has called the computation kernel function. If the current thread's call count has reached a preset threshold, such as 500 times, an error is returned directly, preventing the kernel function from executing on the GPU. For instance, after the encoding / decoding process starts, the interception library initializes a counter for each worker thread of the encoding / decoding process. During video encoding / decoding, threads frequently call CUDA kernel functions for image denoising, color conversion, and other operations. If a thread has called the kernel function 499 times in one second, the 500th call is allowed normally; the 501st call is intercepted, an error is returned, and the encoding / decoding process will skip non-critical processing steps or pause encoding until the next cycle.

[0115] According to the technical solution in the above example embodiment, by maintaining a kernel function call counter for each thread of the encoding and decoding process in the interception library and prohibiting further calls when the threshold is reached, the GPU computing resource usage of the encoding and decoding process can be constrained, preventing the encoding task from occupying too much GPU computing resources and affecting inference performance.

[0116] Furthermore, in the example embodiment, the electronic device 200 resets the counter when the thread corresponding to the encoding / decoding process calls the API related to the encoding / decoding task. For example, an interception library (such as libsched.so) maintains a kernel function call counter for each encoding / decoding thread to limit the number of times it calls the kernel function per unit time, preventing it from excessively consuming GPU computing power. The interception library libsched.so further intercepts high-level APIs related to encoding / decoding. When it detects that a thread is calling these key encoding / decoding task APIs, it determines that the thread has entered a new encoding / decoding task cycle and resets the counter.

[0117] According to the technical solution in the above example embodiment, when the thread of the encoding / decoding process calls the API related to the encoding / decoding type task, the kernel function call counter is cleared to achieve the periodic resource quota reset of the encoding / decoding process's call to computing resources. This enables control over the upper limit of computing resources used by the encoding / decoding process and prevents malicious or erroneous programs from performing covert calculations through the encoding / decoding API.

[0118] Figure 4 A schematic diagram of the node GPU resource scheduling process provided according to some embodiments of this specification is shown.

[0119] Reference Figure 4 As shown, after a task scheduling unit Pod is scheduled to a worker node, the GPU Local Scheduler controls and coordinates scheduling unit Pods of different task types co-located on the same physical card. The interception library libsched.so is responsible for intercepting GPU call requests from various task processes and communicating with the GPU Local Scheduler, which then coordinates resources. The main process is as follows:

[0120] The GPU device plugin injects the client interception library libsched.so into the container to intercept GPU call requests from task processes. (2) The interception library libsched.so in the compute scheduling unit, i.e. the compute Pod, periodically collects GPU usage metrics such as SLA metrics for each task process and reports the collected GPU usage metrics to the GPU LocalScheduler as an auxiliary decision. (3) Computing power control: The GPU local scheduler determines whether the compute process is being interfered with and the degree of interference based on the GPU usage metrics such as SLA metrics of the compute process, and then adjusts the time slice of computing power used by the codec Pod of the worker node to ensure that the SLA metrics of the compute process are within the allowable range. (4) Memory control: The codec Pod has low memory usage requirements and strictly limits the upper limit of usable memory, such as a memory threshold of 1 GiB. When the codec process requests video memory, the interception library libsched.so intercepts the video memory request from the codec process, and determines whether the current amount of video memory used will exceed the video memory threshold, thereby allowing or denying the request.

[0121] According to the technical solution in the above example embodiment, by adjusting the SLA index of the task process based on the feedback of the GPU local scheduler running on the node side, the computing resources and encoding / decoding resources of the GPU can be fully utilized while ensuring the SLA index of the task process.

[0122] In addition, in some embodiments, the GPU local scheduler of the worker node also controls the behavior of the compute process and the codec process, prohibiting them from using each other's hardware resources. The main control methods are as follows: (1) For the compute process: In the interception library libsched.so, it is prohibited to call the API related to the codec. For example, all related API calls return permission errors. (2) For the codec process: Since the codec process also needs to use a very small amount of computing power, it is not possible to simply prohibit the use of computing power. It is necessary to set the computing mode of the codec process. In the example embodiment, in the interception library libsched.so, a counter for the number of kernel function calls is initialized for each thread in the codec process. When the thread submits the kernel function, the counter is incremented by 1; when the thread calls the API for encoding or decoding images, the counter is cleared. If the counter has reached the set threshold when the thread submits the kernel function, the submission of the kernel is prohibited.

[0123] When multiple types of tasks are co-located, there may be situations where co-location is no longer suitable. For example, when computation tasks and encoding / decoding tasks are co-located, slow node detection and rescheduling are used to detect situations where co-location is no longer suitable, and to evict encoding / decoding codec type tasks to avoid interference between computation tasks and encoding / decoding tasks. In some example embodiments, task processes include computation processes or encoding / decoding processes, and task scheduling units include computation scheduling units or encoding / decoding scheduling units. The master node's electronic device 200 determines the co-location operation indicators on the GPUs of each worker node. The co-location operation indicators include the operation indicators of computation processes and the operation indicators of encoding / decoding processes, such as the GPU memory usage indicators of computation processes and encoding / decoding processes. If the GPU resources used by the encoding / decoding process of a worker node are consistently less than a preset resource threshold, the encoding / decoding scheduling unit on the worker node is evicted, and the worker node is marked as non-co-located. If the GPU memory used by the computation process of a worker node is higher than a preset GPU memory threshold, the encoding / decoding scheduling unit on the worker node is evicted, and the worker node is marked as non-co-located.

[0124] For example, suppose there are 3 worker nodes in the cluster (node ​​1, node 2, and node 3), with a preset resource threshold of 10%. Each node runs two types of tasks: a computation scheduling unit for AI large model inference Pods (high priority) and an encoding / decoding scheduling unit for video transcoding Pods (low priority). Node 1's computation Pods have 8 GiB of GPU memory usage and 90% GPU utilization; the encoding / decoding Pods have 0.5 GiB of GPU memory usage and 2% GPU utilization, which remains unchanged for 10 minutes. Electronic device 200 determines that the GPU resources used by the encoding / decoding process are consistently below the preset resource threshold, and evicts the encoding / decoding Pods on node 1, allowing node 1 to be dedicated to computation tasks and improving the GPU resource utilization of node 1.

[0125] According to the technical solutions in the above example embodiments, on the one hand, if the GPU resources occupied by the encoding and decoding process are continuously less than the preset resource threshold, the encoding and decoding scheduling unit on the worker node is evictded, which can release resources when the encoding and decoding task is under low load for a long time and avoid "wasting space"; on the other hand, if the video memory occupied by the computing process on the worker node is higher than the preset video memory threshold, the encoding and decoding scheduling unit on the worker node is evictded, which can prioritize the computing task when the video memory pressure of the computing task is too high.

[0126] Figure 5 A schematic flowchart of slow node detection provided according to some embodiments of this specification is shown.

[0127] Reference Figure 5 As shown, in step S510, the mixed operation indicators of each working node are collected.

[0128] In the example embodiment, mixed-distribution operation metrics for each worker node in the cluster are collected from the GPU Local Scheduler. These metrics include resource usage metrics and resource processing metrics. Resource usage metrics include memory usage metrics for computation processes and encoding / decoding processes, while resource processing metrics include request processing time, request response time, etc.

[0129] In step S520, it is determined whether the computing time slice occupied by encoding and decoding is continuously less than the threshold. If yes, proceed to step S540; otherwise, proceed to step S530.

[0130] In the example embodiment, the electronic device 200 determines whether the computing time slice occupied by the codec scheduling unit of the worker node is continuously less than the threshold. If so, it proves that the computing task of the worker node is under high load and is not suitable for co-location. The codec scheduling unit is expelled, and the worker node is marked as non-co-location and then exits.

[0131] In step S530, it is determined whether the video memory occupied by the computing process is higher than the threshold. If yes, proceed to step S540; otherwise, proceed to step S550.

[0132] In the example embodiment, the electronic device 200 determines whether the video memory occupied by the compute scheduling unit (Compute Pod) of the worker node is higher than the threshold. If so, it proves that the compute task of the worker node is in a high video memory usage state and is not suitable for mixed deployment.

[0133] In step S540, the codec scheduling unit is evicted and the working node is marked as non-combinable.

[0134] In an example embodiment, electronic device 200 evicts the codec scheduling unit Codec Pod via an eviction device, marks the working node as non-compatible, and then exits.

[0135] In step S550, it is determined whether the node is in a mixed state. If it is, no processing is performed; otherwise, proceed to step S560.

[0136] In the example embodiment, the electronic device 200 determines whether the current working node is in a miscible state. If not, it proceeds to step S560. For example, if the GPU resources occupied by the encoding / decoding process are not continuously less than a preset resource threshold and the video memory occupied by the computing process is not higher than a preset video memory threshold, the electronic device 200 determines the label of the working node; if the label shows that the working node is a non-miscible part, the working node is marked as a miscible part.

[0137] In step S560, the working node is marked as a mixable part.

[0138] In an example embodiment, the electronic device 200 marks the working node as a mixable state, for example, by changing the label of the working node to a mixable state.

[0139] According to the technical solutions in the above example embodiments, on the one hand, by using slow node detection and rescheduling technology, the codec task can be rescheduled to a better working node in a timely manner, ensuring the SLA of the computing task while improving the performance of the codec task; on the other hand, when it is determined that the GPU is not suitable for mixed deployment, it is promptly evicted to avoid resource waste.

[0140] Figure 6 A flowchart illustrating a GPU resource processing method provided according to further embodiments of this specification is shown.

[0141] Reference Figure 6 As shown, in step S610, the ListandWatch interface is called to detect GPU resources.

[0142] In the example embodiment, the GPU device plugin acts as a bridge for node GPU resources to access the cluster, responsible for detecting different types of virtualized resources abstracted from the GPU and reporting them to the cluster. The ListAndWatch interface of the GPU device plugin is the standard mode for interaction between the Kubernetes client (such as kubelet) of worker node 120 and the API server of master node 110. It is used to obtain a complete list of currently specified resources (such as all nodes with GPUs) and to establish a long-term connection to continuously listen for incremental changes (addition, modification, deletion) of GPU resource types. Electronic device 200 uses the GPU device plugin to call the ListAndWatch interface to detect GPU resources on worker node 120, obtain the detected virtualized GPU resources on worker node 120, and abstract a physical GPU card into a computing GPU card and a codec GPU card.

[0143] For example, the Kubelet (node ​​daemon) on worker node 120 obtains node device information from the GPU Device Plugin: (1) The GPU Device Plugin obtains device information from the node GPU Local Scheduler, and for each physical GPU, it abstracts it into a Compute card and a Codec card. (2) The GPU Device Plugin returns the abstracted virtualization resources to the Kubelet.

[0144] In step S620, the node expands its resources.

[0145] In the example implementation, the GPU device plugin reports the acquired virtualized GPU resources to the cluster resource orchestration system, providing resource information to the cluster scheduler. For example, Kubelet reports the node's virtualized GPU resources to the API server of master node 110.

[0146] In step S630, the Pod is scheduled.

[0147] In the example embodiment, the scheduler of master node 110 schedules user-submitted task scheduling unit Pods. The task scheduling unit Pod requests either Compute GPU resources or Codec GPU resources, and the requested amount is less than the total number of physical GPUs on a single node. The scheduler schedules the Pod to a node with sufficient idle resources. After the Pod is scheduled to a node, it is allocated the appropriate virtualization resources.

[0148] In step S640, container startup preparation is performed.

[0149] In the example implementation, Kubelet listens for the message that the node container has been successfully scheduled and begins to create the container.

[0150] In step S650, the device plug-in interface is invoked to allocate the GPU.

[0151] In the example implementation, the Kubelet calls the GPU Device Plugin to allocate virtualization resources of the corresponding task type to the task scheduling unit Pod. For example, when the Kubelet creates a user container, it performs a series of operations such as device initialization and environment variable setting through the Allocate interface.

[0152] In step S660, the basic environment is set up for the container.

[0153] In the example embodiment, the GPU device plugin sets up the basic environment for the container based on the virtualization resource information requested by the task scheduling unit Pod, including: (1) mounting the client interception library for node GPU resource scheduling into the container so that the computing power request and video memory request of the task process in the container can be intercepted by the client interception library; (2) setting container-related environment variables.

[0154] In step S670, the use of computing power and video memory is coordinated.

[0155] In the example embodiment, a local GPU scheduler performs node GPU resource scheduling. Running within a single node, the GPU resource scheduler coordinates the computing power and memory resources of both when computational tasks and codec tasks on the same physical card are mixed, ensuring that the SLA (Service Level Agreement) is met.

[0156] In step S680, mixed-part indicators are collected.

[0157] In the example embodiment, the electronic device 200 detects various distributed operation metrics of the task scheduling unit Pod or task process of the worker node through the eviction mechanism. Distributed operation metrics include resource usage metrics and resource processing metrics. Resource usage metrics include the memory usage metrics of the computing process and the encoding / decoding process, and resource processing metrics include request processing time, request response time, etc.

[0158] In step S690, timely expulsion is carried out.

[0159] In the example embodiment, the electronic device 200 detects various operating metrics of the task scheduling unit Pod or task process of the worker node through the eviction device. When it is determined that the worker node is no longer suitable for continued co-location, the encoding and decoding tasks on the worker node are evicted, and the GPU device plugin suspends the reporting of encoding and decoding resources of the worker node until timeout.

[0160] According to the technical solutions in the above example embodiments, on the one hand, based on GPU virtualization technology, various types of tasks can be deployed on a single physical GPU, significantly improving resource utilization; on the other hand, based on the negative feedback adjustment strategy of node SLA index, the computing and encoding / decoding resources of the GPU can be fully utilized while ensuring SLA; furthermore, through slow node detection and rescheduling technology, the encoding / decoding codec task can be rescheduled to a better node in a timely manner, ensuring SLA while improving the performance of the encoding / decoding codec task.

[0161] This specification, in another aspect, provides a non-transitory storage medium storing at least one set of executable instructions for performing GPU resource processing. When these executable instructions are executed by a processor, they instruct the processor to implement the steps of the GPU resource processing method described in this specification. In some possible implementations, various aspects of this specification can also be implemented as a program product comprising program code. When the program product is run on an electronic device 200, the program code causes the electronic device 200 to perform the steps of the GPU resource processing method described in this specification. The program product for implementing the above method may employ a portable compact disc read-only memory (CD-ROM) containing program code and may run on the electronic device 200. However, the program product of this specification is not limited thereto. In this specification, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. The aforementioned computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing the operations described herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on electronic device 200, partially on electronic device 200, as a standalone software package, partially on electronic device 200 and partially on a remote computing device, or entirely on a remote computing device.

[0162] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multi-GPU resource processing and parallel processing are possible or may be advantageous.

[0163] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0164] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0165] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and aiding in the understanding of a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art may readily identify some of the devices as separate embodiments when reading this specification. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. It is also valid when each secondary embodiment contains fewer than all the features of a single foregoing disclosed embodiment.

[0166] Each patent, patent application, publication of a patent application, and other material such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated by reference, except for any identical content appearing in the related documents that may be inconsistent with or conflict with this document, or any identical document content that may have a limiting effect on the widest scope of the claims. For example, in the event of any inconsistency or conflict between the description, definition, and / or use of terms associated with any included material and those related to this document, the terminology herein shall prevail.

[0167] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A GPU resource processing method applied to a GPU hybrid deployment system, the GPU hybrid deployment system comprising at least one worker node, each worker node comprising at least one GPU, each GPU comprising task units of at least two task types, the at least two task types including computation type and encoding / decoding type, each task unit of the at least two task types including a computation unit and an encoding / decoding unit, each GPU being virtualized into at least two virtual GPUs of the at least two different task types, the task type being the task type executed by the task unit on the GPU, the at least two virtual GPUs of the at least two different task types including a computation GPU of the computation type and an encoding / decoding GPU of the encoding / decoding type, the method comprising: Determine the GPU usage metrics of the task processes corresponding to each task type on the GPU of the working node; the task process corresponds to the task scheduling unit of the working node, including a computing process or an encoding / decoding process, and the task scheduling unit includes a computing scheduling unit or an encoding / decoding scheduling unit. as well as Based on the GPU usage metrics, resource scheduling is performed on the task scheduling units corresponding to each task type on the GPU of the working node. Determine the mixed-process operation metrics on the GPU of each of the aforementioned working nodes, wherein the mixed-process operation metrics include the operation metrics of the computing process and the operation metrics of the encoding / decoding process; as well as If the GPU resources occupied by the encoding / decoding process of the working node are continuously less than a preset resource threshold and / or the video memory occupied by the computing process of the working node is higher than a preset video memory threshold, then the encoding / decoding scheduling unit on the working node is evicted, and the working node of the hybrid system is marked as a non-hybrid node; if the GPU resources occupied by the encoding / decoding process of the working node marked as non-hybrid are not continuously less than the preset resource threshold and the video memory occupied by the computing process is not higher than the preset video memory threshold, then the working node is marked as a hybrid node.

2. The method of claim 1, wherein, The task process includes a computing process or an encoding / decoding process, and the task scheduling unit includes a computing scheduling unit or an encoding / decoding scheduling unit.

3. The method of claim 2, wherein, The task type is the computation type, and the resource scheduling of the task scheduling unit corresponding to each task type on the GPU of the worker node based on the GPU usage index includes: Based on the GPU usage metrics corresponding to the computing process, the running status of the computing process on the GPU of the worker node is determined, wherein the computing process corresponds to the computing scheduling unit; Based on the running status of the computing process, the GPU resources used by the encoding / decoding scheduling unit on the GPU of the working node are adjusted accordingly.

4. The method of claim 2, wherein, The task type is the encoding / decoding type. The step of scheduling resources for the task scheduling unit corresponding to each task type on the GPU of the working node based on the GPU usage metrics includes: Based on the GPU usage metrics corresponding to the encoding / decoding process, the GPU resources used by the encoding / decoding process are determined, wherein the encoding / decoding process corresponds to the encoding / decoding scheduling unit; Based on the used GPU resources and the preset encoding / decoding resource threshold, the encoding / decoding scheduling unit corresponding to the encoding / decoding process performs resource control to prevent the used GPU resources from exceeding the preset encoding / decoding resource threshold.

5. The method of claim 1, wherein, The method further includes: The interception library intercepts the GPU call requests of the task process to the GPU of the worker node. The interception library is an application programming interface (API) library that is pre-injected into the task scheduling unit corresponding to the worker node.

6. The method of claim 5, wherein, The task type includes computation type and encoding / decoding type, the task process includes computation process, and the method further includes: The interception library prevents the computing process from calling the APIs related to the codec type.

7. The method of claim 5, wherein, The step of scheduling resources for the task scheduling unit corresponding to each task type on the GPU of the worker node based on the GPU usage metrics includes: Based on the GPU usage metrics, the interception library performs feedback adjustments on the GPU call requests it intercepts.

8. The method of claim 7, wherein, The task process includes an encoding / decoding process, and the step of adjusting the GPU call requests intercepted by the interception library based on the GPU usage metrics includes: In the interception library, a counter is initialized for each thread in the encoding / decoding process. The counter is used to count the number of times the thread calls the kernel function. If the counter reaches a predetermined threshold, the thread is prohibited from calling the computation kernel function.

9. The method of claim 8, wherein, The method further includes: When the thread calls the API related to the task of the codec type, the counter is cleared.

10. The method of claim 5, wherein, The step of determining the GPU usage metrics of the task processes corresponding to each task type on the GPU of the working node includes: Collect the metric parameters corresponding to the GPU call requests intercepted by the interception library; Based on the metric parameters corresponding to the GPU request, determine the GPU usage metrics of the task processes corresponding to each task type on the GPU of the worker node.

11. An electronic device, comprising: At least one storage medium storing at least one instruction set for GPU resource processing; as well as At least one processor is communicatively connected to the at least one storage medium. When the electronic device is running, the at least one processor reads the at least one instruction set and executes the GPU resource processing method according to any one of claims 1-10 according to the instructions of the at least one instruction set.

Images