Heterogeneous GPU scheduling method and device, storage medium, electronic equipment and product

By configuring heterogeneous GPU information and deployment scripts for the management node, the problem of incompatibility in heterogeneous GPU scheduling was solved, enabling efficient collaborative work and improved resource utilization of different types of GPUs in the Kubernetes environment.

CN119045995APending Publication Date: 2024-11-29CHINA CONSTRUCTION BANK +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411072784.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

Existing technologies cannot effectively unify the scheduling of various types of heterogeneous GPUs, resulting in low resource utilization.

Method used

By configuring heterogeneous GPU configuration information and deployment scripts for the management node, the management node switches permissions in case of anomalies, creates appropriate types of compute nodes based on business requests and compute node attribute information, and creates replica compute nodes when necessary, to ensure that different types of GPUs work together efficiently in the Kubernetes environment.

Benefits of technology

It enables efficient collaborative work of heterogeneous GPUs in the same environment, improves resource utilization and system stability, and ensures compatible scheduling between different types of GPUs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119045995A_ABST
    Figure CN119045995A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computers, and discloses a heterogeneous GPU (Graphics Processing Unit) scheduling method and device, a storage medium, electronic equipment and a product, applied to a Kubernetes system comprising at least two management nodes, the method comprises the following steps: receiving a service request, under the condition that the service request and the attribute information of the created computing node meet a first creation condition of the computing node, creating the computing node corresponding to the GPU type corresponding to the service request based on the heterogeneous GPU configuration information and the deployment script, and routing the service request to a new computing node; and in the running process of the created computational nodes, under the condition that the running parameters of any created computational node meet the second creation condition of the computational nodes, creating the copy computational nodes of the created computational nodes based on the heterogeneous GPU configuration information and the deployment script, thereby realizing efficient and compatible scheduling of the heterogeneous GPU.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a heterogeneous GPU scheduling method, device, storage medium, electronic equipment and product. Background Art

[0002] The Graphics Processing Unit (GPU) is one of the core components of artificial intelligence technology. GPU scheduling aims to distribute workloads to available GPU resources in an efficient manner, ensure that all processing cores are effectively utilized, and optimize the performance of the entire system.

[0003] At present, the existing technology usually performs GPU scheduling based on Kubernetes (k8s), where k8s is a container cluster management system used to manage containerized applications on multiple hosts in a cloud platform and provide a device plug-in framework to support the management and scheduling of extended resources (e.g., GPUs).

[0004] However, different types of GPUs may require different management and scheduling strategies. Existing GPU scheduling focuses on a single type of GPU. In a diverse hardware environment, it is difficult to uniformly schedule multiple types (i.e., heterogeneous) GPUs, resulting in overall resource utilization. Summary of the invention

[0005] The present invention provides a heterogeneous GPU scheduling method, device, storage medium, electronic device and product to solve the problem that the existing heterogeneous GPU scheduling method cannot be universally applied to multiple types of GPUs, improve the versatility of heterogeneous GPU scheduling, and ensure that different types of GPUs can work efficiently and collaboratively in the same Kubernetes environment.

[0006] According to one aspect of the present invention, a heterogeneous GPU scheduling method is provided, which is applied to a Kubernetes system. The Kubernetes system includes at least two management nodes. The management authority of the management node is switched to other management nodes when the management node is abnormal. The management node is configured with heterogeneous GPU configuration information and deployment scripts. The management node is used to schedule created computing nodes. Different computing nodes correspond to different GPU types. The computing nodes call GPU resources of corresponding GPU types.

[0007] The method includes: receiving a business request, determining a first creation condition of a computing node based on the business request and attribute information of a created computing node; determining a GPU type corresponding to the business request when the first creation condition of the computing node is met, creating a computing node corresponding to the GPU type based on heterogeneous GPU configuration information and a deployment script, and routing the business request to a new computing node so that the new computing node calls a GPU resource of a corresponding GPU type to respond to the business request;

[0008] Also, during the operation of the created computing nodes, the operating parameters of each created computing node are obtained, and when the operating parameters of any created computing node meet the second creation condition of the computing node, a replica computing node of the created computing node is created based on the heterogeneous GPU configuration information and the deployment script, wherein the replica computing node has the same GPU type as the created computing node.

[0009] Optionally, the first creation condition includes that the created computing node does not support the service request and / or the load data of the created computing node reaches a first load threshold;

[0010] Determining a first creation condition of the computing node based on the service request and the attribute information of the created computing node includes:

[0011] Matching the service type of the service request with the GPU type of the created computing node, and if the service type of the service request does not match the GPU type of each created computing node, determining that the first creation condition is met;

[0012] When the GPU type of at least one created computing node matches the service type of the service request and the load data of at least one created computing node reaches a first load threshold, it is determined that the first creation condition is met.

[0013] Optionally, the operating parameters of the created computing node include performance indicators and load data;

[0014] The second creation condition includes that a performance indicator of the created computing node is abnormal, and / or the load data reaches a second load threshold.

[0015] Optionally, the deployment script includes a scheduling script for the computing nodes and a creation script for the computing nodes corresponding to the heterogeneous GPUs;

[0016] The process of creating a new computing node or a replica computing node includes: calling a corresponding creation script based on the GPU type corresponding to the computing node to be created, creating a new computing node or a replica computing node, and scheduling the new computing node or the replica computing node through a scheduling script, wherein the new computing node or the replica computing node is configured with a unified scheduling interface, and the scheduling interface supports scheduling of the scheduling script.

[0017] Optionally, the created compute node is configured with one or more replica compute nodes;

[0018] The method also includes: when the created computing node fails or the load data of the created computing node meets the second load threshold, starting the replica computing node; wherein the multiple replica computing nodes support switching.

[0019] Optionally, the created computing node is configured with a driver script, a computing library script, and an interface adaptation script; the interface adaptation script is used to call GPU resources of the corresponding GPU type;

[0020] The created computing nodes include new computing nodes and replica computing nodes.

[0021] Optionally, the method further includes: routing the service request to a created computing node when the first creation condition of the computing node is not met.

[0022] Optionally, routing the service request to the created computing node includes: according to the service type of the service request, routing the service request to the created computing node corresponding to the service type.

[0023] Optionally, the created compute node is set with label or taint information;

[0024] According to the business type of the business request, the business request is routed to the created computing node corresponding to the business type, including: matching the business type of the business request with the created computing node set with labels or taint information, and if the match is successful, routing the business request to the created computing node with successful matching.

[0025] Optionally, routing the business request to the created computing node includes: routing the business request to the created computing node according to a load balancing principle; or, when multiple business requests are received, routing the business request to the created computing node based on the priority of the business request.

[0026] Optionally, the method also includes: sending non-functional test information to the created computing node based on a preset time interval, and obtaining the non-functional test result of the created computing node based on the response information of the created computing node; when the non-functional test result of the created computing node is a test failure, sending a restart instruction to the created computing node.

[0027] According to one aspect of the present invention, a heterogeneous GPU scheduling device is provided, which is configured in a Kubernetes system. The Kubernetes system includes at least two management nodes. The management authority of the management node is switched to other management nodes when the management node is abnormal. The management node is configured with heterogeneous GPU configuration information and deployment scripts. The management node is used to schedule created computing nodes. Different computing nodes correspond to different GPU types. The computing nodes call GPU resources of corresponding GPU types.

[0028] The device includes:

[0029] A business request routing module is used to receive a business request, determine a first creation condition of a computing node based on the business request and attribute information of a created computing node; if the first creation condition of the computing node is met, determine a GPU type corresponding to the business request, create a computing node corresponding to the GPU type based on heterogeneous GPU configuration information and a deployment script, and route the business request to a new computing node so that the new computing node calls a GPU resource of a corresponding GPU type to respond to the business request;

[0030] The replica business request routing module is used to obtain the operating parameters of each created computing node during the operation of the created computing node, and when the operating parameters of any created computing node meet the second creation condition of the computing node, create a replica computing node of the created computing node based on the heterogeneous GPU configuration information and deployment script, wherein the GPU type of the replica computing node is the same as that of the created computing node.

[0031] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0032] at least one processor; and

[0033] a memory communicatively connected to at least one processor; wherein,

[0034] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by at least one processor so that the at least one processor can execute the heterogeneous GPU scheduling method of any embodiment of the present invention.

[0035] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the heterogeneous GPU scheduling method of any embodiment of the present invention when executed.

[0036] According to another aspect of the present invention, a computer program product is provided, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the heterogeneous GPU scheduling method of any embodiment of the present invention is implemented.

[0037] The technical solution of the embodiment of the present invention configures heterogeneous GPU configuration information and deployment scripts for the management node, and the management authority of the management node is switched to other management nodes when the management node is abnormal. Different computing nodes correspond to different GPU types. Based on the business request and the attribute information of the created computing node that meet the first creation condition of the computing node, a computing node corresponding to the GPU type corresponding to the business request is created based on the heterogeneous GPU configuration information and the deployment script; and during the operation of the created computing node, when the operating parameters of any created computing node meet the second creation condition of the computing node, a replica computing node of the created computing node is created based on the heterogeneous GPU configuration information and the deployment script, instead of directly using the GPU resources as the node scheduled by the management node, thereby solving the problem of incompatibility of heterogeneous GPU scheduling and ensuring that different types of GPUs can work efficiently together in the same Kubernetes environment.

[0038] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 is a flow chart of a heterogeneous GPU scheduling method provided by Embodiment 1 of the present invention;

[0041] Figure 2 is a flow chart of a heterogeneous GPU scheduling method provided by Embodiment 2 of the present invention;

[0042] Figure 3 It is a structural schematic diagram of a heterogeneous GPU scheduling device provided by Embodiment 3 of the present invention;

[0043] Figure 4 It is a structural schematic diagram of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0045] It should be noted that the terms "first characteristic data", "second characteristic data", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] The acquisition, storage and / or processing of data in the technical solutions involved in this application comply with the relevant provisions of national laws and regulations.

[0047] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0048] Embodiment 1

[0049] Figure 1 This is a flowchart of a heterogeneous GPU scheduling method provided in the first embodiment of the present invention. This embodiment is applicable to the case of unified scheduling of heterogeneous GPUs. The method can be executed by a heterogeneous GPU scheduling device. The heterogeneous GPU scheduling device can be implemented in the form of hardware and / or software. The heterogeneous GPU scheduling device can be configured in the electronic device provided in the embodiment of the present invention.

[0050] The method is applied to the Kubernetes system, which includes at least two management nodes. The management authority of the management node is switched to other management nodes when the management node is abnormal. The management node is configured with heterogeneous GPU configuration information and deployment scripts. The management node is used to schedule the created computing nodes. Different computing nodes correspond to different GPU types. The computing nodes call the GPU resources of the corresponding GPU type.

[0051] In the present embodiment, the Kubernetes system is a distributed system, including a management node (Master) and a worker node (Node), wherein the management node and the worker node are both physical nodes, for example, the management node is composed of one or more physical servers, and each worker node is a GPU. It should be noted that any of the two or more management nodes included in each Kubernetes system is configured with management authority, and the management node configured with management authority can schedule the created computing nodes, and the management node not configured with management authority cannot schedule the created computing nodes. Among them, the management authority is the authority to manage the created computing nodes. The created computing node is a created computing node, and the computing node is a virtual node participating in the calculation in the Kubernetes system. It can be understood that a created computing node corresponds to a minimum unit (Pod) scheduled by the Kubernetes system.

[0052] It should be noted that heterogeneous GPUs include multiple GPU types, where the GPU type is the architecture type of the GPU, for example, GPU types include but are not limited to NVIDIA GPU (graphics processing unit), DCU (Deep-learning Computing Unit) or NPU (Neural-network Process Units). The GPU types corresponding to different computing nodes can be the same or different, and the computing nodes can call the GPU resources of the corresponding GPU type.

[0053] If a management node configured with management permissions is abnormal, it can be considered that the management node cannot manage the created computing nodes normally. In this case, the management permissions of the management node are revoked, and management permissions are configured for other management nodes, so that there are always available management nodes during the heterogeneous GPU scheduling process, ensuring the high availability of heterogeneous GPU scheduling.

[0054] Exemplarily, it is assumed that there are three management nodes in the Kubernetes system, namely Master1, Master2 and Master3; the created computing nodes include at least one computing stage of GPU type, namely computing node 1, computing node 2 and computing node 3, and the GPU types corresponding to the three computing nodes can be the same or different. The management node currently configured with management authority is Master1. In the case of Master1 abnormality, the management authority is switched from Master1 to Master2. Optionally, the multiple management nodes include master nodes and slave nodes, wherein the priority of the master node is higher than the priority of the slave node, and the created computing node is scheduled by the master node. In the case of the master node abnormality, the management authority of the master node is switched to the slave node, and the created computing node is scheduled by the slave node. Among them, there can be at least one slave node, and the priority of the slave node is the same. It can be understood that multiple management nodes have the same management function and can all realize the scheduling of the created computing nodes. By setting at least two management nodes, it can be ensured that the management authority is switched imperceptibly in the case of abnormal management nodes to ensure the normal scheduling of computing nodes.

[0055] The multiple management nodes included in the Kubernetes system are all configured with the same heterogeneous GPU configuration information and deployment scripts, wherein the heterogeneous GPU configuration information is the configuration information of the heterogeneous GPU that the management node can call, and the deployment script includes multiple scripts for deploying computing nodes. Exemplarily, the heterogeneous GPU configuration information and deployment script configured by the management node are obtained based on parsing the job definition (YAML) file. Among them, the YAML file can be a file obtained based on the user's parameter configuration operation on the YAML file template, or it can be a file generated based on the user's configuration operation in the visual interactive configuration interface, and this embodiment does not limit this.

[0056] like Figure 1 As shown, the method includes:

[0057] S110, receiving a business request, and determining a first creation condition of the computing node based on the business request and attribute information of the created computing node; when the first creation condition of the computing node is met, determining a GPU type corresponding to the business request, creating a computing node corresponding to the GPU type based on heterogeneous GPU configuration information and a deployment script, and routing the business request to a new computing node, so that the new computing node calls a GPU resource of the corresponding GPU type to respond to the business request.

[0058] In this embodiment, the service request is a request for a service that depends on GPU resources. It should be noted that the service request is a request generated based on a computing framework selection operation and / or other task information. In some embodiments, during the generation of the service request, the selected computing framework is determined by identifying the user's computing framework selection operation, and the service request is generated based on the selected computing framework.

[0059] The attribute information of the created computing node includes but is not limited to GPU type information and load information, wherein the GPU type information is information characterizing the GPU type corresponding to the created computing node. It can be understood that different GPU types correspond to different computing performances, wherein the computing performance includes but is not limited to machine learning graphics processing efficiency information, floating-point operation performance information, parallel computing performance information and deep learning acceleration performance information; the load information is information characterizing the load that the created computing node is processing and / or the load to be processed.

[0060] Specifically, the management node responds to the received business request and determines the first creation condition of the computing node based on the business request and the attribute information of the created computing node. Among them, the first creation condition is the creation determination condition of the computing node. If the business request and the attribute information of the created computing node meet the first creation condition of the computing node, it indicates that the created computing node does not meet the demand for scheduling GPU resources in response to the business request, that is, the management node cannot schedule the GPU resources that can efficiently respond to the business request by scheduling the created computing node, and a new computing node needs to be created to respond to the business request. The type and content of the business request are not limited here, and the business request can be generated according to the business processing requirements.

[0061] In some embodiments, the first creation condition includes but is not limited to attribute conditions and / or load conditions. Optionally, the first creation condition includes that the created computing node does not support the business request and / or the load data of the created computing node reaches a first load threshold; accordingly, the business type of the business request and the GPU type of the created computing node are matched, and when the business type of the business request does not match the GPU type of each created computing node, it is determined that the first creation condition is met; when the GPU type of at least one created computing node matches the business type of the business request and the load data of at least one created computing node reaches the first load threshold, it is determined that the first creation condition is met.

[0062] In this embodiment, the load data is data used to express the amount of business requests processed by the created computing node. The business type is the type of business corresponding to the business request. It should be noted that the business type of the business request is determined based on the selected computing framework. Different GPU types are pre-configured with different computing frameworks. Exemplarily, computing frameworks such as the Python framework, the Java framework, and the C++ framework are pre-configured on computing nodes corresponding to different GPU types.

[0063] Specifically, the management node determines the selected computing framework corresponding to the business request by parsing the business request, determines the business type based on the selected computing framework, and matches the business type of the business request with the GPU type of the created computing node based on a preset matching algorithm. If the selected computing framework corresponding to the business type is the same as the pre-configured computing framework of the GPU type, the matching result corresponding to the created computing node is a successful match; if the selected computing framework corresponding to the business type is different from the pre-configured computing framework of the GPU type, the matching result corresponding to the created computing node is an unsuccessful match, wherein the preset matching algorithm includes but is not limited to label matching and taint matching, and the matching results include successful matching and unsuccessful matching.

[0064] If the matching results corresponding to all created computing nodes are all unsuccessful matches, it can be considered that the business type of the business request does not match the GPU type of each created computing node, and the created computing node does not support the business request, and it is determined that the first creation condition is met.

[0065] If the matching result corresponding to at least one created computing node is a successful match, it can be considered that the business type of the business request matches the GPU type of at least one created computing node, and the load data of at least one successfully matched created computing node is determined. In the case where the load data of the created computing node is greater than or equal to the first load threshold, it can be considered that the GPU resources that can be scheduled by the created computing node cannot provide sufficient computing resources to respond to the business request, and the created computing node is determined to be unschedulable. Among them, the first load threshold is a preset threshold of the load data of the created computing node, which is used to determine that the created computing node is unschedulable. If it is determined that at least one successfully matched created computing node is not schedulable, it is determined that the first creation condition is met.

[0066] It should be noted that, for a created computing node that has a corresponding created replica computing node, and the created computing node and the created replica computing node are configured with scheduling permissions for different GPU resources of the same GPU type, the load data of the created computing node includes the first load data of the created computing node and the second load data of the created replica computing node of the created computing node, wherein the first load data is the sum of the load data of one or more GPU resources that can be scheduled by the created computing node itself, and the second load data is the sum of the load data of one or more GPU resources that can be scheduled by the created replica computing node itself. Among them, the created replica computing node of the created computing node is a replica node of the created computing node, and a created replica computing node corresponds to a replica Pod of the Pod corresponding to the created computing node.

[0067] For a created computing node for which there is no corresponding created replica computing node, the load data of the created computing node includes first load data of the created computing node.

[0068] For a corresponding created replica computing node, and the created computing node and the created replica computing node are configured with scheduling permissions for the same GPU resources, but only one of the created computing node and the created replica computing node is started, when the created computing node is started, the load data of the created computing node includes the first load data of the created computing node configured with management permissions; when the created replica computing node is started, the load data of the created computing node includes the second load data of the created replica computing node configured with management permissions.

[0069] Exemplarily, assume that there are two created computing nodes, Com1 and Com2, where Com1 has a created replica computing node RCom1, the first load threshold is X0, the load data of Com1 is X1, and the load data of Com1 is X2. When the matching results corresponding to Com1 and Com2 are both unmatched, it can be determined that the first creation condition is met; when the matching results corresponding to Com1 and Com2 are both successful matches, if X1≥X0 and X2≥X0, it can be determined that the first creation condition is met; when the matching results corresponding to Com1 are both successful matches, and the matching results corresponding to Com2 are unsuccessful matches, if X1≥X0, it can be determined that the first creation condition is met.

[0070] The technical solution of this embodiment, by setting the first creation condition to that the created computing node does not support the business request and / or the load data of the created computing node reaches a first load threshold, makes the first creation condition include attribute conditions and / or load conditions, thereby avoiding routing the business request to an inappropriate created computing node, and preventing the scheduled GPU resources from being unable to support the business request and / or being overloaded, which helps to improve the reliability and stability of heterogeneous GPU scheduling.

[0071] When the business request and the attribute information of the created computing node meet the first creation condition of the computing node, it can be considered that a new computing node supporting the business request needs to be created so that the new computing node can call the GPU resources that can respond to the business request, and the GPU type corresponding to the business request is determined as the target GPU type. Exemplarily, the GPU type corresponding to the business request can be obtained based on neural network prediction.

[0072] By parsing the heterogeneous GPU configuration information and calling the script in the deployment script for creating a computing node corresponding to the target GPU type, a new computing node is created, and the GPU resources that can be scheduled by the new computing node can respond to the business request. Exemplarily, assuming that the GPU type corresponding to the business request is a DCU type, a new computing node of the DCU type is created. A management node with management authority is configured to route the business request to the new computing node. During operation, the new computing node calls the GPU resources of the DCU type so that the called GPU resources respond to the business request.

[0073] S120. During the operation of the created computing nodes, the operating parameters of each created computing node are obtained. When the operating parameters of any created computing node meet the second creation condition of the computing node, a replica computing node of the created computing node is created based on the heterogeneous GPU configuration information and the deployment script, wherein the GPU type of the replica computing node is the same as that of the created computing node.

[0074] In this embodiment, the operating parameters are parameters that characterize the working status of the created computing nodes. Optionally, the operating parameters of the created computing nodes include performance indicators and load data, wherein the performance indicators are indicators that characterize the computing performance. For example, the performance indicators include memory utilization, central processing unit (CPU) computing resource utilization, GPU resource utilization, and the size of computing resources occupied by the process.

[0075] Specifically, during the operation of the created computing nodes, the management node acquires the parameters characterizing the working status of each created computing node by calling the operation parameter acquisition related script to obtain the operation parameters of the created computing nodes. Exemplarily, assuming that the GPU type corresponding to the created computing node is a DCU type, the acquisition related script may include the command "rocm-smi" for obtaining GPU resource utilization.

[0076] The second creation condition of the computing node is determined based on the operating parameters of the created computing node. The second creation condition is a condition for determining the creation of a replica computing node. If the operating parameters of the created computing node meet the second creation condition of the computing node, it can be considered that the created computing node needs to set a corresponding replica computing node. Optionally, the second creation condition includes that the performance indicator of the created computing node is abnormal, and / or the load data reaches a second load threshold.

[0077] In some embodiments, the second creation condition includes an abnormal performance indicator of the created computing node. When the performance indicator of the created computing node meets the abnormal judgment condition, it can be considered that the performance indicator is abnormal, and it is determined that the operating parameters of the created computing node meet the second creation condition of the computing node. Exemplarily, the abnormal judgment condition includes but is not limited to the GPU resource utilization rate being less than a preset utilization rate threshold, the fluctuation of the GPU resource utilization rate for a continuous preset time period being greater than a preset utilization rate fluctuation threshold, and the GPU power consumption being greater than a preset power consumption threshold.

[0078] In some embodiments, the second creation condition includes that the load data reaches a second load threshold. When the load data of the created computing node is greater than or equal to the second load threshold, it is determined that the operating parameters of the created computing node meet the second creation condition of the computing node. The second load threshold is a pre-set threshold of the load data of the created computing node, which is used to determine that the GPU resources that can be called by the created computing node are overloaded. It should be noted that the second load threshold is less than the first load threshold.

[0079] In some embodiments, the second creation condition includes that the performance indicator of the created computing node is abnormal and the load data reaches the second load threshold. When at least one of the performance indicator of the created computing node satisfies the abnormality determination condition and the load data of the created computing node is greater than or equal to the second load threshold, it is determined that the operating parameters of the created computing node meet the second creation condition of the computing node.

[0080] The technical solution of this embodiment, by setting performance indicators and load data as operating parameters, and setting abnormal performance indicators of created computing nodes, and / or load data reaching a second load threshold as the second creation condition, so that the second creation condition includes performance conditions and / or load conditions, can avoid routing business requests to created computing nodes with abnormal performance, and / or avoid overloading of scheduled GPU resources, which helps to improve the reliability and stability of heterogeneous GPU scheduling.

[0081] For a created computing node that meets the second creation condition of the computing node, a new replica computing node of the created computing node is created by parsing the heterogeneous GPU configuration information and calling the script in the deployment script for creating a replica computing node of the same GPU type as the created computing node.

[0082] It should be noted that, in the case where the created computing node and the replica computing node of the created computing node are configured with scheduling permissions for different GPU resources of the same GPU type, the new replica computing node and the computing node can schedule different GPU resources of the same GPU type. Exemplarily, assuming that the GPU type corresponding to the created computing node is a DCU type, the GPU type corresponding to the replica computing node created of the created computing node is also a DCU type. In the case where the created computing node and the replica computing node of the created computing node are configured with scheduling permissions for the same GPU resource, the new replica computing node and the created computing node can schedule GPU resources of the same GPU type.

[0083] The creation process of a computing node is the same as the creation process of a replica computing node. Optionally, the deployment script includes a scheduling script for the computing node and a creation script for the computing node corresponding to the heterogeneous GPU; the creation process of a new computing node or a replica computing node includes: calling the corresponding creation script based on the GPU type corresponding to the computing node to be created, creating a new computing node or a replica computing node, and scheduling the new computing node or the replica computing node through the scheduling script, wherein the new computing node or the replica computing node is configured with a unified scheduling interface, and the scheduling interface supports the scheduling of the scheduling script.

[0084] In this embodiment, the scheduling script is used to guide the management node to schedule the computing nodes or replica computing nodes, and the creation script is used to guide the creation of computing nodes or replica computing nodes. It can be understood that each creation script is used to guide the creation of computing nodes or replica computing nodes corresponding to a GPU type, and the scheduling script is used to guide the scheduling of computing nodes or replica computing nodes corresponding to all GPU types, thereby achieving heterogeneous GPU compatible scheduling.

[0085] Specifically, in the process of creating a new computing node or a replica computing node, the GPU type corresponding to the computing node to be created is determined based on the GPU type corresponding to the business request. It can be understood that the computing node to be created is a non-replica computing node, and the GPU type corresponding to the computing node to be created is the GPU type corresponding to the business request.

[0086] During the creation of a computing node, the management node creates a new computing node by calling the creation script corresponding to the GPU type corresponding to the computing node to be created, and repackages the new computing node. During the repackage process, the scheduling interface is configured for the new computing node.

[0087] During the creation process of the replica computing node, the management node creates a new computing node by calling the creation script corresponding to the GPU type corresponding to the computing node to be created. Since there is already a created computing node corresponding to the computing node to be created, the new computing node is used as a new replica computing node of the created computing node. During the secondary packaging process of the new replica computing node, a scheduling interface is configured for the new replica computing node.

[0088] Exemplarily, assuming that the Kubernetes system is used to execute scheduling of GPU resources of two GPU types, DCU type and NPU type, the deployment script includes a scheduling script for the computing node and creation scripts for computing nodes corresponding to the DCU type and the NPU type, respectively.

[0089] The technical solution of this embodiment creates a new computing node or a replica computing node by calling the corresponding creation script based on the GPU type corresponding to the computing node to be created, and schedules the new computing node or the replica computing node through the scheduling script, so that the management node can create computing nodes or replica computing nodes corresponding to different GPU types, and realize compatible scheduling of computing nodes or replica computing nodes corresponding to different GPU types, thereby improving the versatility of heterogeneous GPU scheduling and ensuring that different types of GPUs can work efficiently together in the same Kubernetes environment.

[0090] The computing node and the replica computing node both call the GPU resources based on the configured script. Optionally, the created computing node is configured with a driver script, a computing library script, and an interface adapter script; the interface adapter script is used to call the GPU resources of the corresponding GPU type; wherein the created computing node includes a new computing node and a replica computing node.

[0091] In this embodiment, the driver script is a script used to drive the creation of a computing node to run. The computing library script is a script used to call a computing library related to a business request, wherein the computing library includes multiple computing function libraries. Exemplarily, when the computing architecture pre-configured for the GPU type corresponding to the computing node is a Python architecture, the computing library includes but is not limited to the pytorch library and the transformer library. The interface adaptation script is a script used to adapt the GPU resource interface.

[0092] It should be noted that different GPU types have different interfaces for responding to scheduling, and different GPU types have different interface adaptation scripts for computing node configurations.

[0093] Exemplarily, if the business request received at the computing node or the replica computing node is a model training business request based on the Python architecture, the computing library script may call the pytorch library for model training in response to the business request.

[0094] The technical solution of this embodiment configures the created computing node with a driver script, a computing library script and an interface adaptation script, so that the created computing node has the function of calling GPU resources.

[0095] During the operation of the created computing node, the created computing node may fail or be overloaded. In some embodiments, the created computing node is configured with one or more replica computing nodes; accordingly, optionally, when the created computing node fails or the load data of the created computing node meets a second load threshold, the replica computing node is started; wherein, multiple replica computing nodes support switching.

[0096] Specifically, if the created computing node does not respond or incorrectly responds to the business request, it can be considered that the created computing node is faulty and the created computing node cannot support the business request, then any one of the one or more replica computing nodes configured for the created computing node is started; if the load data of the created computing node is greater than or equal to the second load threshold, it can be considered that the load data of the created computing node meets the second load threshold, and the GPU resources called by the created computing node will be overloaded when responding to the business request, then any one of the one or more replica computing nodes configured for the created computing node is started.

[0097] It should be noted that when a computing node has been created and multiple replica computing nodes are configured, if the started replica computing node fails or the load data of the created computing node meets the second load threshold, other replica computing nodes can be started to switch the business request to the newly started replica computing node.

[0098] For example, assume that a computing node Com1 has been created and is configured with two replica computing nodes RCom1 and RCom2. When a business request is routed to Com1, if Com1 fails, RCom1 is started and the business request is switched from Com1 to RCom1; if RCom1 subsequently fails, RCom2 is started and the business request is switched from RCom1 to RCom2.

[0099] The technical solution of this embodiment, by starting a replica computing node when a created computing node fails or the load data of a created computing node meets a second load threshold, can ensure that when the management node schedules any created computing node, at least one of the created computing node and the replica computing nodes configured for the created computing node supports scheduling, which helps to ensure the stability and high availability of heterogeneous GPU scheduling.

[0100] In some embodiments, if a created computing node is deactivated, for example, if the created computing node does not call GPU resources within a preset time period or the created computing node calls GPU resources without load within a preset time period, the created computing node is automatically deactivated, wherein the preset time period is a pre-set time period used to control the survival status of the created computing node. Optionally, non-functional test information is sent to the created computing node based on a preset time interval, and the non-functional test result of the created computing node is obtained based on the response information of the created computing node; if the non-functional test result of the created computing node is a test failure, a restart instruction is sent to the created computing node.

[0101] In this embodiment, the preset time interval is a preset time interval between two consecutive non-functional test information transmissions, and it is understandable that the preset time interval is less than the preset duration. The non-functional test information is information used to perform non-functional tests on the created computing nodes.

[0102] Specifically, the management node sends non-functional test information to the created computing nodes at preset time intervals. If the management node receives response information fed back by the created computing node within the preset waiting time when sending non-functional test information to any created computing node, the non-functional test result of the created computing node is determined to be a test success; if the management node does not receive response information fed back by the created computing node within the preset waiting time when sending non-functional test information to the created computing node, the non-functional test result of the created computing node is determined to be a test failure, and the created computing node can be considered to be inactivated, and the management node sends a restart instruction to the created computing node, so that the created computing node restarts in response to the restart instruction, wherein the preset waiting time is less than the preset time interval.

[0103] Exemplarily, assuming that the non-functional test is a heartbeat test, the non-functional test information is heartbeat information, and exemplary, the preset time interval is 0.2 seconds.

[0104] The technical solution of this embodiment ensures that the created computing nodes scheduled by the management node can operate normally by performing non-functional tests on the created computing nodes, which helps to ensure the stability of heterogeneous GPU scheduling.

[0105] The technical solution of this embodiment configures heterogeneous GPU configuration information and deployment scripts for the management node. The management authority of the management node is switched to other management nodes when the management node is abnormal. Different computing nodes correspond to different GPU types. When the business request and the attribute information of the created computing node meet the first creation condition of the computing node, the computing node corresponding to the GPU type corresponding to the business request is created based on the heterogeneous GPU configuration information and the deployment script; and during the operation of the created computing node, when the operating parameters of any created computing node meet the second creation condition of the computing node, a replica computing node of the created computing node is created based on the heterogeneous GPU configuration information and the deployment script, instead of directly using the GPU resources as the node scheduled by the management node, which solves the problem of incompatibility of heterogeneous GPU scheduling and ensures that different types of GPUs can work efficiently together in the same Kubernetes environment.

[0106] Embodiment 2

[0107] Figure 2 This is a flow chart of a heterogeneous GPU scheduling method provided by Embodiment 2 of the present invention. The technical solution of the embodiment of the present invention is improved on the basis of the above embodiment. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.

[0108] The method is applied to the Kubernetes system, which includes at least two management nodes. The management authority of the management node is switched to other management nodes when the management node is abnormal. The management node is configured with heterogeneous GPU configuration information and deployment scripts. The management node is used to schedule the created computing nodes. Different computing nodes correspond to different GPU types. The computing nodes call the GPU resources of the corresponding GPU type.

[0109] like Figure 2 As shown, the method includes:

[0110] S210, receiving a service request, and determining a first creation condition of a computing node based on the service request and attribute information of a created computing node; if the first creation condition of the computing node is not met, routing the service request to the created computing node.

[0111] Specifically, if the service request and the attribute information of the created computing node do not meet the first creation condition of the computing node, it can be considered that the existing computing node can meet the demand for scheduling GPU resources in response to the service request, that is, the management node can schedule the GPU resources that can efficiently respond to the service request by scheduling the created computing node, and there is no need to create a new computing node, so the service request is routed to the created computing node. Optionally, according to the service type of the service request, the service request is routed to the created computing node corresponding to the service type.

[0112] Specifically, the management node determines the business type of the business request based on the business request, and based on a preset matching algorithm, matches the business type of the business request with the GPU type of each created computing node, and obtains a matching result corresponding to each created computing node. In the case where the matching result corresponding to any created computing node is a successful match, it can be considered that the created computing node corresponds to the business type of the business request, and the business request is routed to the created computing node. Exemplarily, assuming that the business type of the business request is image processing training, and there is a GPU resource that can be scheduled by a created computing node to support image processing training and is overloaded, the management node routes the business request to the created computing node.

[0113] In some embodiments, the preset matching algorithm is label matching or taint matching. Optionally, the created computing node is provided with labels or taint information; accordingly, the service type of the service request is matched with the label or taint information of the created computing node, and if the match is successful, the service request is routed to the created computing node that has successfully matched.

[0114] In this embodiment, the label set on the created computing node is used to characterize the service type of the service request supported by the created computing node. The label values ​​of different created computing nodes may be different.

[0115] Specifically, in the case where the created computing node is set with a label, the management node determines the service type of the service request and reads the label configured for the created computing node. If the service type of the service request is the service type of the service request supported by the created computing node represented by the label, it is determined that the service request successfully matches the created computing node, and the management node routes the service request to the created computing node that successfully matches.

[0116] Exemplarily, assuming that the service type of the service request corresponds to label label1, the management node routes the service request to the created computing node with label label1.

[0117] The taint information is information indicating that the created computing node does not match any service type. The taint information includes a taint key and a taint value, and each taint value corresponds to a service type.

[0118] Specifically, when the created computing node is set with taint information, the management node parses the taint value in the taint information. If the taint value corresponds to the business type of the business request, it can be determined that the created computing node fails to match the business type of the business request; if the taint value does not correspond to the business type of the business request, it can be determined that the created computing node successfully matches the business type of the business request, and the management node routes the business request to the created computing node that successfully matches.

[0119] Exemplarily, assuming that the service type of the service request corresponds to the taint value value1, the management node routes the service request to a created computing node whose taint value is not value1.

[0120] The technical solution of this embodiment sets labels or taint information for the created computing nodes; matches the business type of the business request with the label or taint information set for the created computing nodes, and when the match is successful, routes the business request to the created computing node with successful matching. The management node can quickly route the business request to the created computing node that supports the business request, and avoid routing the business request to the created computing node that does not support the business request, thereby ensuring that the business corresponding to the business request can run in the optimal computing environment, which helps to improve the efficiency and stability of heterogeneous GPU scheduling.

[0121] In some embodiments, optionally, based on a load balancing principle, the service request is routed to the created computing node.

[0122] In this embodiment, the load balancing principle is a service request routing principle for ensuring load balancing of GPU resources respectively called by multiple created computing nodes.

[0123] Specifically, the deployment script configured by the management node includes a script based on the load balancing principle. The management node calls the script based on the load balancing principle in response to the service request. The management node predicts a target created computing node among multiple created computing nodes based on the script based on the load balancing principle, wherein the target created computing node is any created computing node that can ensure load balancing of multiple created computing nodes after being routed to the service request; and routes the service request to the target created computing node.

[0124] Exemplarily, the load balancing principle may be the principle of avoiding overloading of GPU resources called by any created computing node while GPU resources called by other created computing nodes are idle; or it may be the principle of preferentially routing business requests to created computing nodes with the smallest GPU resource load, which is not limited in this embodiment.

[0125] The technical solution of this embodiment ensures the maximization of GPU resource utilization efficiency and the optimization of Kubernetes system performance by routing business requests to created computing nodes according to the load balancing principle.

[0126] In some embodiments, optionally, when multiple service requests are received, the service requests are routed to the created computing nodes based on the priorities of the service requests.

[0127] In this embodiment, the priority is used to indicate the importance of a service request relative to other service requests.

[0128] Specifically, if the management node receives multiple service requests at the same time, the management node determines the priority of each received service request, and routes the service requests to the created computing nodes in descending order of priority.

[0129] Exemplarily, a priority may be set in a service request, and the management node may determine the priority of the service request by parsing the service request; a priority may not be set in a service request, and the management node may predict the service type of the service request and determine the preset priority corresponding to the service type as the priority of the service request. This embodiment does not impose any restrictions on this.

[0130] The technical solution of this embodiment routes the business requests to the created computing nodes based on the priority of the business requests when multiple business requests are received, so that business requests with high priority are routed to the created computing nodes before business requests with low priority, thereby ensuring that the Kubernetes system quickly schedules GPUs to process important business requests and meeting user expectations.

[0131] S220. During the operation of the created computing nodes, the operating parameters of each created computing node are obtained. When the operating parameters of any created computing node meet the second creation condition of the computing node, a replica computing node of the created computing node is created based on the heterogeneous GPU configuration information and the deployment script, wherein the GPU type of the replica computing node is the same as that of the created computing node.

[0132] The technical solution of this embodiment can improve the performance and availability of heterogeneous GPU scheduling while reducing the waste of computing resources by routing business requests to created computing nodes without creating computing nodes when the first creation condition of the computing nodes is not met.

[0133] Embodiment 3

[0134] Figure 3 This is a structural diagram of a heterogeneous GPU scheduling device provided in the third embodiment of the present invention. The device is configured in a Kubernetes system, and the Kubernetes system includes at least two management nodes. The management authority of the management node is switched to other management nodes when the management node is abnormal; the management node is configured with heterogeneous GPU configuration information and deployment scripts; the management node is used to schedule the created computing nodes, and different computing nodes correspond to different GPU types, and the computing nodes call the GPU resources of the corresponding GPU type.

[0135] like Figure 3 As shown, the device comprises:

[0136] The service request routing module 310 is used to receive a service request, determine the first creation condition of the computing node based on the service request and the attribute information of the created computing node; if the first creation condition of the computing node is met, determine the GPU type corresponding to the service request, create a computing node corresponding to the GPU type based on the heterogeneous GPU configuration information and the deployment script, and route the service request to the new computing node so that the new computing node calls the GPU resource of the corresponding GPU type to respond to the service request;

[0137] The replica business request routing module 320 is used to obtain the operating parameters of each created computing node during the operation of the created computing node, and when the operating parameters of any created computing node meet the second creation condition of the computing node, create a replica computing node of the created computing node based on the heterogeneous GPU configuration information and the deployment script, wherein the GPU type of the replica computing node is the same as that of the created computing node.

[0138] The technical solution of this embodiment configures heterogeneous GPU configuration information and deployment scripts for the management node. The management authority of the management node is switched to other management nodes when the management node is abnormal. Different computing nodes correspond to different GPU types. When the business request and the attribute information of the created computing node meet the first creation condition of the computing node, the computing node corresponding to the GPU type corresponding to the business request is created based on the heterogeneous GPU configuration information and the deployment script; and during the operation of the created computing node, when the operating parameters of any created computing node meet the second creation condition of the computing node, a replica computing node of the created computing node is created based on the heterogeneous GPU configuration information and the deployment script, instead of directly using the GPU resources as the node scheduled by the management node, which solves the problem of incompatibility of heterogeneous GPU scheduling and ensures that different types of GPUs can work efficiently together in the same Kubernetes environment.

[0139] Based on the above embodiment, optionally, the first creation condition includes that the created computing node does not support the business request and / or the load data of the created computing node reaches a first load threshold; accordingly, the business request routing module 310 is specifically used to match the business type of the business request with the GPU type of the created computing node, and when the business type of the business request does not match the GPU type of each created computing node, it is determined that the first creation condition is met; when the GPU type of at least one created computing node matches the business type of the business request and the load data of at least one created computing node reaches the first load threshold, it is determined that the first creation condition is met.

[0140] Based on the above embodiment, optionally, the operating parameters of the created computing node include performance indicators and load data; the second creation condition includes that the performance indicators of the created computing node are abnormal, and / or the load data reaches a second load threshold.

[0141] On the basis of the above embodiment, optionally, the deployment script includes a scheduling script for the computing node and a creation script for the computing nodes corresponding to the heterogeneous GPUs; accordingly, the business request routing module 310 or the replica business request routing module 320 includes a node creation submodule; wherein the node creation submodule is used to call the corresponding creation script based on the GPU type corresponding to the computing node to be created, create a new computing node or a replica computing node, and schedule the new computing node or the replica computing node through the scheduling script, wherein the new computing node or the replica computing node is configured with a unified scheduling interface, and the scheduling interface supports the scheduling of the scheduling script.

[0142] On the basis of the above embodiment, optionally, the created computing node is configured with one or more replica computing nodes; accordingly, the heterogeneous GPU scheduling device also includes a replica computing node startup module; wherein the replica computing node startup module is used to start the replica computing node when the created computing node fails or the load data of the created computing node meets the second load threshold; wherein multiple replica computing nodes support switching.

[0143] Based on the above embodiment, optionally, the created computing node is configured with a driver script, a computing library script and an interface adaptation script; the interface adaptation script is used to call GPU resources of the corresponding GPU type; wherein the created computing node includes a new computing node and a replica computing node.

[0144] Based on the above embodiment, optionally, the service request routing module 310 is further configured to route the service request to a created computing node when the first creation condition of the computing node is not met.

[0145] Based on the above embodiment, optionally, the service request routing module 310 is further configured to route the service request to a created computing node corresponding to the service type according to the service type of the service request.

[0146] On the basis of the above embodiment, optionally, the created computing node is set with a label or taint information; accordingly, the business request routing module 310 is also used to match the business type of the business request with the label or taint information set on the created computing node, and if the match is successful, the business request is routed to the created computing node with successful matching.

[0147] Based on the above embodiment, optionally, the service request routing module 310 is further configured to route the service request to the created computing node according to the load balancing principle; or,

[0148] In the case where multiple service requests are received, the service requests are routed to the created computing nodes based on the priorities of the service requests.

[0149] On the basis of the above embodiment, optionally, the heterogeneous GPU scheduling device also includes a non-functional test module; wherein the non-functional test module is used to send non-functional test information to the created computing node based on a preset time interval, and obtain the non-functional test result of the created computing node based on the response information of the created computing node; when the non-functional test result of the created computing node is a test failure, a restart instruction is sent to the created computing node.

[0150] The heterogeneous GPU scheduling device provided in the embodiment of the present invention can execute the heterogeneous GPU scheduling method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0151] Embodiment 4

[0152] Figure 4 1 is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0153] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0154] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0155] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a heterogeneous GPU scheduling method.

[0156] In some embodiments, the heterogeneous GPU scheduling method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the heterogeneous GPU scheduling method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the heterogeneous GPU scheduling method in any other appropriate manner (e.g., by means of firmware).

[0157] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0158] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0159] Embodiment 5

[0160] Embodiment 5 of the present invention further provides a computer-readable storage medium, the computer-readable storage medium stores computer instructions, the computer instructions are used to enable a processor to execute a heterogeneous GPU scheduling method, the method is applied to a Kubernetes system, the Kubernetes system includes at least two management nodes, the management authority of the management node is switched to other management nodes when the management node is abnormal; the management node is configured with heterogeneous GPU configuration information and a deployment script; the management node is used to schedule the created computing nodes, different computing nodes correspond to different GPU types, and the computing nodes call GPU resources corresponding to the GPU type; the method includes:

[0161] Receive a business request, and determine a first creation condition of the computing node based on the business request and attribute information of the created computing node; if the first creation condition of the computing node is met, determine the GPU type corresponding to the business request, create a computing node corresponding to the GPU type based on heterogeneous GPU configuration information and a deployment script, and route the business request to the new computing node so that the new computing node calls the GPU resources of the corresponding GPU type to respond to the business request; and, during the operation of the created computing node, obtain the operation parameters of each created computing node, and if the operation parameters of any created computing node meet the second creation condition of the computing node, create a replica computing node of the created computing node based on the heterogeneous GPU configuration information and the deployment script, wherein the replica computing node has the same GPU type as the created computing node.

[0162] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0163] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0164] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0165] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0166] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0167] Embodiment 6

[0168] Embodiment 6 of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the heterogeneous GPU scheduling method according to any embodiment of the present invention.

[0169] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A heterogeneous GPU scheduling method, characterized in that: Applied to a Kubernetes system, the Kubernetes system includes at least two management nodes, and the management authority of the management node is switched to other management nodes when the management node is abnormal; The management node is configured with heterogeneous GPU configuration information and deployment scripts; the management node is used to schedule the created computing nodes, different computing nodes correspond to different GPU types, and the computing nodes call GPU resources corresponding to the GPU type; The method comprises: Receive a service request, and determine a first creation condition of a computing node based on the service request and attribute information of a created computing node; if the first creation condition of the computing node is met, determine a GPU type corresponding to the service request, create a computing node corresponding to the GPU type based on the heterogeneous GPU configuration information and a deployment script, and route the service request to a new computing node, so that the new computing node calls a GPU resource of a corresponding GPU type to respond to the service request; And, during the operation of the created computing nodes, the operation parameters of each of the created computing nodes are obtained, and when the operation parameters of any of the created computing nodes meet the second creation condition of the computing node, a replica computing node of the created computing node is created based on the heterogeneous GPU configuration information and the deployment script, wherein the replica computing node has the same GPU type as the created computing node.

2. The method according to claim 1, characterized in that: The first creation condition includes that the created computing node does not support the service request and / or the load data of the created computing node reaches a first load threshold; Determining a first creation condition of the computing node based on the service request and the attribute information of the created computing node includes: Matching the service type of the service request with the GPU type of the created computing node, and determining that the first creation condition is satisfied when the service type of the service request does not match the GPU type of each of the created computing nodes; When the GPU type of at least one of the created computing nodes matches the service type of the service request and the load data of the at least one of the created computing nodes reaches the first load threshold, it is determined that the first creation condition is met.

3. The method according to claim 1, characterized in that The operating parameters of the created computing nodes include performance indicators and load data; The second creation condition includes that a performance indicator of the created computing node is abnormal, and / or the load data reaches a second load threshold.

4. The method according to claim 1, characterized in that: The deployment script includes a scheduling script for the computing node and a creation script for computing nodes corresponding to the heterogeneous GPUs respectively; The process of creating the new computing node or the replica computing node includes: Based on the GPU type corresponding to the computing node to be created, the corresponding creation script is called to create the new computing node or the replica computing node, and the new computing node or the replica computing node is scheduled by the scheduling script, wherein the new computing node or the replica computing node is configured with a unified scheduling interface, and the scheduling interface supports the scheduling of the scheduling script.

5. The method according to claim 1, characterized in that The created computing node is configured with one or more replica computing nodes; The method further comprises: When the created computing node fails or the load data of the created computing node meets a second load threshold, the replica computing node is started; wherein a plurality of the replica computing nodes support switching.

6. The method according to claim 1, characterized in that The created computing node is configured with a driver script, a computing library script and an interface adaptation script; the interface adaptation script is used to call GPU resources of the corresponding GPU type; The created computing nodes include the new computing nodes and the replica computing nodes.

7. The method according to claim 1, characterized in that The method further comprises: If the first creation condition of the computing node is not met, the service request is routed to the created computing node.

8. The method according to claim 7, characterized in that Routing the service request to the created computing node includes: According to the service type of the service request, the service request is routed to a created computing node corresponding to the service type.

9. The method according to claim 8, characterized in that The created computing node is provided with a label or taint information; The step of routing the service request to a created computing node corresponding to the service type according to the service type of the service request includes: The service type of the service request is matched with the label or taint information set on the created computing node, and if the match is successful, the service request is routed to the created computing node with successful matching.

10. The method according to claim 7, characterized in that Routing the service request to the created computing node includes: According to the load balancing principle, the service request is routed to the created computing node; or, In the case where multiple service requests are received, the service requests are routed to the created computing nodes based on the priorities of the service requests.

11. The method according to claim 1, characterized in that: The method further comprises: Sending non-functional test information to the created computing node based on a preset time interval, and obtaining a non-functional test result of the created computing node based on response information of the created computing node; When the non-functional test result of the created computing node is a test failure, a restart instruction is sent to the created computing node.

12. A heterogeneous GPU scheduling device, characterized in that: Configured in a Kubernetes system, the Kubernetes system includes at least two management nodes, and the management authority of the management node is switched to other management nodes when the management node is abnormal; The management node is configured with heterogeneous GPU configuration information and deployment scripts; the management node is used to schedule the created computing nodes, different computing nodes correspond to different GPU types, and the computing nodes call GPU resources corresponding to the GPU type; The device comprises: A service request routing module is used to receive a service request, determine a first creation condition of a computing node based on the service request and attribute information of a created computing node; if the first creation condition of the computing node is met, determine a GPU type corresponding to the service request, create a computing node corresponding to the GPU type based on the heterogeneous GPU configuration information and a deployment script, and route the service request to a new computing node, so that the new computing node calls a GPU resource of a corresponding GPU type to respond to the service request; A replica business request routing module is used to obtain the operating parameters of each of the created computing nodes during the operation of the created computing nodes, and when the operating parameters of any of the created computing nodes meet the second creation condition of the computing node, create a replica computing node of the created computing node based on the heterogeneous GPU configuration information and the deployment script, wherein the GPU type of the replica computing node is the same as that of the created computing node.

13. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the heterogeneous GPU scheduling method described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the heterogeneous GPU scheduling method described in any one of claims 1-11 when executed.

15. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the heterogeneous GPU scheduling method according to any one of claims 1 to 11.