Task scheduling method and system of heterogeneous accelerator card cluster, computer device and readable storage medium

Abstract

The application relates to a task scheduling method and system of a heterogeneous accelerator card cluster, a computer device and a readable storage medium. The method comprises the following steps: extracting environment requirement information in a computing task; obtaining state information of each computing node in the heterogeneous accelerator card cluster, and constructing an environment capability graph corresponding to each computing node based on the state information; performing compatibility judgment according to the environment requirement information and the environment capability graph of each node, and determining a computing node meeting a preset compatibility condition as a candidate node; obtaining resource state information of each candidate node, and determining a target node from each candidate node based on the information; determining a target container image matched with the target node based on the environment capability graph of the target node; scheduling the computing task to the target node, and creating a running instance on the target node based on the target container image, so as to run the computing task. The scheme can realize adaptive matching of the computing task and the node running environment in the heterogeneous accelerator card cluster.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a task scheduling method, system, computer device, and readable storage medium for a heterogeneous accelerator card cluster. Background Technology

[0002] With the development of computer technology, heterogeneous accelerator card clusters are increasingly being applied to scenarios such as model training, model inference, and high-performance computing. A heterogeneous accelerator card cluster can include accelerator cards from different vendors. Due to differences in driver interfaces, deep learning frameworks, and other aspects used by different accelerator cards, the runtime environment requirements for the same computing task vary across different nodes.

[0003] In existing technologies, container orchestration platforms primarily schedule computing tasks to nodes with corresponding resources based on resource request information such as processors and memory. However, because they do not consider the runtime environment, situations frequently arise where scheduling succeeds but task execution fails. Some scheduling schemes allow users to declare environment requirements through annotations or preset task templates, but this relies on users understanding the underlying hardware differences and manually configuring environment requirements, increasing the barrier to entry.

[0004] There is currently no effective solution for how to achieve adaptive matching between computing tasks and node operating environments in heterogeneous accelerator card clusters. Summary of the Invention

[0005] Therefore, it is necessary to provide a task scheduling method, system, computer device, and readable storage medium for heterogeneous accelerator card clusters that can achieve adaptive matching between computing tasks and node operating environments in heterogeneous accelerator card clusters, addressing the aforementioned technical problems.

[0006] Firstly, this application provides a task scheduling method for a heterogeneous accelerator card cluster, including:

[0007] Extract the environmental requirements information from the computational task to be executed;

[0008] Obtain the status information of each computing node in the heterogeneous accelerator card cluster, and construct an environmental capability map corresponding to each computing node based on the status information;

[0009] Compatibility is determined based on the environmental requirements information and the environmental capability map of each node, and the computing nodes that meet the preset compatibility conditions are identified as candidate nodes.

[0010] Obtain the resource status information of each candidate node, and determine the target node from each candidate node based on the resource status information;

[0011] Based on the environmental capability map of the target node, determine the target container image that matches the target node;

[0012] The computing task is scheduled to the target node, and a running instance is created on the target node based on the target container image to run the computing task.

[0013] In one embodiment, the environmental capability map includes compatibility layer information;

[0014] The step of performing compatibility judgment based on the environmental demand information and the environmental capability map of each node, and determining the computing nodes that meet the preset compatibility conditions as candidate nodes includes:

[0015] For any of the aforementioned computing nodes, based on the computing node's environmental capability map, it is determined whether the computing node can directly match the environmental requirement information without enabling the compatibility layer. If so, the computing node is identified as a native compatibility node and is designated as a candidate node. If not, based on the computing node's compatibility layer information, it is determined whether the computing node can support the operation of the computing task through the compatibility layer or in a degraded manner. If so, the computing node is identified as a compatibility layer node, and computing nodes that can support the operation of the computing task through the compatibility layer are designated as candidate nodes.

[0016] In one embodiment, the environmental requirement information includes the deep learning framework version; the environmental capability graph includes the accelerator card driver version of each node; and the compatibility layer nodes include compatibility layer type nodes and conditional compatibility nodes.

[0017] The determination of whether a computing node can support the execution of the computing task through the compatibility layer or in a degraded manner, based on the compatibility layer information of the computing node, includes:

[0018] Determine whether the deep learning framework version of the computing task and the accelerator card driver version of the computing node meet the preset version compatibility relationship;

[0019] Determine whether the compatibility layer version of the computing node meets the minimum compatibility layer version requirement corresponding to the computing task;

[0020] Under the condition that the preset version compatibility relationship is met and the minimum compatibility layer version requirement is met, determine the coverage of the compatibility layer of the computing node on the application interface called by the computing task;

[0021] When the compatibility layer of the computing node completely covers the application programming interface called in the computing task, it is determined that the computing node can support the operation of the computing task through the compatibility layer, the computing node is identified as the compatibility layer type node, and the computing node is identified as a candidate node;

[0022] When the coverage is greater than a first preset threshold and the compatibility layer of the computing node has an uncovered application programming interface, it is determined that the computing node can support the operation of the computing task in a degraded manner through the compatibility layer, and the computing node is determined as the conditionally compatible node.

[0023] In one embodiment, determining the target container image matching the target node based on the target node's environmental capability map includes:

[0024] Obtain the logical image identifier corresponding to the computing task;

[0025] Based on the logical mirror identifier and the environmental capability map of the target node, a pre-stored mirror variant mapping table is queried to obtain the target physical mirror identifier corresponding to the target node; wherein, the pre-stored mirror variant mapping table records the physical mirror identifier corresponding to the logical mirror identifier under different environmental capability maps.

[0026] The container image corresponding to the target physical image identifier is identified as the target container image.

[0027] In one embodiment, querying a pre-stored image variant mapping table to obtain the target physical image identifier corresponding to the target node includes:

[0028] When the target node is a natively compatible node, based on the logical image identifier and the environmental capability map of the target node, the image variant mapping table is queried to obtain the native physical image identifier corresponding to the native image variant, and the native physical image identifier is determined as the target physical image identifier;

[0029] When the target node is a compatibility layer node, based on the logical image identifier and the environmental capability map of the target node, the image variant mapping table is queried to obtain the compatibility layer physical image identifier corresponding to the compatibility layer image variant containing the corresponding compatibility layer component, and the compatibility layer physical image identifier is determined as the target physical image identifier.

[0030] In one embodiment, obtaining the resource status information of each candidate node and determining the target node from the candidate nodes based on the resource status information includes:

[0031] Obtain the resource status information of each native compatible node and each compatible layer type node, and determine whether there is a native compatible node that meets the resource requirements of the computing task based on the resource status information;

[0032] If it exists, the target node is determined from the natively compatible nodes that meet the resource requirements;

[0033] If not, then determine whether there is a compatible layer type node that meets the resource requirements of the computing task based on the resource status information of each of the aforementioned compatible layer type nodes;

[0034] If it exists, the target node is determined from the compatible layer type nodes that meet the resource requirements;

[0035] If it does not exist, the conditionally compatible node will be included in the selection range of the target node.

[0036] In one embodiment, the task scheduling method for the heterogeneous accelerator card cluster further includes:

[0037] Monitor node status events in a heterogeneous accelerator card cluster;

[0038] Update the environmental capability map of the corresponding computing node based on the monitored node status events.

[0039] Secondly, this application also provides a task scheduling system for a heterogeneous accelerator card cluster, the system comprising:

[0040] The task parsing module is used to extract environmental requirement information from the computing tasks to be executed.

[0041] The map construction module is used to obtain the status information of each computing node in the heterogeneous accelerator card cluster, and construct the environmental capability map corresponding to each computing node based on the status information.

[0042] The joint matching module is used to perform compatibility judgment based on the environmental requirement information and the environmental capability map of each node, and determine the computing nodes that meet the preset compatibility conditions as candidate nodes; obtain the resource status information of each candidate node, and determine the target node from each candidate node based on the resource status information;

[0043] The image determination module is used to determine the target container image that matches the target node based on the environmental capability map of the target node;

[0044] The scheduling and execution module is used to schedule the computing task to the target node and create a running instance on the target node based on the target container image to run the computing task.

[0045] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0046] Extract the environmental requirements information from the computational task to be executed;

[0047] Obtain the status information of each computing node in the heterogeneous accelerator card cluster, and construct an environmental capability map corresponding to each computing node based on the status information;

[0048] Compatibility is determined based on the environmental requirements information and the environmental capability map of each node, and the computing nodes that meet the preset compatibility conditions are identified as candidate nodes.

[0049] Obtain the resource status information of each candidate node, and determine the target node from each candidate node based on the resource status information;

[0050] Based on the environmental capability map of the target node, determine the target container image that matches the target node;

[0051] The computing task is scheduled to the target node, and a running instance is created on the target node based on the target container image to run the computing task.

[0052] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0053] Extract the environmental requirements information from the computational task to be executed;

[0054] Obtain the status information of each computing node in the heterogeneous accelerator card cluster, and construct an environmental capability map corresponding to each computing node based on the status information;

[0055] Compatibility is determined based on the environmental requirements information and the environmental capability map of each node, and the computing nodes that meet the preset compatibility conditions are identified as candidate nodes.

[0056] Obtain the resource status information of each candidate node, and determine the target node from each candidate node based on the resource status information;

[0057] Based on the environmental capability map of the target node, determine the target container image that matches the target node;

[0058] The computing task is scheduled to the target node, and a running instance is created on the target node based on the target container image to run the computing task.

[0059] The aforementioned task scheduling method, system, computer equipment, and readable storage medium for heterogeneous accelerator card clusters determine candidate nodes through environmental requirement information and environmental capability maps, determine target nodes through resource status information, select target container images that match the target nodes through the environmental capability maps of the target nodes, and create running instances based on the target container images to run the computing tasks. This enables adaptive matching between computing tasks and node operating environments, reduces task failures caused by incompatible node operating environments, insufficient resources, or unsuitable container images, and improves the success rate of task scheduling. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart illustrating a task scheduling method for a heterogeneous accelerator card cluster in one embodiment;

[0062] Figure 2 This is a schematic diagram of the environmental capability map construction process in an example embodiment;

[0063] Figure 3 This is a structural block diagram of a task scheduling method for a heterogeneous accelerator card cluster in an example embodiment;

[0064] Figure 4 This is a block diagram of a task scheduling system for a heterogeneous accelerator card cluster in one embodiment;

[0065] Figure 5 This is an internal structural diagram of a computer device in an example embodiment. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0067] In one embodiment, such as Figure 1 As shown, a task scheduling method for a heterogeneous accelerator card cluster is provided, the method comprising the following steps:

[0068] Step 101: Extract the environmental requirements information from the computing task to be executed.

[0069] Among them, environmental requirements information is used to characterize the operating environment required for the normal operation of computing tasks.

[0070] Optionally, the environmental requirements information includes, but is not limited to, the deep learning framework version, the deep learning framework type, the accelerator card driver interface type, and the distributed communication library type.

[0071] Optionally, the computation task to be executed includes source code files. Before extracting the environment requirement information from the computation task, static analysis is performed on the source code files included in the computation task to extract the environment requirement information from the computation task. Static analysis includes parsing dependency management configuration files in the source code files, including import statements, and / or parsing dependency declarations in requirements.txt, pyproject.toml, environment.yml, and Dockerfile files included in the computation task to automatically extract the deep learning framework type, deep learning framework version, and accelerator card driver interface type.

[0072] It is understandable that by performing static analysis on the source code files, the environmental requirements information in the computing tasks can be automatically extracted, eliminating the need for users to understand the hardware differences between different computing nodes in a heterogeneous accelerator card cluster and manually configure the corresponding information.

[0073] Step 102: Obtain the status information of each computing node in the heterogeneous accelerator card cluster, and construct the environmental capability map corresponding to each computing node based on the status information.

[0074] Among them, the environmental capability map is used to characterize the operating environment that the corresponding computing node can provide.

[0075] Optionally, the environment capability map includes, but is not limited to, the accelerator card type, accelerator card driver version, list of deployed container images, compatibility layer information, and communication library availability information for each node.

[0076] Optionally, the real-time status of each compute node in the heterogeneous accelerator card cluster can be scanned to obtain the status information of each compute node.

[0077] Step 103: Based on the environmental requirements information and the environmental capability map of each node, a compatibility judgment is made, and the computing nodes that meet the preset compatibility conditions are determined as candidate nodes.

[0078] Step 104: Obtain the resource status information of each candidate node, and determine the target node from the candidate nodes based on the resource status information.

[0079] Among them, the resource status information is used to characterize the resource situation of the corresponding candidate node that is currently available for running computing tasks.

[0080] Step 105: Based on the environmental capability map of the target node, determine the target container image that matches the target node.

[0081] Step 106: Schedule the computing task to the target node and create a running instance on the target node based on the target container image to run the computing task.

[0082] In the above-mentioned task scheduling method for heterogeneous accelerator card clusters, candidate nodes are determined through environmental requirement information and environmental capability graphs, target nodes are determined through resource status information, target container images that match the target nodes are selected through the environmental capability graphs of the target nodes, and running instances are created based on the target container images to run the computing tasks. This can achieve adaptive matching between computing tasks and node running environments, reduce task running failures caused by incompatible node running environments, insufficient resources, or unsuitable container images, and improve the running success rate after task scheduling.

[0083] In one example embodiment, the process of constructing an environmental capability map corresponding to each computing node based on state information is as follows: Figure 2 As shown. The system acquires the status information of each compute node, including information from the hardware, system, software, and compatibility layers. Specifically, the hardware layer acquires the accelerator card model, driver version, and PCIe topology information; the system layer acquires the operating system type and version, kernel version, and container runtime type; the software layer acquires PyTorch, TensorFlow, CUDA, MUSA, CANN, NCCL, MCCL, or HCCL components and their versions; and the compatibility layer acquires Torchada availability, CANN Next version, and application programming interface (API) coverage. Based on the status information acquired from the hardware, system, software, and compatibility layers, feature extraction, scanning, and labeling are performed to generate an environmental capability map for each compute node. Figure 2 The image shows the environmental capability graphs of three computing nodes: compute node A, compute node B, and compute node C. After constructing the environmental capability graphs for each compute node, the system monitors node status events in the heterogeneous accelerator card cluster and updates the corresponding compute node's environmental capability graph based on the monitored node status events.

[0084] In one embodiment, the environmental capability map includes compatibility layer information; the step of performing compatibility judgment based on the environmental requirement information and the environmental capability map of each node, and determining the computing nodes that meet the preset compatibility conditions as candidate nodes includes: for any computing node, based on the environmental capability map of the computing node, determining whether the computing node can directly match the environmental requirement information without enabling the compatibility layer; if so, determining the computing node as a native compatibility node and determining it as a candidate node; if not, based on the compatibility layer information of the computing node, determining whether the computing node can support the operation of the computing task through the compatibility layer or in a degraded manner; if so, determining the computing node as a compatibility layer node, and determining the computing nodes that can support the operation of the computing task through the compatibility layer as candidate nodes.

[0085] Optionally, the compatibility layer information includes, but is not limited to, the compatibility layer version.

[0086] It is understood that the compatibility layer is used to translate application programming interface calls of the first accelerator card type into equivalent calls of the second accelerator card type. Therefore, when a compute node cannot directly match the environmental requirements without enabling the compatibility layer, it can be determined whether the compute node can support the execution of computing tasks with the help of the compatibility layer based on the compatibility layer information of the compute node.

[0087] In this embodiment, by determining whether any computing node can directly match the environmental requirement information without enabling the compatibility layer, the identification of native compatibility nodes and compatibility layer nodes can be achieved respectively. Further judgment is made based on the compatibility layer information, and computing nodes that can support the operation of computing tasks through the compatibility layer are identified as candidate nodes, thus expanding the range of selectable nodes.

[0088] In one example implementation, the user submits a training script train.py, and the system statically parses train.py to extract environment requirement information: {Deep learning framework type: PyTorch, accelerator card driver interface type: CUDA, Distributed: Yes}.

[0089] Assume that the environment capability profile of compute node A is as follows: {Accelerator card type: NVIDIA A100, GPU: NVIDIA, Native support: CUDA 11.8, Communication library availability: NCCL available, Compatibility layer information: None}. This compute node can directly match the aforementioned environment requirements without enabling the compatibility layer, thus identifying it as a natively compatible node and designating it as a candidate node.

[0090] Assume the environment capability profile of compute node B is as follows: {Accelerator card type: Moore Threads MTT S4000, GPU: MUSA, Native support: MUSA, Communication library availability: MCCL available, Compatibility layer information: Torchada, which is the CUDA API translation layer, version 2.1, API coverage 100%}. This compute node cannot directly match the aforementioned environment requirements without enabling the compatibility layer; therefore, this compute node is identified as a compatibility layer node. Compatibility layer nodes include compatibility layer type nodes and conditional compatibility nodes. If the deep learning framework type PyTorch version is 2.1, and the PyTorch version meets the preset version compatibility relationship with the compute node's accelerator card driver version, and since Torchada version 2.1 meets the minimum requirements and has 100% API coverage, it is determined to be a compatibility layer type node.

[0091] In one embodiment, the environmental requirement information includes the deep learning framework version; the environmental capability map includes the accelerator card driver version of each node; the compatibility layer nodes include compatibility layer type nodes and conditional compatibility nodes; determining whether the computing node can support the computing task through the compatibility layer or in a degraded manner based on the compatibility layer information of the computing node includes: determining whether the deep learning framework version of the computing task and the accelerator card driver version of the computing node meet a preset version compatibility relationship; determining whether the compatibility layer version of the computing node meets the minimum compatibility layer version requirement corresponding to the computing task; and determining whether the computing node can support the computing task through the compatibility layer or in a degraded manner based on the compatibility layer information of the computing node. Under the minimum compatibility layer version requirement, determine the coverage of the compatibility layer of the computing node to the application programming interfaces (APIs) called by the computing task; when the compatibility layer of the computing node completely covers the APIs called by the computing task, determine that the computing node can support the operation of the computing task through the compatibility layer, identify the computing node as the compatibility layer type node, and identify the computing node as a candidate node; when the coverage is greater than a first preset threshold and the compatibility layer of the computing node has uncovered APIs, determine that the computing node can support the operation of the computing task in a degraded manner through the compatibility layer, and identify the computing node as the conditional compatibility node.

[0092] Optionally, when the coverage is less than a first preset threshold, it is determined that the computing node cannot support the operation of the computing task in a degraded manner through the compatibility layer, and the computing node is determined as an incompatible node.

[0093] It is understandable that the first preset threshold can be set according to the specific needs of the task.

[0094] It is understandable that the compatibility layer is not a simple software library replacement; its working principle comprises two layers: The first layer is a runtime application interface interception component. Through dynamic link library injection, during the process loading phase of the running instance, a symbol table with the same name as the native accelerator card driver is preferentially resolved to the compatibility layer. This allows for transparent interception of the application interfaces called by the computing task and the completion of virtual device mapping. The second layer involves an intermediate code recompilation component that extracts the intermediate representation of the computing kernel captured by the runtime application interface interception component. This intermediate representation can be PTX instructions. The compatibility layer compiler can then recompile this intermediate representation on the fly, generating executable instructions suitable for the accelerator card in the target node. These executable instructions are then submitted through the accelerator card driver interface of the target node, enabling the accelerator card in the target node to perform the corresponding computing operations. This two-layer mechanism is completely transparent to user code, maintaining semantic equivalence on heterogeneous hardware without requiring modification of the training script.

[0095] It is understood that when the compatibility layer supports all application programming interfaces (APIs) called by the computing task, it can be determined that the compatibility layer of the computing node completely covers the APIs called in the computing task, and that the computing node can support the operation of the computing task through the compatibility layer.

[0096] In this embodiment, by determining whether the deep learning framework version of the computing task and the accelerator card driver version of the computing node meet a preset version compatibility relationship, and by determining whether the compatibility layer version of the computing node meets the minimum compatibility layer version requirement corresponding to the computing task, computing nodes that do not meet basic compatibility conditions and computing nodes with insufficient compatibility layer capabilities can be excluded. By further determining the coverage of the application programming interfaces called by the computing task by the compatibility layer when both the preset version compatibility relationship and the minimum compatibility layer version are met, a fine-grained classification of compatibility layer nodes can be achieved.

[0097] In one example implementation, the Torchada-compatible layer in compute node B employs a two-layer translation mechanism combining runtime binary interception and intermediate representation recompilation. Taking the computation task corresponding to the training script train.py as an example, assuming the deep learning framework version used by the computation task is 2.1, the specific processing procedure of the Torchada-compatible layer is as follows.

[0098] The first layer is the runtime API interception layer. During PyTorch initialization, torchada injects the dynamic link library libtorchada.so via the LD_PRELOAD mechanism. This library exports a symbol table with the same name as the NVIDIA CUDA Runtime and is resolved by the linker before the native CUDA library during process loading. When the training script executes torch.cuda.device() or the underlying cudaSetDevice() call, the interception stub function in libtorchada.so is actually hit. The interception stub function maintains a virtual device mapping table, mapping the CUDA device identifier dev:0 to the MUSA device identifier musa:0, and calls MUSARuntime's musartSetDevice() to complete the actual device binding.

[0099] The second layer is the kernel recompilation layer. For CUDA kernels submitted via cudaLaunchKernel(), Torchada intercepts them, extracts their CUDA PTX intermediate representation, and calls the Moore Threads MUSA compiler mcc for just-in-time recompilation: first, each PTX instruction is translated into a MUSA intermediate representation, and then optimized by the MUSA backend to generate MUSA binary instructions for MTT S4000; the recompiled kernel is submitted to the accelerator card for execution via the MUSA driver interface musartLaunchKernel. For the Triton IR generated by torch.compile() introduced in PyTorch 2.1, Torchada version 2.1 additionally provides a Triton-to-MUSA path, directly reducing the Triton intermediate representation to MUSA MIR, avoiding semantic loss caused by PTX translation.

[0100] The aforementioned two-layer mechanism is transparent to PyTorch users. Statements such as import torch and torch.cuda.amp.autocast in the training script do not require any modification and can be executed semantically equivalently on the MUSA accelerator card.

[0101] In one example implementation, the user submits a training script train.py and a dependency description file requirements.txt. The system statically parses train.py and requirements.txt to extract the environment requirement information: {Deep learning framework type: PyTorch, deep learning framework version: 2.1.0, third-party kernel: flash-attn, a third-party library containing custom CUDA operators}.

[0102] Assume the capability profile of compute node D is as follows: {Accelerator card type: Moore Threads MTT S4000, GPU: MUSA, Native support: MUSA, Compatibility layer information: torchada, which is the CUDA API translation layer, version 2.0, API coverage 0%}. Although the deep learning framework version and the accelerator card driver version of this compute node meet the preset version compatibility relationship, torchada version 2.0 does not meet the minimum compatibility layer version requirement corresponding to the compute task. Furthermore, the custom CUDA operator of the third-party library flash-attn involves the underlying PTX instruction, and torchada 2.0 does not implement the MUSA recompilation path of the core kernel of this operator, resulting in 0% API coverage. Therefore, it is determined to be an incompatible node and is not included in the selection range of target nodes.

[0103] Assume the capability profile of compute node E is as follows: {Accelerator card type: Moore Threads MTT S4000, GPU: MUSA, Native support: MUSA, Compatibility layer information: Torchada, which is the CUDA API translation layer, version 2.1, API coverage 85%}. Since the deep learning framework version and the accelerator card driver version of this compute node meet the preset version compatibility relationship, Torchada version 2.1 meets the minimum compatibility layer version requirement corresponding to the computation task. In the custom CUDA operators of the third-party library flash-attn, the forward propagation kernel is functionally equivalent after being recompiled with Torchada. However, the asynchronous stream synchronization primitives that the backpropagation kernel depends on require polling to simulate waiting on MUSA hardware, resulting in a performance decrease of approximately 30%. The API coverage of 85% is greater than the first preset threshold of 80%, but the compatibility layer of this compute node has uncovered application programming interfaces. Therefore, this compute node is determined as a conditionally compatible node.

[0104] Assuming the environment capability map of compute node F is: {accelerator card type: NVIDIA A100, GPU: NVIDIA, native support: CUDA12.0}, since this compute node can directly match the environment requirement information without enabling the compatibility layer, this compute node is identified as a natively compatible node.

[0105] In one embodiment, determining the target container image matching the target node based on the target node's environmental capability map includes: obtaining the logical image identifier corresponding to the computing task; querying a pre-stored image variant mapping table based on the logical image identifier and the target node's environmental capability map to obtain the target physical image identifier corresponding to the target node; wherein, the pre-stored image variant mapping table records the physical image identifiers corresponding to the logical image identifier under different environmental capability maps; and determining the container image corresponding to the target physical image identifier as the target container image.

[0106] It is understandable that the target physical image identifier can be automatically queried and determined based on the environmental capability map of the target node, without requiring the user to explicitly specify the image version.

[0107] Optionally, the container image corresponding to the target physical image identifier can be selected from a pre-stored container image repository.

[0108] This embodiment enables the matching of the target container image with the target node's runtime environment, reducing the risk of computing tasks failing due to incompatibility between the container image and the node's runtime environment.

[0109] In one embodiment, querying the pre-stored image variant mapping table to obtain the target physical image identifier corresponding to the target node includes: when the target node is a native compatibility node, querying the image variant mapping table based on the logical image identifier and the environmental capability map of the target node to obtain the native physical image identifier corresponding to the native image variant, and determining the native physical image identifier as the target physical image identifier; when the target node is a compatibility layer node, querying the image variant mapping table based on the logical image identifier and the environmental capability map of the target node to obtain the compatibility layer physical image identifier corresponding to the compatibility layer image variant containing the corresponding compatibility layer component, and determining the compatibility layer physical image identifier as the target physical image identifier.

[0110] The image variant mapping table records the physical image identifiers corresponding to logical image identifiers under different environment capability maps. Logical image identifiers are used to characterize the image requirements corresponding to computing tasks, while physical image identifiers are used to identify container images.

[0111] In this embodiment, selecting a matching image variant based on the type of the target node can reduce the risk of incompatibility between the running instance and the target node's operating environment and improve the success rate of task scheduling.

[0112] In one embodiment, obtaining the resource status information of each candidate node and determining the target node from the candidate nodes based on the resource status information includes: obtaining the resource status information of each native compatible node and each compatible layer type node; determining, based on the resource status information, whether there is a native compatible node that meets the resource requirements of the computing task; if there is, determining the target node from the native compatible nodes that meet the resource requirements; if not, determining, based on the resource status information of each compatible layer type node, whether there is a compatible layer type node that meets the resource requirements of the computing task; if there is, determining the target node from the compatible layer type nodes that meet the resource requirements; if not, including the conditionally compatible node in the selection range of the target node.

[0113] Optionally, determining the target node from natively compatible nodes that meet the resource requirements includes: determining the target node from natively compatible nodes that meet the resource requirements based on resource sufficiency indicators; determining the target node from compatible layer type nodes that meet the resource requirements includes: determining the target node from compatible layer type nodes that meet the resource requirements based on resource sufficiency indicators.

[0114] Optionally, the resource sufficiency indicators include, but are not limited to, video memory availability, queue length, and load balancing.

[0115] Optionally, after including conditionally compatible nodes in the selection range of target nodes, a degraded operation alarm flag can be attached during scheduling execution.

[0116] This embodiment enables hierarchical selection of target nodes, improving task schedulability while ensuring compatibility.

[0117] In one example implementation, if the logical image identifier ai-training:v1 corresponds to both an NVIDIA variant and a Moore's Threads variant, where the Moore's Threads variant includes Torchada 2.1, and a target node needs to be selected between compute node F and compute node E, if both compute node F and compute node E have sufficient resources, compute node F will be selected as the target node because it meets the resource requirements of the computing task and is a natively compatible node. The native physical image identifier corresponding to the NVIDIA variant will then be selected as the target physical image identifier. If compute node F has insufficient resources but compute node E has sufficient resources, compute node E will be selected as the target node, and the compatible layer physical image identifier corresponding to the Moore's Threads variant will be selected as the target physical image identifier. Throughout this process, the user remains unaware of the underlying accelerator card type differences.

[0118] In one embodiment, the task scheduling method for the heterogeneous accelerator card cluster further includes: monitoring node status events in the heterogeneous accelerator card cluster; and updating the environmental capability map of the corresponding computing node based on the monitored node status events.

[0119] Optionally, node status events include node status change events and software version reporting events deployed on the node.

[0120] This embodiment enables dynamic updates of the environmental capability map as node status changes, supports automatic adaptation after adding new nodes or software upgrades in the cluster, and reduces the complexity of operation and maintenance.

[0121] In one example implementation, Kubernetes node status events are monitored, including node Ready state change events and DaemonSet software version reporting events deployed on the node, which update the environmental capability graph of the corresponding compute node.

[0122] In one example embodiment, a task scheduling method for a heterogeneous accelerator card cluster is provided. Figure 3 The flowchart of this method includes the following steps:

[0123] Extract the environment requirement information from the computing task to be executed, including the deep learning framework version; obtain the status information of each computing node in the heterogeneous accelerator card cluster, and construct an environment capability map corresponding to each computing node based on the status information. The environment capability map includes the accelerator card driver version and compatibility layer information of each node; for any computing node, based on the environment capability map of that computing node, determine whether the computing node can directly match the environment requirement information without enabling the compatibility layer; if so, determine the computing node as a native compatibility node and identify it as a candidate node; if not, based on the compatibility layer information, determine whether the computing node can support the operation of the computing task through the compatibility layer or in a degraded manner; if so, identify the computing node that can support the operation of the computing task through the compatibility layer as a compatibility layer type node and identify it as a candidate node; Compute nodes that support computational tasks through a compatibility layer in a degraded manner are identified as conditionally compatible nodes. Resource status information for each natively compatible node and each compatibility layer type node is obtained. Based on this information, it is determined whether a natively compatible node that meets the resource requirements of the computational task exists. If so, a target node is selected from among these natively compatible nodes. If not, the resource status information for each compatibility layer type node is used to determine whether a compatibility layer type node that meets the resource requirements of the computational task exists. If so, a target node is selected from among these compatibility layer type nodes. If not, conditionally compatible nodes are included in the selection range for target nodes. Based on the environmental capability map of the target node, a target container image matching the target node is determined. The computational task is scheduled to the target node, and a running instance is created on the target node based on the target container image to run the computational task.

[0124] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0125] Based on the same inventive concept, this application also provides a task scheduling system for heterogeneous accelerator card clusters for implementing the task scheduling method for heterogeneous accelerator card clusters described above. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the task scheduling system for heterogeneous accelerator card clusters provided below can be found in the limitations of the task scheduling method for heterogeneous accelerator card clusters described above, and will not be repeated here.

[0126] In one embodiment, such as Figure 4 As shown, a task scheduling system for a heterogeneous accelerator card cluster is provided, including:

[0127] Task parsing module 41 is used to extract environmental requirement information from the computing task to be executed;

[0128] The map construction module 42 is used to obtain the status information of each computing node in the heterogeneous accelerator card cluster, and construct the environmental capability map corresponding to each computing node based on the status information.

[0129] The joint matching module 43 is used to perform compatibility judgment based on the environmental requirement information and the environmental capability map of each node, and determine the computing nodes that meet the preset compatibility conditions as candidate nodes; obtain the resource status information of each candidate node, and determine the target node from each candidate node based on the resource status information;

[0130] The image determination module 44 is used to determine the target container image that matches the target node based on the environmental capability map of the target node;

[0131] The scheduling and execution module 45 is used to schedule the computing task to the target node and create a running instance on the target node based on the target container image to run the computing task.

[0132] Each module in the aforementioned heterogeneous accelerator card cluster task scheduling system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device, or stored in the computer device's memory as software, so that the processor can invoke and execute the corresponding operations of each module.

[0133] Furthermore, the environmental capability map includes compatibility layer information; the step of performing compatibility judgment based on the environmental requirement information and the environmental capability map of each node, and determining the computing nodes that meet the preset compatibility conditions as candidate nodes includes: for any computing node, based on the environmental capability map of the computing node, determining whether the computing node can directly match the environmental requirement information without enabling the compatibility layer; if so, the computing node is determined as a native compatibility node and is determined as a candidate node; if not, based on the compatibility layer information of the computing node, determining whether the computing node can support the operation of the computing task through the compatibility layer or in a degraded manner; if so, the computing node is determined as a compatibility layer node, and the computing nodes that can support the operation of the computing task through the compatibility layer are determined as candidate nodes.

[0134] Furthermore, the environment requirement information includes the deep learning framework version; the environment capability map includes the accelerator card driver version of each node; the compatibility layer nodes include compatibility layer type nodes and conditional compatibility nodes; the step of determining whether the computing node can support the computing task through the compatibility layer or in a degraded manner based on the compatibility layer information of the computing node includes: determining whether the deep learning framework version of the computing task and the accelerator card driver version of the computing node meet a preset version compatibility relationship; determining whether the compatibility layer version of the computing node meets the minimum compatibility layer version requirement corresponding to the computing task; and determining whether the minimum compatibility layer version of the computing node meets the minimum compatibility layer version requirement of the computing task. When a low compatibility layer version is required, the coverage of the compatibility layer of the computing node to the application programming interfaces (APIs) called by the computing task is determined. When the compatibility layer of the computing node completely covers the APIs called by the computing task, it is determined that the computing node can support the operation of the computing task through the compatibility layer, and the computing node is identified as a compatibility layer type node and a candidate node. When the coverage is greater than a first preset threshold and the compatibility layer of the computing node has uncovered APIs, it is determined that the computing node can support the operation of the computing task in a degraded manner through the compatibility layer, and the computing node is identified as a conditional compatibility node.

[0135] Further, determining the target container image matching the target node based on the target node's environmental capability map includes: obtaining the logical image identifier corresponding to the computing task; querying a pre-stored image variant mapping table based on the logical image identifier and the target node's environmental capability map to obtain the target physical image identifier corresponding to the target node; wherein, the pre-stored image variant mapping table records the physical image identifiers corresponding to the logical image identifier under different environmental capability maps; and determining the container image corresponding to the target physical image identifier as the target container image.

[0136] Further, the step of querying the pre-stored image variant mapping table to obtain the target physical image identifier corresponding to the target node includes: when the target node is a native compatibility node, based on the logical image identifier and the environmental capability map of the target node, querying the image variant mapping table to obtain the native physical image identifier corresponding to the native image variant, and determining the native physical image identifier as the target physical image identifier; when the target node is a compatibility layer node, based on the logical image identifier and the environmental capability map of the target node, querying the image variant mapping table to obtain the compatibility layer physical image identifier corresponding to the compatibility layer image variant containing the corresponding compatibility layer component, and determining the compatibility layer physical image identifier as the target physical image identifier.

[0137] Further, the step of obtaining the resource status information of each candidate node and determining the target node from the candidate nodes based on the resource status information includes: obtaining the resource status information of each native compatible node and each compatible layer type node; determining whether there is a native compatible node that meets the resource requirements of the computing task based on the resource status information; if there is, determining the target node from the native compatible nodes that meet the resource requirements; if there is no such node, determining whether there is a compatible layer type node that meets the resource requirements of the computing task based on the resource status information of each compatible layer type node; if there is, determining the target node from the compatible layer type nodes that meet the resource requirements; if there is no such node, including the conditionally compatible node in the selection range of the target node.

[0138] Furthermore, the graph construction module 42 is also used to monitor node status events in the heterogeneous accelerator card cluster; and to update the environmental capability graph of the corresponding computing node based on the monitored node status events.

[0139] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores environmental requirements information, status information of each computing node, etc. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a task scheduling method for a heterogeneous accelerator card cluster.

[0140] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0141] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0142] The process involves: extracting environment requirement information from the computation task to be executed; obtaining the status information of each computation node in the heterogeneous accelerator card cluster and constructing an environment capability map corresponding to each computation node based on the status information; performing compatibility judgment based on the environment requirement information and the environment capability map of each node, and identifying computation nodes that meet preset compatibility conditions as candidate nodes; obtaining the resource status information of each candidate node and determining a target node from the candidate nodes based on the resource status information; determining a target container image that matches the target node based on the environment capability map of the target node; scheduling the computation task to the target node and creating a running instance on the target node based on the target container image to run the computation task.

[0143] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0144] Extract the environmental requirement information from the computing task to be executed; obtain the status information of each computing node in the heterogeneous accelerator card cluster, and construct an environmental capability map corresponding to each computing node based on the status information; perform compatibility judgment based on the environmental requirement information and the environmental capability map of each node, and determine the computing nodes that meet the preset compatibility conditions as candidate nodes; obtain the resource status information of each candidate node, and determine the target node from each candidate node based on the resource status information; determine the target container image that matches the target node based on the environmental capability map of the target node; schedule the computing task to the target node, and create a running instance on the target node based on the target container image to run the computing task.

[0145] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0147] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0148] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A task scheduling method for a heterogeneous accelerator card cluster, characterized in that, include: Extract the environmental requirements information from the computational task to be executed; Obtain the status information of each computing node in the heterogeneous accelerator card cluster, and construct an environmental capability map corresponding to each computing node based on the status information; Compatibility is determined based on the environmental requirements information and the environmental capability map of each node, and the computing nodes that meet the preset compatibility conditions are identified as candidate nodes. Obtain the resource status information of each candidate node, and determine the target node from each candidate node based on the resource status information; Based on the environmental capability map of the target node, determine the target container image that matches the target node; The computing task is scheduled to the target node, and a running instance is created on the target node based on the target container image to run the computing task.

2. The method according to claim 1, characterized in that, The environmental capability map includes compatibility layer information; The step of performing compatibility judgment based on the environmental demand information and the environmental capability map of each node, and determining the computing nodes that meet the preset compatibility conditions as candidate nodes includes: For any of the aforementioned computing nodes, based on the computing node's environmental capability map, it is determined whether the computing node can directly match the environmental requirement information without enabling the compatibility layer. If so, the computing node is identified as a native compatibility node and is designated as a candidate node. If not, based on the computing node's compatibility layer information, it is determined whether the computing node can support the operation of the computing task through the compatibility layer or in a degraded manner. If so, the computing node is identified as a compatibility layer node, and computing nodes that can support the operation of the computing task through the compatibility layer are designated as candidate nodes.

3. The method according to claim 2, characterized in that, The environmental requirements information includes the deep learning framework version; the environmental capability graph includes the accelerator card driver version of each node; the compatibility layer nodes include compatibility layer type nodes and conditional compatibility nodes; The determination of whether a computing node can support the execution of the computing task through the compatibility layer or in a degraded manner based on the compatibility layer information includes: Determine whether the deep learning framework version of the computing task and the accelerator card driver version of the computing node meet the preset version compatibility relationship; Determine whether the compatibility layer version of the computing node meets the minimum compatibility layer version requirement corresponding to the computing task; Under the condition that the preset version compatibility relationship is met and the minimum compatibility layer version requirement is met, determine the coverage of the compatibility layer of the computing node on the application interface called by the computing task; When the compatibility layer of the computing node completely covers the application programming interface called in the computing task, it is determined that the computing node can support the operation of the computing task through the compatibility layer, the computing node is identified as the compatibility layer type node, and the computing node is identified as a candidate node; When the coverage is greater than a first preset threshold and the compatibility layer of the computing node has an uncovered application programming interface, it is determined that the computing node can support the operation of the computing task in a degraded manner through the compatibility layer, and the computing node is determined as the conditionally compatible node.

4. The method according to claim 2, characterized in that, The process of determining the target container image that matches the target node based on the environmental capability map of the target node includes: Obtain the logical image identifier corresponding to the computing task; Based on the logical mirror identifier and the environmental capability map of the target node, a pre-stored mirror variant mapping table is queried to obtain the target physical mirror identifier corresponding to the target node; wherein, the pre-stored mirror variant mapping table records the physical mirror identifier corresponding to the logical mirror identifier under different environmental capability maps. The container image corresponding to the target physical image identifier is identified as the target container image.

5. The method according to claim 4, characterized in that, The query of the pre-stored image variant mapping table to obtain the target physical image identifier corresponding to the target node includes: When the target node is a natively compatible node, based on the logical image identifier and the environmental capability map of the target node, the image variant mapping table is queried to obtain the native physical image identifier corresponding to the native image variant, and the native physical image identifier is determined as the target physical image identifier; When the target node is a compatibility layer node, based on the logical image identifier and the environmental capability map of the target node, the image variant mapping table is queried to obtain the compatibility layer physical image identifier corresponding to the compatibility layer image variant containing the corresponding compatibility layer component, and the compatibility layer physical image identifier is determined as the target physical image identifier.

6. The method according to claim 3, characterized in that, The step of obtaining the resource status information of each of the candidate nodes and determining the target node from the candidate nodes based on the resource status information includes: Obtain the resource status information of each native compatible node and each compatible layer type node, and determine whether there is a native compatible node that meets the resource requirements of the computing task based on the resource status information; If it exists, the target node is determined from the natively compatible nodes that meet the resource requirements; If not, then determine whether there is a compatible layer type node that meets the resource requirements of the computing task based on the resource status information of each of the aforementioned compatible layer type nodes; If it exists, the target node is determined from the compatible layer type nodes that meet the resource requirements; If it does not exist, the conditionally compatible node will be included in the selection range of the target node.

7. The method according to claim 1, characterized in that, The method further includes: Monitor node status events in a heterogeneous accelerator card cluster; Update the environmental capability map of the corresponding computing node based on the monitored node status events.

8. A task scheduling system for a heterogeneous accelerator card cluster, characterized in that, The system includes: The task parsing module is used to extract environmental requirement information from the computing tasks to be executed. The map construction module is used to obtain the status information of each computing node in the heterogeneous accelerator card cluster, and construct the environmental capability map corresponding to each computing node based on the status information. The joint matching module is used to perform compatibility judgment based on the environmental requirement information and the environmental capability map of each node, and determine the computing nodes that meet the preset compatibility conditions as candidate nodes; obtain the resource status information of each candidate node, and determine the target node from each candidate node based on the resource status information; The image determination module is used to determine the target container image that matches the target node based on the environmental capability map of the target node; The scheduling and execution module is used to schedule the computing task to the target node and create a running instance on the target node based on the target container image to run the computing task.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images