Accelerator card-based processing method, device, equipment, medium and product
By using a symbolic link mechanism to dynamically update the resource topology in the supercomputing cluster, the compatibility and flexibility issues in heterogeneous accelerator card management are resolved, achieving efficient resource management and low-cost heterogeneous resource scheduling.
Patent Information
- Application Number
- CN202511810108.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-03
- Publication Date
- 2026-04-24
AI Technical Summary
Existing technologies suffer from poor compatibility, insufficient flexibility, and high maintenance costs when managing heterogeneous accelerator cards in supercomputing clusters, especially in hybrid deployments and frequent scaling-up/down scenarios where effective resource management is difficult to achieve.
By creating a global resource declaration file with symbolic links in the shared storage directory, pointing to the local resource declaration file of each computing node, resource registration and dynamic updating of the global resource topology are achieved. The symbolic link mechanism is used to realize autonomous resource detection and configuration of nodes, and dynamically adapt to heterogeneous accelerator card resources.
It improves the flexibility and efficiency of resource management, reduces the complexity and cost of resource management, supports autonomous detection and compatible scheduling of multiple heterogeneous resources, and maintains system stability and compatibility.
Smart Images
Figure CN121918984A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a processing method, apparatus, device, medium and product based on an accelerator card. Background Technology
[0002] Currently, supercomputing clusters can be managed in a centralized manner. Within a supercomputing cluster, computing nodes serve as basic computing units, providing computing power through the integration of heterogeneous accelerator cards, such as GPUs (Graphics Processing Units) and NPUs (Neural-network Processing Units). Slurm (Simple Linux Utility for Resource Management) acts as the cluster's workload manager, responsible for resource scheduling and job management.
[0003] In existing technologies, there are two ways to manage heterogeneous accelerator cards. The first is that for some types of accelerator cards (such as GPUs), Slurm uses a native plugin mechanism combined with a global configuration file to achieve automatic detection and scheduling. The second is that for some types of accelerator cards (such as NPUs), resource management is achieved through manual explicit configuration or secondary customized plugin development, combined with a global configuration file.
[0004] When using the above approach, the cluster is based on Slurm's centralized configuration management, which leads to poor compatibility, insufficient flexibility, and high maintenance costs when dealing with heterogeneous accelerator card mixed deployments and frequent scaling up and down scenarios. Summary of the Invention
[0005] In view of the above problems, a processing method, apparatus, device, medium, and product based on an accelerator card is proposed to overcome or at least partially solve the above problems, including: A processing method based on an accelerator card, the method being applied to a computing cluster having multiple computing nodes, the method comprising: In the shared storage directory, create a global resource declaration file containing symbolic links; wherein the symbolic links point to local resource declaration files located on each compute node. When it is detected that the current computing node is reading the global resource declaration file, the accelerator card resource information is obtained from the local resource declaration file located on the local computing node through the symbolic link, so that the current computing node can register resources based on the accelerator card resource information; Update the global resource topology of the computing cluster based on the registered accelerator card resource information.
[0006] Optionally, it also includes: Receive the target computation job; Based on the global resource topology, a target computing node adapted to the target computing job is determined, and the target computing job is processed through the target accelerator card in the target computing node.
[0007] Optionally, the global resource topology includes the registered accelerator card resource information among the plurality of computing nodes; determining the target computing node adapted to the target computing job based on the global resource topology includes: Determine the accelerator card resource information required for the target computing job; Based on the accelerator card resource information already registered in the global resource topology Based on the required accelerator card resource information and the registered accelerator card resource information, determine the target computing node that is compatible with the target computing job.
[0008] Optionally, the accelerator card resource information includes: The type and quantity of acceleration cards.
[0009] Optionally, the accelerator card may be of any of the following types: graphics processor, neural network processor, or machine learning processor.
[0010] Optionally, when the computing cluster is expanded, the newly added current computing node performs the following operations: The accelerator card resources of the current computing node itself are detected; Based on the detection results, a local resource declaration file is generated.
[0011] Optionally, based on the detection results, a local resource declaration file is generated, including: Based on the detection results, configuration entries are created for each type of accelerator card; wherein, the configuration entries include the type and quantity of the accelerator cards; Based on the configuration entries, a local resource declaration file is generated.
[0012] Optionally, the accelerator card resources of the current computing node itself are probed, including: An automated script is used to probe the accelerator card resources of the current computing node itself.
[0013] Optionally, before probing the accelerator card resources of the current computing node itself, the method further includes: Mount the shared storage directory.
[0014] Optionally, the current computing node reads the global resource declaration file when starting the job scheduling and resource management service.
[0015] Optionally, the shared storage directory is the storage directory where the cluster master control configuration file of the computing cluster is located.
[0016] An accelerator card-based processing device is applied to a computing cluster, the computing cluster having multiple computing nodes, the device comprising: The symbolic link creation module is used to create a global resource declaration file containing symbolic links in a shared storage directory; wherein the symbolic links point to local resource declaration files located on each compute node. The accelerator card resource information reading and registration module is used to obtain accelerator card resource information from the local resource declaration file located on the local machine of the current computing node through the symbolic link when it is detected that the current computing node is reading the global resource declaration file, so that the current computing node can register resources based on the accelerator card resource information; The global resource topology update module is used to update the global resource topology of the computing cluster based on the registered accelerator card resource information.
[0017] An electronic device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method described above.
[0018] A computer-readable storage medium on which a computer program is stored, which, when executed by a processor, implements the method described above.
[0019] A computer program product includes a computer program that, when executed by a processor, implements the method described above.
[0020] The embodiments of the present invention have the following advantages: In this embodiment of the invention, a global resource declaration file containing symbolic links is created in a shared storage directory. These symbolic links point to local resource declaration files located on each compute node. When the current compute node is detected reading the global resource declaration file, accelerator card resource information is obtained from the local resource declaration file via the symbolic links, enabling the current compute node to register resources based on the accelerator card resource information. The global resource topology of the compute cluster is updated according to the registered accelerator card resource information. This achieves the dynamic integration of accelerator card resource information from the local resource declaration file into the global resource topology of the compute cluster, thereby improving the flexibility and efficiency of resource management and reducing its complexity and cost. Attached Figure Description
[0021] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 This is a flowchart illustrating the steps of centralized configuration management for a computing cluster, provided in some embodiments of the present invention. Figure 2 This is a flowchart of a processing method based on an accelerator card provided in some embodiments of the present invention; Figure 3 This is a flowchart illustrating the steps of centralized configuration management for a computing cluster provided in some embodiments of the present invention; Figure 4 This is a flowchart of a second processing method based on an accelerator card provided in some embodiments of the present invention; Figure 5 This is a structural block diagram of a processing device based on an accelerator card provided in some embodiments of the present invention. Detailed Implementation
[0023] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0024] In related technologies, there are two technical approaches to managing heterogeneous accelerator cards. The first approach is that for some types of accelerator cards (such as GPUs), Slurm uses a native plugin mechanism combined with a global configuration file to achieve automatic detection and scheduling. The second approach is that for some types of accelerator cards (such as NPUs), resource management is achieved through manual explicit configuration or secondary customized plugin development, combined with a global configuration file.
[0025] When using the above method, the following problems may arise because the cluster is based on centralized configuration management using Slurm: 1. Centralized management only supports a single type of accelerator card. In scenarios where accelerator cards are deployed in a mixed manner (for example, a single cluster contains different models of the same type of accelerator cards, or different types of accelerator cards such as GPUs and NPUs coexist), the existing centralized configuration makes it difficult to achieve differentiated and compatible management of different types of devices.
[0026] 2. Centralized configuration lacks flexibility in scenarios with frequent scaling up and down. Adding or deleting computing nodes can lead to differences in configuration reading between nodes, posing a potential risk of inconsistent configurations and chaotic resource management.
[0027] 3. For accelerator cards (such as NPUs) that Slurm does not natively support, secondary plugin development can achieve functional expansion, but it requires additional code maintenance, and adaptation work needs to be repeated when Slurm version is upgraded, resulting in high long-term maintenance costs.
[0028] Centralized configuration management is a deployment paradigm for Slurm. It places the Slurm binary program and configuration files in a shared storage directory, which is then mounted and read by all compute nodes. By maintaining a single configuration source, it enables the unified implementation of cluster strategies and convenient software upgrades.
[0029] Based on this, this invention proposes to map the local resource declaration files of each computing node through a global symbolic link template, achieving heterogeneous resource detection (i.e., automatic detection of resources of accelerator cards with different architectures) and configuration autonomy (i.e., nodes autonomously completing configuration management), without the need for global configuration synchronization, thus eliminating the burden of centralized maintenance. Simultaneously, it satisfies the constraint that when the Slurm process starts, it must read gres.conf (the local resource declaration file) and slurm.conf (the Slurm main configuration file, i.e., the cluster main control configuration file) must be located in the same path (i.e., the system's mandatory requirement for file storage paths), preserving the convenient upgrade feature of the Slurm program in the shared storage directory.
[0030] In some examples, such as Figure 1 The embodiments of the present invention may include the following steps: S1. Decentralized configuration management: It achieves localized configuration autonomy for computing nodes through a symbolic link mechanism, avoiding the problem of centralized configuration synchronization.
[0031] S2. Node Autonomous Detection and Configuration: Automatically identifies GPU / NPU hardware types and dynamically generates node configurations, using Slurm native plugins or explicit declarations to complete resource registration.
[0032] S3. Standardized submission of heterogeneous jobs: Uses a unified job syntax to submit computational jobs in a standardized way, shielding the differences in underlying heterogeneous resources.
[0033] S4. Dynamic scheduling of heterogeneous resources: Automatically matches nodes according to job resource requirements, and ensures collaborative computing and job isolation of multi-architecture accelerator cards through the native scheduling framework.
[0034] S5. Cluster expansion: Repeat S2-S4 to expand nodes.
[0035] In the above example, the present invention can solve the problems of poor compatibility, insufficient flexibility in scaling up and down, and high cost of secondary development of source code caused by centralized configuration in related technologies by dynamically configuring, managing and scheduling decentralized heterogeneous accelerator cards based on node autonomy within a supercomputing cluster.
[0036] As examples, the application scenarios of this invention can include, but are not limited to, high-performance computing and AI (Artificial Intelligence) training scenarios where Slurm clusters manage heterogeneous accelerator cards (GPUs and NPUs). Based on relevant cluster scheduling technologies, this invention can achieve dynamic adaptation and indiscriminate scheduling of heterogeneous accelerator card resources through decentralized configuration management and node autonomous detection architecture. Specifically, it can include the following functional modules: 1. The decentralized configuration management module achieves local configuration autonomy for cluster nodes through a symbolic link mechanism. When dynamically scaling up or down, only local files are updated, avoiding the problem of centralized configuration synchronization.
[0037] 2. The node autonomy detection and configuration generation module automatically identifies the GPU / NPU hardware type and dynamically generates node configurations, using Slurm native plugins or explicit declarations to complete resource registration.
[0038] 3. The Slurm framework reuses modules based on the GRES (Generic RESources) mechanism to achieve standardized scheduling of heterogeneous accelerator cards, maintaining compatibility with community versions and deployment stability.
[0039] 4. The unified job submission and scheduling module adopts a unified and standardized job script, automatically matches nodes according to job resource requirements, and ensures collaborative computing and job isolation of multi-architecture accelerator cards through the native scheduling framework.
[0040] In the above examples, this invention achieves autonomous detection, registration, and compatible scheduling management of multiple heterogeneous resources within a single Slurm cluster through a localized autonomous mechanism of symbolic links. It supports the hybrid deployment of GPUs natively supported by Slurm and accelerator cards such as NPUs that require explicit configuration. This enhances the flexibility of cluster scaling. By reusing Slurm community source code, system stability is ensured, development costs are reduced, and standard heterogeneous computing power cluster management capabilities are provided for typical supercomputing simulation GPU acceleration services and large model inference services.
[0041] The present invention will be further described below with reference to the accompanying drawings: Reference Figure 2 The diagram illustrates a flowchart of a processing method based on an accelerator card according to some embodiments of the present invention, applied to a computing cluster having multiple computing nodes.
[0042] In some examples, a computing cluster can be a group of multiple interconnected computing nodes used to collaboratively process computing tasks. Computing nodes can be the basic hardware units (such as server nodes) within the cluster that handle computing tasks, and they can integrate heterogeneous accelerator cards (such as GPUs and NPUs) to provide computing power. Slurm can act as the management node of the computing cluster, responsible for resource management and task allocation.
[0043] Slurm can include the central management daemon slurmctld, the node daemon slurmd, and user command tools such as srun and sbatch.
[0044] Specifically, it may include the following steps: Step 201: In the shared storage directory, create a global resource declaration file containing symbolic links; wherein the symbolic links point to local resource declaration files located on each compute node.
[0045] As examples, a shared storage directory can be a network storage space that is mounted and accessed by all compute nodes within a cluster; a symbolic link is a special file type in Linux systems that serves as a path pointer to another file or directory. A global resource declaration file refers to a symbolic link file (such as gres.conf) created in a shared storage directory.
[0046] The local resource declaration file is the actual configuration file stored on the local disk of each compute node. This file may contain the accelerator card resource configuration information for that compute node.
[0047] In practical applications, Slurm's path constraints must be met (i.e., the local resource declaration file `gres.conf` must be in the same directory as the cluster master configuration file `slurm.conf`). Therefore, a global resource declaration file containing symbolic links can be created in a shared directory to satisfy the framework specifications. When each compute node reads this global resource declaration file containing symbolic links, the system kernel can transparently redirect it to the node's own local resource declaration file, thus enabling each node to load differentiated configurations while satisfying Slurm's path constraints.
[0048] For example, a global resource declaration file named gres.conf with a symbolic link is created in the shared storage directory / share / slurm / etc, and it is made to point to / etc / default / gres.conf (the local resource declaration file). When compute nodes of different architectures (such as GPU nodes or NPU nodes) read this global resource declaration file, they will load their respective local resource declaration files.
[0049] In some examples, such as Figure 3 It can initialize the cluster environment, create a shared file storage system, create a Slurm management node, and mount the Slurm core components and configuration to the / share / slurm directory (shared storage directory).
[0050] Then, the control node creates a global symbolic link template in the shared storage directory, establishes a mapping relationship between / share / slurm / etc / gres.conf (symbolic link) and the local / etc / default / gres.conf (local resource declaration file), and provides a foundation for decentralized reading by subsequent computing nodes.
[0051] In some embodiments of the present invention, the shared storage directory is the storage directory where the cluster master control configuration file of the computing cluster is located.
[0052] As examples, the cluster master configuration file can be a file that defines global properties of the computing cluster, such as slurm.conf in Slurm. This cluster master configuration file can include information such as a list of nodes and partition configurations.
[0053] Step 202: When it is detected that the current computing node is reading the global resource declaration file, the accelerator card resource information is obtained from the local resource declaration file located on the local computing node through the symbolic link, so that the current computing node can register resources based on the accelerator card resource information.
[0054] In some embodiments of the present invention, the current computing node reads the global resource declaration file when starting the job scheduling and resource management service.
[0055] As examples, compute nodes can automatically run the slurmd service (i.e., job scheduling and resource management service) when they start up, and according to the requirements of the Slurm framework, it must read the gres.conf file (global resource declaration file) in the same directory as slurm.conf.
[0056] In practical applications, when a compute node starts the job scheduling and resource management service to read the global resource declaration file, the system can automatically redirect it to the local resource declaration file of the node based on the symbolic link, and determine the accelerator card resource information by parsing the local resource declaration file.
[0057] In some embodiments of the present invention, the accelerator card resource information includes: the type of accelerator card and the number of accelerator cards; the type of accelerator card includes any of the following: graphics processing unit (GPU), neural network processor (NPU), and machine learning unit (MLU).
[0058] In some examples, compute nodes can parse local resource declaration files to obtain information such as the type and number of accelerator cards they are equipped with, and report this information to the cluster management node for resource registration in the format specified by the Slurm framework. After receiving the accelerator card resource information reported by each compute node, the management node can integrate this information and update the global resource topology of the compute cluster.
[0059] Step 203: Update the global resource topology of the computing cluster based on the registered accelerator card resource information.
[0060] As some examples, the global resource topology can be a real-time resource map maintained by the cluster management node. The global resource topology can record information such as the type, quantity, and status of all available computing nodes and their accelerator card resources.
[0061] After the compute nodes complete the resource registration of the accelerator cards, the cluster management node can integrate these distributed accelerator card resources and then dynamically update the global resource topology it maintains to form a unified heterogeneous resource pool.
[0062] In some embodiments of the present invention, the method further includes: receiving a target computing job; determining a target computing node adapted to the target computing job based on the global resource topology; and processing the target computing job through a target accelerator card in the target computing node.
[0063] As examples, a target computing job can refer to a computing task submitted by a user through a command tool; a target computing node refers to a computing node selected to execute the target computing job after being matched by a control node; and a target accelerator card refers to an accelerator card that is a collection of target computing nodes.
[0064] In practical applications, users can submit computing jobs through command-line tools. The management node determines the required accelerator card resource information based on the target computing job, queries the types and quantities of accelerator cards registered by each computing node in the global resource topology, and assigns the computing job to the computing node with the corresponding accelerator card resource information so that the computing task can be executed on the matched accelerator card.
[0065] For example, if a user submits an AI training job that requires 4 GPUs, the control node queries the global resource topology and finds that compute node A has registered 8 A100 GPUs and has sufficient resources. Therefore, the job can be assigned to compute node A and the training can be performed through the 4 A100 GPUs on that compute node, thus matching the demand with the resources.
[0066] As examples, Slurm's native GRES mechanism's GPU resource management capabilities can be reused to unify GPUs and NPU accelerator cards as GRES GPU common resources for seamless scheduling, eliminating the need for users to perceive underlying differences. Both GPUs and NPUs can submit computation jobs using the unified `--gres=gpu:<number>` syntax, and jobs from different accelerator cards can be routed to the corresponding compute nodes through specified queues.
[0067] In some embodiments of the present invention, the global resource topology includes accelerator card resource information registered in the plurality of computing nodes; determining a target computing node adapted to the target computing job based on the global resource topology includes: determining the accelerator card resource information required by the target computing job; and determining a target computing node adapted to the target computing job based on the required accelerator card resource information and the registered accelerator card resource information in the global resource topology.
[0068] As examples, the required accelerator card resource information refers to the type and quantity of accelerator cards specified by the user when submitting a computing job through setting parameters; the matching target computing node refers to the computing node that meets the required accelerator card resource information requirements.
[0069] In practical applications, the control node can parse the computing job to determine the required accelerator card resource information, match and compare the accelerator card resource information with the accelerator card resource information registered by each computing node in the global resource topology, and select computing nodes with matching accelerator card resource types and sufficient quantity as target computing nodes.
[0070] In some examples, Slurm can allocate corresponding compute nodes based on the registered compute nodes and global resource topology, according to the accelerator card resource information required by the compute job, and run the job on the target accelerator card to achieve isolated invocation of heterogeneous jobs.
[0071] In some embodiments of the present invention, when the computing cluster is expanded, the newly added current computing node performs the following operations: probes the accelerator card resources of the current computing node itself; and generates a local resource declaration file based on the probe results.
[0072] In some examples, when the slurmd service of a newly added compute node starts, it can read gres.conf (the global resource declaration file) in the same directory as / share / slurm / etc / slurm.conf (the cluster master control configuration file) through the native GRES framework to complete the reading of accelerator card resource information.
[0073] In some embodiments of the present invention, before probing the accelerator card resources of the current computing node itself, the method further includes: mounting the shared storage directory.
[0074] In some embodiments of the present invention, probing the accelerator card resources of the current computing node itself includes: probing the accelerator card resources of the current computing node itself through an automated script.
[0075] As some examples, an automation script can be an executable program pre-installed on a compute node; accelerator card resource detection refers to a compute node detecting its own accelerator card resource information through an automation script.
[0076] In practical applications, when adding computing nodes to a computing cluster, the new computing node can mount a shared storage directory, and then use an automatically executed detection script to identify its own accelerator card resource information and generate a local resource declaration file based on the identification results.
[0077] In some embodiments of the present invention, generating a local resource declaration file based on the detection results includes: creating configuration entries for each type of accelerator card based on the detection results; wherein the configuration entries include the type and quantity of the accelerator cards; and generating a local resource declaration file based on the configuration entries.
[0078] As some examples, configuration entries can be resource definition lines according to the Slurm syntax specification.
[0079] In practical applications, compute nodes can generate corresponding configuration entries according to the format specified by Slurm based on the detected accelerator card resource information, and generate local resource declaration files based on the configuration entries.
[0080] For example, for GPU, an AutoDetect=nvml configuration entry is generated; for NPU, entries such as Name=gpu Type=910b File= / dev / davinci[0-7] are generated, and then the configuration entries are written to the local resource declaration file.
[0081] As some examples, such as Figure 3 When expanding a computing cluster, newly added compute nodes can mount shared storage directories and use automated scripts to detect their own heterogeneous resources (i.e., accelerator card resource information), generating localized configuration files (i.e., local resource declaration files) to achieve autonomous configuration of the compute nodes. Specifically, this can include the following steps: 1. Accelerator cards are automatically identified, and a detection script ( / usr / local / launcher / bin / detect_accelerator_card.sh) is executed when the compute node starts.
[0082] If the `nvidia-smi` command or the ` / dev / nvidia0` device file is detected, it is identified as a GPU node, and automatic detection is performed using Slurm's native plugin capabilities. If the `npu-smi` command or the ` / dev / davinci0` device file is detected, it is an NPU node, and the NPU model and quantity are obtained by parsing the `npu-smi inf`.
[0083] 2. Localized configuration file generation (i.e., generating local resource declaration files): Based on hardware detection results, the compute node automatically generates configuration entries. #Example of GPU node configuration entry: AutoDetect=nvml.
[0084] #Example of NPU node configuration entries: Name=gpu Type=910b2 File= / dev / davinci[0-7].
[0085] 3. Decentralized GRES configuration mapping activation. After the local configuration file / etc / default / gres.conf is generated, the symbolic link templates in the shared storage directory are automatically activated for mapping using the symbolic link feature, without manual intervention.
[0086] In some examples, based on the above embodiments, it can also be extended as an extension technology to the compatible and unified dynamic scheduling of a third or even more types of accelerator cards (such as AMD GPU, MLU, etc.) to further meet the diverse heterogeneous acceleration resource requirements of GPU acceleration and large model recognition in future supercomputing, making the self-developed supercomputing platform more universal.
[0087] In this embodiment of the invention, a global resource declaration file containing symbolic links is created in a shared storage directory. These symbolic links point to local resource declaration files located on each compute node. When the current compute node is detected reading the global resource declaration file, accelerator card resource information is obtained from the local resource declaration file via the symbolic links, enabling the current compute node to register resources based on the accelerator card resource information. The global resource topology of the compute cluster is updated according to the registered accelerator card resource information. This achieves the dynamic integration of accelerator card resource information from the local resource declaration file into the global resource topology of the compute cluster, thereby improving the flexibility and efficiency of resource management and reducing its complexity and cost.
[0088] Reference Figure 4 The diagram illustrates a flowchart of another processing method based on an accelerator card provided by some embodiments of the present invention, applied to a computing cluster having multiple computing nodes.
[0089] In some examples, a computing cluster can be a cluster of multiple interconnected computing nodes used to collaboratively process computing tasks. Computing nodes can be the basic hardware units (such as server nodes) within the cluster that handle computing tasks, and they can integrate heterogeneous accelerator cards (such as GPUs and NPUs) to provide computing power. Slurm can act as the management node of the computing cluster, responsible for resource management and task allocation.
[0090] Slurm can include the central management daemon slurmctld, the node daemon slurmd, and user command tools such as srun and sbatch.
[0091] Specifically, it may include the following steps: Step 401: In the shared storage directory, create a global resource declaration file containing symbolic links; wherein the symbolic links point to local resource declaration files located on each compute node.
[0092] As examples, a shared storage directory can be a network storage space that is mounted and accessed by all compute nodes within a cluster; a symbolic link is a special file type in Linux systems that serves as a path pointer to another file or directory. A global resource declaration file refers to a symbolic link file (such as gres.conf) created in a shared storage directory.
[0093] The local resource declaration file is the actual configuration file stored on the local disk of each compute node. This file may contain the accelerator card resource configuration information for that compute node.
[0094] In practical applications, Slurm's path constraints must be met (i.e., the local resource declaration file `gres.conf` must be in the same directory as the cluster master configuration file `slurm.conf`). Therefore, a global resource declaration file containing symbolic links can be created in a shared directory to satisfy the framework specifications. When each compute node reads this global resource declaration file containing symbolic links, the system kernel can transparently redirect it to the node's own local resource declaration file, thus enabling each node to load differentiated configurations while satisfying Slurm's path constraints.
[0095] For example, a global resource declaration file named gres.conf with a symbolic link is created in the shared storage directory / share / slurm / etc, and it is made to point to / etc / default / gres.conf (the local resource declaration file). When compute nodes of different architectures (such as GPU nodes or NPU nodes) read this global resource declaration file, they will load their respective local resource declaration files.
[0096] In some examples, such as Figure 3 It can initialize the cluster environment, create a shared file storage system, create a Slurm management node, and mount the Slurm core components and configuration to the / share / slurm directory (shared storage directory).
[0097] Then, the control node creates a global symbolic link template in the shared storage directory, establishes a mapping relationship between / share / slurm / etc / gres.conf (symbolic link) and the local / etc / default / gres.conf (local resource declaration file), and provides a foundation for decentralized reading by subsequent computing nodes.
[0098] In some embodiments of the present invention, the shared storage directory is the storage directory where the cluster master control configuration file of the computing cluster is located.
[0099] As examples, the cluster master configuration file can be a file that defines global properties of the computing cluster, such as slurm.conf in Slurm. This cluster master configuration file can include information such as a list of nodes and partition configurations.
[0100] Step 402: When it is detected that the current computing node is reading the global resource declaration file, the accelerator card resource information is obtained from the local resource declaration file located on the local machine of the current computing node through the symbolic link, so that the current computing node can register resources based on the accelerator card resource information.
[0101] In some embodiments of the present invention, the current computing node reads the global resource declaration file when starting the job scheduling and resource management service.
[0102] As examples, compute nodes can automatically run the slurmd service (i.e., job scheduling and resource management service) when they start up, and according to the requirements of the Slurm framework, it must read the gres.conf file (global resource declaration file) in the same directory as slurm.conf.
[0103] In practical applications, when a compute node starts the job scheduling and resource management service to read the global resource declaration file, the system can automatically redirect it to the local resource declaration file of the node based on the symbolic link, and determine the accelerator card resource information by parsing the local resource declaration file.
[0104] In some embodiments of the present invention, the accelerator card resource information includes: the type of accelerator card and the number of accelerator cards; the type of accelerator card includes any of the following: graphics processing unit (GPU), neural network processor (NPU), and machine learning processor (MLU).
[0105] In some examples, compute nodes can parse local resource declaration files to obtain information such as the type and number of accelerator cards they are equipped with, and report this information to the cluster management node for resource registration in the format specified by the Slurm framework. After receiving the accelerator card resource information reported by each compute node, the management node can integrate this information and update the global resource topology of the compute cluster.
[0106] Step 403: Update the global resource topology of the computing cluster based on the registered accelerator card resource information.
[0107] As some examples, the global resource topology can be a real-time resource map maintained by the cluster management node. The global resource topology can record information such as the type, quantity, and status of all available computing nodes and their accelerator card resources.
[0108] After the compute nodes complete the resource registration of the accelerator cards, the cluster management node can integrate these distributed accelerator card resources and then dynamically update the global resource topology it maintains to form a unified heterogeneous resource pool.
[0109] Step 404: Receive the target computation job.
[0110] As some examples, a target computation job can refer to a computation task submitted by a user through a command tool.
[0111] Step 405: Based on the global resource topology, determine the target computing node that is compatible with the target computing job, and process the target computing job through the target accelerator card in the target computing node.
[0112] As examples, a target compute node refers to a compute node selected to execute the target compute job after being matched with a control node; a target accelerator card refers to an accelerator card that is a collection of target compute nodes.
[0113] In practical applications, users can submit computing jobs through command-line tools. The management node determines the required accelerator card resource information based on the target computing job, queries the types and quantities of accelerator cards registered by each computing node in the global resource topology, and assigns the computing job to the computing node with the corresponding accelerator card resource information so that the computing task can be executed on the matched accelerator card.
[0114] For example, if a user submits an AI training job that requires 4 GPUs, the control node queries the global resource topology and finds that compute node A has registered 8 A100 GPUs and has sufficient resources. Therefore, the job can be assigned to compute node A and the training can be performed through the 4 A100 GPUs on that compute node, thus matching the demand with the resources.
[0115] As examples, Slurm's native GRES mechanism's GPU resource management capabilities can be reused to unify GPUs and NPU accelerator cards as GRES GPU common resources for seamless scheduling, eliminating the need for users to perceive underlying differences. Both GPUs and NPUs can submit computation jobs using the unified `--gres=gpu:<number>` syntax, and jobs from different accelerator cards can be routed to the corresponding compute nodes through specified queues.
[0116] In some embodiments of the present invention, the global resource topology includes registered accelerator card resource information among the plurality of computing nodes; determining a target computing node adapted to the target computing job based on the global resource topology includes: determining the accelerator card resource information required by the target computing job; and determining a target computing node adapted to the target computing job based on the required accelerator card resource information and the registered accelerator card resource information in the global resource topology.
[0117] As examples, the required accelerator card resource information refers to the type and quantity of accelerator cards specified by the user when submitting a computing job through setting parameters; the matching target computing node refers to the computing node that meets the required accelerator card resource information requirements.
[0118] In practical applications, the control node can parse the computing job to determine the required accelerator card resource information, match and compare the accelerator card resource information with the accelerator card resource information registered by each computing node in the global resource topology, and select computing nodes with matching accelerator card resource types and sufficient quantity as target computing nodes.
[0119] In some examples, Slurm can allocate corresponding compute nodes based on the registered compute nodes and global resource topology, according to the accelerator card resource information required by the compute job, and run the job on the target accelerator card to achieve isolated invocation of heterogeneous jobs.
[0120] In some embodiments of the present invention, when the computing cluster is expanded, the newly added current computing node performs the following operations: probes the accelerator card resources of the current computing node itself; and generates a local resource declaration file based on the probe results.
[0121] In some examples, when the slurmd service of a newly added compute node starts, it can read gres.conf (the global resource declaration file) in the same directory as / share / slurm / etc / slurm.conf (the cluster master control configuration file) through the native GRES framework to complete the reading of accelerator card resource information.
[0122] In some embodiments of the present invention, before probing the accelerator card resources of the current computing node itself, the method further includes: mounting the shared storage directory.
[0123] In some embodiments of the present invention, probing the accelerator card resources of the current computing node itself includes: probing the accelerator card resources of the current computing node itself through an automated script.
[0124] As some examples, an automation script can be an executable program pre-installed on a compute node; accelerator card resource detection refers to a compute node detecting its own accelerator card resource information through an automation script.
[0125] In practical applications, when adding computing nodes to a computing cluster, the new computing node can mount a shared storage directory, and then use an automatically executed detection script to identify its own accelerator card resource information and generate a local resource declaration file based on the identification results.
[0126] In some embodiments of the present invention, generating a local resource declaration file based on the detection results includes: creating configuration entries for each type of accelerator card based on the detection results; wherein the configuration entries include the type and quantity of the accelerator cards; and generating a local resource declaration file based on the configuration entries.
[0127] As some examples, configuration entries can be resource definition lines according to the Slurm syntax specification.
[0128] In practical applications, compute nodes can generate corresponding configuration entries according to the format specified by Slurm based on the detected accelerator card resource information, and generate local resource declaration files based on the configuration entries.
[0129] For example, for GPU, an AutoDetect=nvml configuration entry is generated; for NPU, entries such as Name=gpu Type=910b File= / dev / davinci[0-7] are generated, and then the configuration entries are written to the local resource declaration file.
[0130] As some examples, such as Figure 3 When expanding a computing cluster, newly added compute nodes can mount shared storage directories and use automated scripts to detect their own heterogeneous resources (i.e., accelerator card resource information), generating localized configuration files (i.e., local resource declaration files) to achieve autonomous configuration of the compute nodes. Specifically, this can include the following steps: 1. Accelerator cards are automatically identified, and a detection script ( / usr / local / launcher / bin / detect_accelerator_card.sh) is executed when the compute node starts.
[0131] If the `nvidia-smi` command or the ` / dev / nvidia0` device file is detected, it is identified as a GPU node, and automatic detection is performed using Slurm's native plugin capabilities. If the `npu-smi` command or the ` / dev / davinci0` device file is detected, it is an NPU node, and the NPU model and quantity are obtained by parsing the `npu-smi inf`.
[0132] 2. Localized configuration file generation (i.e., generating local resource declaration files): Based on hardware detection results, the compute node automatically generates configuration entries. #Example of GPU node configuration entry: AutoDetect=nvml.
[0133] #Example of NPU node configuration entries: Name=gpu Type=910b2 File= / dev / davinci[0-7].
[0134] 3. Decentralized GRES configuration mapping activation. After the local configuration file / etc / default / gres.conf is generated, the symbolic link templates in the shared storage directory are automatically activated for mapping using the symbolic link feature, without manual intervention.
[0135] In some examples, based on the above embodiments, it can also be extended as an extension technology to the compatible and unified dynamic scheduling of a third or even more types of accelerator cards (such as AMD GPU, MLU, etc.) to further meet the diverse heterogeneous acceleration resource requirements of GPU acceleration and large model recognition in future supercomputing, making the self-developed supercomputing platform more universal.
[0136] In this embodiment of the invention, a global resource declaration file containing symbolic links is created in a shared storage directory. These symbolic links point to local resource declaration files located on each compute node. When the current compute node is detected reading the global resource declaration file, accelerator card resource information is obtained from the local resource declaration file via the symbolic links, enabling the current compute node to register resources based on the accelerator card resource information. The global resource topology of the compute cluster is updated according to the registered accelerator card resource information. This achieves the dynamic integration of accelerator card resource information from the local resource declaration file into the global resource topology of the compute cluster, thereby improving the flexibility and efficiency of resource management and reducing its complexity and cost.
[0137] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.
[0138] Reference Figure 5The diagram shows a schematic of the structure of a processing device based on an accelerator card provided in some embodiments of the present invention, which is applied to a computing cluster having multiple computing nodes.
[0139] Specifically, it can include the following modules: The symbolic link creation module 501 is used to create a global resource declaration file containing symbolic links in a shared storage directory; wherein the symbolic links point to local resource declaration files located on each computing node. The accelerator card resource information reading and registration module 502 is used to obtain accelerator card resource information from the local resource declaration file located on the local machine of the current computing node through the symbolic link when it is detected that the current computing node is reading the global resource declaration file, so that the current computing node can register resources based on the accelerator card resource information; The global resource topology update module 503 is used to update the global resource topology of the computing cluster based on the registered accelerator card resource information.
[0140] In some embodiments of the present invention, the apparatus further includes: The target computation job receiving module is used to receive target computation jobs; The target computing node determination module is used to determine the target computing node that is compatible with the target computing job based on the global resource topology, and to process the target computing job through the target accelerator card in the target computing node.
[0141] In some embodiments of the present invention, the global resource topology includes registered accelerator card resource information among the plurality of computing nodes; the target computing node determination module is used for: Determine the accelerator card resource information required for the target computing job; Based on the accelerator card resource information already registered in the global resource topology Based on the required accelerator card resource information and the registered accelerator card resource information, determine the target computing node that is compatible with the target computing job.
[0142] In some embodiments of the present invention, the accelerator card resource information includes: the type of accelerator card and the number of accelerator cards.
[0143] In some embodiments of the present invention, the accelerator card includes any of the following types: graphics processor, neural network processor, and machine learning processor.
[0144] In some embodiments of the present invention, when the computing cluster is expanded, the newly added current computing node performs the following operations: The accelerator card resources of the current computing node itself are detected; Based on the detection results, a local resource declaration file is generated.
[0145] In some embodiments of the present invention, a local resource declaration file is generated based on the detection results, including: Based on the detection results, configuration entries are created for each type of accelerator card; wherein, the configuration entries include the type and quantity of the accelerator cards; Based on the configuration entries, a local resource declaration file is generated.
[0146] In some embodiments of the present invention, the accelerator card resources of the current computing node itself are probed, including: An automated script is used to probe the accelerator card resources of the current computing node itself.
[0147] In some embodiments of the present invention, before probing the accelerator card resources of the current computing node itself, the method further includes: Mount the shared storage directory.
[0148] In some embodiments of the present invention, the current computing node reads the global resource declaration file when starting the job scheduling and resource management service.
[0149] In some embodiments of the present invention, the shared storage directory is the storage directory where the cluster master control configuration file of the computing cluster is located.
[0150] In this embodiment of the invention, a global resource declaration file containing symbolic links is created in a shared storage directory. These symbolic links point to local resource declaration files located on each compute node. When the current compute node is detected reading the global resource declaration file, accelerator card resource information is obtained from the local resource declaration file via the symbolic links, enabling the current compute node to register resources based on the accelerator card resource information. The global resource topology of the compute cluster is updated according to the registered accelerator card resource information. This achieves the dynamic integration of accelerator card resource information from the local resource declaration file into the global resource topology of the compute cluster, thereby improving the flexibility and efficiency of resource management and reducing its complexity and cost.
[0151] Some embodiments of the present invention also provide an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method described above.
[0152] Some embodiments of the present invention also provide a computer-readable storage medium on which a computer program is stored, and which, when executed by a processor, implements the method described above.
[0153] Some embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the method described above.
[0154] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.
[0155] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0156] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0157] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0158] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0159] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0161] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.
[0162] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the aforementioned element.
[0163] The above provides a detailed description of a processing method, apparatus, device, medium, and product based on an accelerator card. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A processing method based on an accelerator card, characterized in that, The method is applied to a computing cluster, which has multiple computing nodes. The method includes: In the shared storage directory, create a global resource declaration file containing symbolic links; wherein the symbolic links point to local resource declaration files located on each compute node. When it is detected that the current computing node is reading the global resource declaration file, the accelerator card resource information is obtained from the local resource declaration file located on the local computing node through the symbolic link, so that the current computing node can register resources based on the accelerator card resource information; Update the global resource topology of the computing cluster based on the registered accelerator card resource information.
2. The method according to claim 1, characterized in that, Also includes: Receive the target computation job; Based on the global resource topology, a target computing node adapted to the target computing job is determined, and the target computing job is processed through the target accelerator card in the target computing node.
3. The method according to claim 2, characterized in that, The global resource topology includes the registered accelerator card resource information among the plurality of computing nodes; based on the global resource topology, determining the target computing node adapted to the target computing job includes: Determine the accelerator card resource information required for the target computing job; Based on the accelerator card resource information already registered in the global resource topology Based on the required accelerator card resource information and the registered accelerator card resource information, determine the target computing node that is compatible with the target computing job.
4. The method according to claim 3, characterized in that, The accelerator card resource information includes: The type and quantity of acceleration cards.
5. The method according to claim 4, characterized in that, The accelerator card type includes any of the following: graphics processor, neural network processor, machine learning processor.
6. The method according to any one of claims 1-5, characterized in that, When the computing cluster is expanded, the newly added current computing node performs the following operations: The accelerator card resources of the current computing node itself are detected; Based on the detection results, a local resource declaration file is generated.
7. The method according to claim 6, characterized in that, Based on the detection results, a local resource declaration file is generated, including: Based on the detection results, configuration entries are created for each type of accelerator card; wherein, the configuration entries include the type and quantity of the accelerator cards; Based on the configuration entries, a local resource declaration file is generated.
8. The method according to claim 6, characterized in that, The accelerator card resources of the current computing node itself are probed, including: An automated script is used to probe the accelerator card resources of the current computing node itself.
9. The method according to claim 6, characterized in that, Before probing the accelerator card resources of the current computing node itself, the process also includes: Mount the shared storage directory.
10. The method according to claim 1, characterized in that, The current computing node reads the global resource declaration file when starting the job scheduling and resource management service.
11. The method according to claim 1, characterized in that, The shared storage directory is the storage directory where the cluster master control configuration file of the computing cluster is located.
12. A processing device based on an accelerator card, characterized in that, The device is applied to a computing cluster, the computing cluster having multiple computing nodes, the device comprising: The symbolic link creation module is used to create a global resource declaration file containing symbolic links in a shared storage directory; wherein the symbolic links point to local resource declaration files located on each compute node. The accelerator card resource information reading and registration module is used to obtain accelerator card resource information from the local resource declaration file located on the local machine of the current computing node through the symbolic link when it is detected that the current computing node is reading the global resource declaration file, so that the current computing node can register resources based on the accelerator card resource information; The global resource topology update module is used to update the global resource topology of the computing cluster based on the registered accelerator card resource information.
13. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1 to 11.
14. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method as described in any one of claims 1 to 11.
15. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 11.