An Ascend NPU computing power scheduling plug-in and method for a Nomad cluster

Abstract

The application belongs to the technical field of distributed cluster resource scheduling, and provides an Ascend NPU computing power scheduling plug-in and method for a Nomad cluster. The plug-in runs on the computing nodes of the Nomad cluster in the form of an executable file, and includes a plug-in initial module, an Ascend driver adaptation module, an NPU identification and collection module, a task scheduling management module and a log monitoring module. The application integrates the computing power resources of the Ascend NPU into the unified scheduling management system of the Nomad cluster as standard resources that can be measured and managed, realizes the functions of automatic detection and identification of NPU resources, state collection, task scheduling, log recording and operation monitoring, completes the whole-process closed loop from the access of the Ascend NPU resources to the scheduling management of the Nomad cluster, and improves the diversity of the controllable computing power resource devices and the overall utilization efficiency and scheduling capability of the hybrid computing power Nomad cluster.

Description

Technical Field

[0001] This invention relates to the field of distributed cluster resource scheduling technology, specifically to an Ascend NPU computing power scheduling plugin and method for Nomad clusters. Background Technology

[0002] Nomad Cluster is an ultra-lightweight and highly flexible cluster management and service orchestration system that can be deployed and managed across multiple servers in multiple domains, including containers, virtual machines, native executables, and batch applications. With its simple technical architecture, flexible deployment, and efficient scheduling, it is widely used in the field of distributed clusters.

[0003] Nomad achieves the scheduling and management of AI computing hardware resources through a standardized device plugin mechanism. However, the existing Nomad hardware device plugin ecosystem has certain limitations. Currently, it only supports the scheduling and management of NVIDIA GPU resources and lacks adaptation for Huawei Ascend series NPUs. The Ascend NPU's driver protocol, resource management mechanism, and underlying communication interface are fundamentally different from NVIDIA GPUs. This results in the inability to directly reuse the NVML (Nvidia Manager Library) binding logic and configuration parameters of existing GPU plugins. Furthermore, the two are incompatible in terms of hardware attributes, data acquisition dimensions, and containerized deployment dependencies.

[0004] The aforementioned technical deficiencies directly result in the Nomad cluster being unable to detect, identify, manage, and schedule NPU device resources after nodes equipped with Ascend NPUs are connected to the cluster. When users build hybrid computing clusters, Ascend NPU resources can only be deployed and managed independently, unable to be coordinated and scheduled with heterogeneous computing resources such as CPUs and GPUs. This not only leads to idle and wasted hardware resources and low utilization, but also significantly increases the complexity of cluster operation and maintenance, restricts the flexible deployment of AI tasks across hardware platforms, and to some extent hinders the application expansion and large-scale deployment of Nomad clusters in domestic AI computing power scenarios. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an Ascend NPU computing power scheduling plugin and method for Nomad clusters, enabling accurate detection, dynamic allocation, real-time monitoring, and full lifecycle management of Ascend NPU resources in Nomad clusters. This allows Nomad clusters to collaboratively schedule Ascend NPU resources, effectively improving the resource utilization, deployment flexibility, and ease of operation and maintenance of hybrid computing power clusters.

[0006] In a first aspect, the present invention provides an Ascend NPU computing power scheduling plugin for Nomad clusters, which runs as an executable file on the computing nodes of the Nomad cluster, comprising: The plugin initialization module is used to parse and load the plugin configuration file, complete the initialization of each module in the plugin, and after the initialization is successful, submit a registration request to the Nomad client and perform verification, and establish a valid communication link with the Nomad client. The Ascend driver adapter module is used to automatically detect the Ascend NPU and complete the Ascend NPU driver initialization, establishing a low-level communication link between the plug-in and the Ascend NPU hardware. The NPU identification and acquisition module is used to collect fixed resource information of NPU devices, load the device filtering processing in the plugin configuration parameters, and then report it to the Nomad client to provide available computing resources for subsequent task scheduling. The task scheduling and management module is used to verify the allocated Ascend NPU resource tasks, generate the ASCEND_VISIBLE_DEVICES environment variable required for tasks, store the list of NPU device IDs that are allowed to be accessed in this variable, and work with the Nomad client to complete the scheduling and management of the target tasks. The logging and monitoring module is used to collect real-time operating status data of the NPU device and report it to the Nomad client, while also collecting the operating logs of each module in the plugin.

[0007] As can be seen from the above technical solution, the Ascend NPU computing power scheduling plugin for Nomad clusters provided by this invention follows the official development specifications of Nomad device plugins. The plugin acts as a server, establishing an efficient communication link with the Nomad client via the gRPC protocol. Relying on the Ascend CANN software stack interface, it interacts with the NPU device on the node via the PCIE bus for low-level hardware detection and scheduling management. This achieves automatic detection and identification of NPU resources, status acquisition, task scheduling, log recording, and operation monitoring, completing a closed-loop process from Ascend NPU resource access to Nomad cluster scheduling management. This plugin starts with the Nomad main process and maintains a long lifespan.

[0008] Optionally, the plugin initialization module is specifically used for: Implement the PluginInfo(), ConfigSchema(), and SetConfig() methods through the BasePlugin interface, including: The PluginInfo() method is used to get and return plugin information. The Nomad client matches and checks the supported version of the plugin information with its own version to ensure version compatibility. The ConfigSchema() method configures validation rules through HCL, controlling the type, value range, and default value of core configuration items; The SetConfig() method is used to decode and load plugin configuration parameters, and complete the initialization configuration of plugin activation status, list of ignored devices, and scanning collection cycle.

[0009] Optionally, the Ascend driver adapter module is further configured to: When a plugin driver initialization failure is detected, the error information is recorded through the logging and monitoring module.

[0010] Optionally, the NPU identification and acquisition module is specifically used for: The Fingerprint() method of the DevicePlugin interface is used to collect and report fixed NPU resources on the node, including: The Ascend CANN software stack interface is used to obtain a list of all Ascend NPU devices on the node. Devices that need to be ignored are filtered based on configuration items, and the obtained valid NPU device information is cached locally. After collecting the core indicator attributes, the data is converted and encapsulated into the Nomad standard format and reported to the Nomad client via the Fingerprint() method. The core indicator attributes include chip information, board-level information, HBM memory capacity, and driver version.

[0011] Optionally, the task scheduling management module is specifically used for: The `Reserve()` method of the `DevicePlugin` interface is used to implement task scheduling verification and the configuration of the environment required for task execution, including: When creating tasks with on-demand NPU computing resources in the Nomad cluster, the Nomad server uses the task scheduling management module to filter and schedule tasks to target nodes that meet the requirements based on the NPU resource information reported by Nomad clients across the entire domain. After receiving the task information, the task scheduling and management module will read the device ID in the local cache to verify the validity of the NPU device in the cluster task allocation. For invalid devices, it will return a save prompt and display detailed information. After successful verification, the system automatically generates environment variables and returns them to the Nomad client for use by the task.

[0012] Optionally, the logging and monitoring module is specifically used for: The Stats() method of the DevicePlugin interface is used to collect monitoring metrics for the NPU device, including: It runs as an independent coroutine, collecting NPU runtime data periodically with fixed configuration options. It also integrates the hwlog logging component to achieve log standardization and disk persistence management. Logs are persisted to a specified directory to fully record all events and exception information throughout the process. The system collects and transforms the monitoring metrics, encapsulates the data, and then reports it to the Nomad client via the Stats() method. When data is missing, it fills in the missing data with default values. The monitoring metrics are real-time data from the Ascend NPU, including device health status, power consumption, temperature, AI Core utilization, and memory usage.

[0013] Secondly, the present invention provides an Ascend NPU computing power scheduling method for Nomad clusters, and an Ascend NPU computing power scheduling plugin for Nomad clusters based on any implementation of the first aspect, comprising: The Nomad main server calculates the target computing node based on the task request submitted by the user with the NPU resource quota. After receiving the assigned task, the Nomad client on the target compute node calls the plugin to verify whether the task is executable. If the verification is successful, the environment configuration returned by the plugin is generated and obtained. The Nomad client executes the corresponding task based on the environment variables returned by the plugin, and completes the allocation and management of the target computing node's NPU computing power resources for the task. Once a task is completed or actively terminated, the Nomad client destroys the corresponding local task, releasing the occupied NPU computing resources.

[0014] Optionally, before performing computing power scheduling, the process also includes initializing the plugin and integrating it with the Nomad client, including: After the Nomad client starts and loads the plugin configuration file, if it detects a startup requirement for the plugin, it asynchronously starts the plugin from the default plugin directory via a coroutine using the plugin's executable file. After receiving and parsing the Nomad startup configuration parameters, the plugin verifies the plugin configuration parameters. If the parameter verification is valid, the logging and monitoring module is initialized. If the parameter verification is invalid, parameter configuration exception information is returned to the Nomad client. After the logging and monitoring module is initialized, the Ascend CANN software stack interface is called to initialize the Ascend driver adaptation module and perform a self-test. If the self-test passes, the plugin initialization module calls the PluginInfo() function to report its own data to the Nomad client and enters the ready state. If the self-test fails, the plugin initialization fails and returns detailed error information.

[0015] Optionally, after the plugin successfully initializes and enters the ready state, the method further includes monitoring the computing resources of the Ascend NPU based on the plugin, including: The NPU identification and acquisition module starts an independent coroutine based on the Fingerprint() interface, filters the specified devices by the device ID configured, and then collects the fixed resources of the node's NPU for the first time, reports the NPU resource data to the Nomad cluster and stores it in the local cache; The NPU identification and acquisition module continuously collects NPU resources based on the Fingerprint() interface at a fixed configuration period specified in the plugin configuration file. It compares the currently collected device information with the device information in the cache to determine if there has been a change in the fixed resource. If there is a change, it generates new data for reporting and updates the local device cache; if there is no change, it skips the reporting and update. The logging and monitoring module starts another independent coroutine based on the Stats() interface. It periodically collects runtime data of the Ascend NPU device in the cache according to the fingerprint_period parameter in the plugin configuration file, encapsulates it into a standard format, and then submits it to the Nomad client to report to the Nomad server for storage.

[0016] By adopting the above technical solution, this application has the following beneficial effects: This invention expands the heterogeneous hardware ecosystem of the Nomad cluster, achieving for the first time precise plug-in adaptation between Ascend NPU and Nomad cluster. It breaks through the technical limitation that Nomad cluster has long only supported the single architecture of NVIDIA GPU. It constructs a complete set of NPU device plug-ins, resource abstraction layer and scheduling interface, providing the first mature solution for domestic high-performance Ascend NPU computing devices to access Nomad cluster. It significantly enriches Nomad's hardware support system in the field of heterogeneous computing and fills a key functional gap in its ecosystem of independently controllable AI accelerator card devices.

[0017] This invention incorporates Ascend NPU computing power resources as measurable and manageable standard resources into the unified scheduling and management system of the Nomad cluster. It realizes the collaborative orchestration and fine-grained scheduling of diverse computing power resources such as CPU, GPU, and NPU, breaking down the barriers of independent deployment and isolated management of heterogeneous AI computing power resources in the Nomad cluster. Through unified computing power resource discovery, quota management, and task binding strategies, it realizes the dynamic optimization and orchestration of cross-architecture AI computing power resources, improving the overall utilization efficiency and scheduling capability of the hybrid computing power Nomad cluster. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0019] Figure 1 A schematic diagram of an Ascend NPU computing power scheduling plugin for Nomad clusters provided by an embodiment of the present invention is shown; Figure 2 The following is a flowchart illustrating a method for using the Ascend NPU computing power scheduling plugin for a Nomad cluster, as provided in an embodiment of the present invention. Figure 3 This diagram illustrates the server-side configuration in the usage method provided by an embodiment of the present invention. Figure 4 A schematic diagram of client configuration in the usage method provided in an embodiment of the present invention is shown; Figure 5 A schematic diagram of the NPU plugin configuration in the usage method provided in the embodiment of the present invention is shown; Figure 6 The flowchart illustrates a method for scheduling Ascend NPU computing power for a Nomad cluster, as provided in an embodiment of the present invention. Figure 7 This invention provides an example of an Ascend NPU plugin initialization flowchart for a Nomad cluster. Figure 8 The flowchart illustrates a method for monitoring Ascend NPU computing resources in a Nomad cluster, as provided in an embodiment of the present invention. Detailed Implementation

[0020] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of the present invention and are therefore merely examples, and should not be construed as limiting the scope of protection of the present invention.

[0021] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0022] To address the current issues in Nomad clusters, such as their inability to detect, identify, and manage Ascend NPU resources, their inability to coordinate scheduling with heterogeneous computing resources like CPUs and GPUs, low hardware resource utilization, and high cluster operation and maintenance complexity, one embodiment is proposed, such as... Figure 1 As shown, an Ascend NPU computing power scheduling plugin for Nomad clusters is provided, which runs as an executable file on Nomad cluster nodes, including: The plugin initialization module is used to parse and load the plugin configuration file, complete the initialization of each module of the plugin, and after the initialization is detected to be successful, submit a registration request to the Nomad client and perform verification, and establish a valid communication link with the Nomad client. The Ascend driver adapter module is used to automatically detect the Ascend NPU and complete the Ascend NPU driver initialization, establishing a low-level communication link between the plug-in and the NPU hardware. The NPU identification and acquisition module is used to collect fixed resource information of NPU devices, load the device filtering processing in the plugin configuration parameters, and then report it to the Nomad client to provide available computing resources for subsequent task scheduling. The task scheduling and management module is used to verify the allocated Ascend NPU resource tasks, generate the ASCEND_VISIBLE_DEVICES environment variable required for tasks, store the list of NPU device IDs that are allowed to be accessed in the variable, and work with the Nomad client to complete the scheduling and management of the target tasks. The logging and monitoring module is used to collect real-time operating status data of the Ascend NPU and report it to the Nomad client, while also collecting the operating logs of each module of the computing power scheduling plugin.

[0023] The plugin provided in this embodiment runs as an executable file on Nomad cluster nodes, relying on domestic hardware platforms such as Phytium 2000+, D2000, and Kunpeng 920S, and is compatible with multiple domestic operating systems such as Kylin, Deepin, and OpenEuler. The Nomad cluster adopts a "3+N" deployment mode (i.e., 3 management nodes and N compute nodes). The cluster consists of at least 3 management nodes to provide redundancy and avoid single points of failure. The N compute nodes serve as the carriers for task execution, providing computing resources such as CPU, GPU, NPU, and memory. In actual deployment, the minimum scale is achieved by overlapping and cross-deploying management nodes and compute nodes.

[0024] like Figure 1 As shown, the management node, acting as the Nomad server, typically employs a cluster model. A master election mechanism ensures high availability and data consistency, with the elected Nomad server being called the Nomad master server. Compute nodes, acting as Nomad clients, are the actual workload nodes in the Nomad cluster, responsible for handling job loads. Their functions primarily include message communication, task allocation and execution, task type plugins, and device plugins. Server nodes store metadata (such as task definitions, allocation status, node registration information, and cluster configuration). Client nodes send read and write requests to server nodes via RPC. The Nomad master server coroutine is used to implement functions such as task resource scheduling, constraint verification, scheduling rule matching, fault domain awareness, and rapid recovery.

[0025] Optionally, the plugin initialization module is specifically used to: implement the PluginInfo(), ConfigSchema(), and SetConfig() methods through the BasePlugin interface, including: The PluginInfo() method is used to get and return plugin information (i.e., plugin type, plugin name, and supported version). The Nomad client matches and checks the supported version of the plugin information with its own version to ensure version compatibility. The `ConfigSchema()` method uses HCL configuration validation rules to control the type, value range, and default value of core configuration items. HCL declarative configuration language is used to set the three core configuration items of the NPU plugin (i.e., plugin activation status, list of ignored devices, and scan cycle), achieving overall configuration control or loading of pre-defined constraints upon plugin startup. After the plugin configuration item parameters are decoded and loaded, parameter checks are triggered; if invalid, a formatting error message is returned. HCL is a configuration file used to define the basic implementation as code (IaC), allowing the writing of tasks, resource requirements, and control constraint rules within the Nomad cluster.

[0026] The SetConfig() method is used to decode and load plugin configuration parameters, complete the initialization configuration of plugin enable status (enable), ignored device list (ignored_npu_ids), and scanning collection period (fingerprint_period), and has a built-in parsing exception handling mechanism to ensure the validity and availability of plugin configuration.

[0027] Optionally, the Ascend driver adapter module is specifically used for: By integrating Huawei Ascend NPU's dedicated driver interface and implementing automatic driver initialization based on the Ascend CANN software stack interface specification, the low-level communication link between the plug-in and the NPU hardware can be established without specifying the driver path. The module has a built-in driver exception handling mechanism. When a plugin driver initialization failure is detected, the error information can be recorded in detail through the log and monitoring module to trace the exception and investigate the cause. It has cross-architecture and cross-system compatibility capabilities, and can adapt to both ARM64 and X86 hardware architectures, as well as multiple domestic operating systems such as Kylin, DeepIn, and OpenEuler. It implements low-level communication between the plug-in and the Ascend NPU device in accordance with the Ascend CANN software stack interface specification, providing stable and reliable hardware access support for upper-layer modules.

[0028] Optionally, the NPU identification and acquisition module is specifically used for: The Fingerprint() method of the DevicePlugin interface is used to collect and report fixed NPU resources on the node, including: The list of all Ascend NPU devices on the node is obtained through the Ascend CANN software stack interface. Devices that need to be ignored are filtered based on configuration items, and the obtained valid NPU device information is cached locally. NPU device information updates are only triggered when a device is added, removed, or its status changes, which effectively reduces redundant reporting and lowers communication overhead within the Nomad cluster. After collecting core indicator attributes, the data is converted and encapsulated into the Nomad standard format and reported to the Nomad client through the Fingerprint() method to ensure that the Nomad cluster accurately knows the node's NPU computing power resources and operating status. The core indicator attributes include chip information (type, model, version, number of AI Cores), board-level information (board ID, PCB ID, BOM ID), HBM memory capacity, and driver version.

[0029] Optionally, the task scheduling management module is specifically used for: The `Reserve()` method of the `DevicePlugin` interface is used to implement task scheduling verification and the configuration of the environment required for task execution, including: When creating tasks with on-demand NPU computing resources in the Nomad cluster, the Nomad server uses the task scheduling management module to filter and schedule tasks to target nodes that meet the requirements based on the NPU resource information reported by Nomad clients across the entire domain. After receiving the task information, the task scheduling and management module will read the device ID in the local cache to verify the validity of the NPU device in the cluster task allocation. For invalid devices, it will return a save prompt and display detailed information. After successful verification, the environment variable (ASCEND_VISIBLE_DEVICES) is automatically generated and returned to the Nomad client for the task to use, thus enabling the task to access and occupy the target NPU resource.

[0030] Optionally, the logging and monitoring module is specifically used for: The Stats() method of the DevicePlugin interface is used to collect monitoring metrics for the NPU device, including: Running as an independent coroutine, it collects NPU runtime data at a fixed configuration period (the `fingerprint_period` parameter in the plugin configuration file). It also integrates the `hwlog` logging component to achieve log standardization and disk persistence management. Logs are persisted to a specified directory (e.g., ` / var / log / nomad-device / ascend.log`) and support a 7-day rotation storage and 30 backup files configuration strategy. It comprehensively records all events and anomalies throughout the entire process, including plugin initialization, configuration loading, device identification and collection, and task scheduling, providing clear evidence for maintenance troubleshooting and log tracing. The system collects and transforms the data of the monitoring metrics, and then reports it to the Nomad client through the Stats() method. When data is missing, it fills in the missing data with default values. The monitoring metrics are real-time data of the Ascend NPU, including device health status, power consumption, temperature, AI Core utilization, and memory usage.

[0031] The following details the usage of the aforementioned computing power scheduling plugin. This usage occurs on the Nomad cluster server (where the management node has completed leader election and the cluster is in a ready state). After multiple clients (compute nodes, one of which is equipped with an NPU device) register and join the cluster, the plugin plays a role in task scheduling by receiving NPU device availability checks, resource usage status, and status reports. For example... Figure 2 As shown, the usage methods include: A1. Server-side deployment of the Nomad cluster involves installing the Nomad Server component on the management node and ensuring its high availability and data consistency through a master election mechanism to build a highly available Nomad cluster with multiple management nodes. A2. The Nomad Server component configuration file nomad.hcl must enable server mode (i.e., the enabled parameter in the server configuration item must be set to true), specifying core parameters such as the cluster communication address, Raft port, failover heartbeat TTL, and heartbeat grace period to ensure stable communication of the Nomad server cluster. Nomad server configuration (actually used parameters can be incrementally supplemented and added) is as follows: Figure 3 As shown.

[0032] The server_join parameter in the server configuration sets the pool of server IP addresses currently configured in the Nomad cluster, which is used to report status and receive assigned tasks; the bootstrap_expect parameter sets the minimum number of servers that can survive in the cluster. If the number is less than this, the cluster cannot work; setting enabled to true starts the server mode. A3. The Nomad server uses the Raft protocol to achieve the Nomad cluster server leader election mechanism and metadata state consistency. Replication (state replication) and forwarding (request forwarding) synchronize scheduling instructions and node state data. It is responsible for task scheduling, node management and metadata storage across the entire Nomad cluster.

[0033] Metadata typically includes node metadata: ID, tag, IP, port, resource information, health status, etc.; and task metadata: image, startup command, resource request, scheduling strategy, version information, etc.

[0034] A4. Deploy Nomad client nodes. Install the Nomad Client component on all compute nodes. In the configuration file nomad.hcl, disabling server mode is equivalent to enabling client mode. Specify the server IP address to allow clients to register, join, and be managed by the Nomad cluster. A Nomad client configuration example (actual usage parameters can be incrementally supplemented and added) is shown below. Figure 4 As shown.

[0035] In the client configuration, the servers parameter is set to the Nomad cluster's existing server IP address pool, which is used to report status and receive assigned tasks; setting enabled to true enables client mode. The A5.Nomad client is responsible for receiving task scheduling instructions from the Nomad server and managing the local task lifecycle. It also serves as the hosting carrier for this plugin and works with the plugin to complete the NPU computing resource management and allocation functions. A6. Hardware environment preparation for Ascend NPU: Install the Ascend NPU driver package and firmware package on the computing node equipped with Ascend NPU computing power resources, and complete the initialization of the underlying hardware on the computing node. A7. Verify and obtain device status through tools such as npu-smi to ensure that Ascend NPU devices are online and computing resources can be accessed normally, providing a hardware support foundation for subsequent resource management and scheduling.

[0036] To enable this plugin in the A8 Nomad client configuration, modify the Nomad client plugin configuration file on the compute node equipped with the Ascend NPU computing power device. Set whether to enable the plugin, configure the list of ignored devices, and the data acquisition cycle, among other parameters. Then restart the client. A complete example of the plugin configuration file is provided (actual parameters can be incrementally added). Figure 5 As shown.

[0037] When the A9.Nomad client starts, it will automatically scan for enabled plugins in the configuration, without requiring any additional startup parameters.

[0038] The plugin provided in this embodiment runs in single-instance mode and is managed by the Nomad client process. If the plugin initialization fails, the Nomad client restarts the plugin every 30 seconds (the default value of health_timeout), and the plugin returns specific error information to the client. If the plugin terminates abnormally during operation (such as crashing or exhausting resources), the Nomad client will trigger an automatic restart mechanism to release resources and restart the plugin, ensuring the continuity of NPU device management services.

[0039] Based on the Ascend NPU computing power scheduling plugin for the Nomad cluster, the computing power resources of Ascend NPU are incorporated into the unified scheduling and management system of the Nomad cluster as measurable and manageable standard resources. This supports the collaborative orchestration and fine-grained quota management of diversified computing power resources such as CPU, GPU, and NPU, and enables dynamic optimization and allocation of cross-architecture AI computing power resources. This improves the diversity of manageable computing power resource devices and the overall utilization efficiency and scheduling capabilities of the hybrid computing power Nomad cluster.

[0040] In one embodiment, based on the aforementioned Ascend NPU computing power scheduling plugin, a computing power scheduling method is provided, such as... Figure 6 As shown, it includes: S110. The Nomad master server node receives a user-submitted task resource request that utilizes NPU resources, and calculates the target computing node based on the task resource request. Specifically, S110 includes: The S111.Nomad server receives a task request from a user via the command-line tool CLI or the console interface that calls the API interface, specifying the resource type, NPU model, NPU computing power, and other resource requirements. After verifying that the syntax is correct, it converts the request into an HCL format file containing the NPU resources required for the task. S112. The Nomad cluster server will forward the request to the Nomad master server (above the management node of the selected master). The server will then schedule and evaluate the request, and make node selection and scheduling decisions based on the task's NPU computing power requirements, scheduling strategies (affinity, directionality, etc.), NPU model, and other attributes. S113. The Nomad main server combines the scheduling algorithm with the NPU device information reported by all computing power scheduling plugins across the entire domain to complete node selection and task allocation. It then pushes and distributes tasks containing Ascend NPU allocation requirements to the target nodes that meet the requirements (i.e., the Nomad client) via RPC.

[0041] S120. The Nomad client on the target node receives the task assignment request, calls the plugin to verify whether the task is executable, and if the verification is successful, generates and obtains the environment configuration returned by the plugin. After receiving the task assignment, the Nomad client on the target node receives and parses the device list in the task through the Reserve() interface of this NPU device plugin, verifies the validity of the local NPU devices, and returns an error message if the verification fails, indicating that the scheduling has failed; if the verification succeeds, a configuration object is generated and returned to the Nomad client for it to configure the task's running environment.

[0042] The S130.Nomad client executes corresponding tasks based on the environment variables returned by the computing power scheduling plugin, completing the allocation and management of the target node's NPU computing power resources for the task.

[0043] The Nomad client loads the environment variables (ASCEND_VISIBLE_DEVICES) returned by the computing power scheduling plugin into the task for startup, enabling the task to access the target NPU resources. Simultaneously, it continuously monitors the task's lifecycle status. During task execution, the Stats() interface periodically collects and reports NPU runtime data for Nomad cluster monitoring, collection, and aggregation. S140. After a task is completed or actively terminated, the Nomad client destroys the corresponding local task and releases the occupied NPU computing resources. These released resources are then reintegrated into the Nomad cluster's resource pool for reuse in subsequent task scheduling after the next cycle of data collection and reporting.

[0044] Before executing the computing power scheduling method, the process also includes plugin initialization. This initialization method completes configuration verification, log component initialization, and Ascend driver adaptation module initialization during startup, followed by plugin registration and integration with the Nomad client. For example... Figure 7 As shown, the initialization process includes: After the S210.Nomad client starts and loads the plugin configuration file, if it detects that the plugin needs to be started in the plugin module, it will be started asynchronously by a coroutine from the default plugin directory (usually the plugins directory under the data directory when the Nomad client starts) through the plugin executable file. S220. After receiving and parsing the Nomad startup configuration parameters, the computing power scheduling plugin verifies the plugin configuration parameters. If the parameter validation fails, the plugin initialization will fail and return a parameter configuration error message to the Nomad client. If the parameter validation is valid, initialize the logging and monitoring module (hwlog logging component), configure the log storage path (usually / var / log / nomad-device), rolling strategy and log level (such as debug info warn error), which will be used to record logs for each module later; S230. After the logging and monitoring module completes initialization, it calls the Ascend CANN software stack interface to initialize the Ascend driver adaptation module and performs a self-test. If the self-test passes, it calls the PluginInfo() method based on the plugin initialization module to report its own data (i.e., plugin type, plugin name, and supported version) to the Nomad client and enters the ready state. If the self-test fails, the plugin initialization fails and returns detailed error information. The Nomad client restarts the plugin at fixed intervals.

[0045] In one embodiment, a computing resource monitoring method is provided. After the plugin successfully initializes and enters the ready state, two parallel periodic acquisition coroutines are used to collect the fixed resources and operational data of the Ascend NPU, respectively, to achieve computing resource monitoring and reporting. The process is as follows: Figure 8 As shown, it includes: The S310.NPU identification and acquisition module starts an independent coroutine based on the Fingerprint() interface, filters the specified devices by the device ID configured; then it collects the fixed resources of the node's NPU for the first time, reports the NPU resource data to the Nomad cluster and stores it in the local cache. The S320 NPU identification and acquisition module continuously collects NPU resources based on the Fingerprint() interface at a fixed configuration period specified in the plugin configuration file. It compares the currently collected device information with the device information in the cache using the device ID difference to determine whether a fixed resource change has occurred (such as device addition or deletion). Only when a device change occurs is new data generated and reported, and the local device cache is updated. If there is no change, the reporting and update are skipped to minimize cluster communication overhead. The S330. Log and monitoring module starts another independent coroutine based on the Stats() interface. It periodically collects runtime data of the Ascend NPU device in the cache according to the fingerprint_period parameter in the plugin configuration file, encapsulates it into a standard format, and then sends it to the Nomad client to report to the Nomad server for storage.

[0046] The above embodiments are only used to provide a detailed description of the technical solutions of this application. However, the descriptions of the above embodiments are only for the purpose of helping to understand the methods of the embodiments of the present invention and should not be construed as limiting the embodiments of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art should be covered within the protection scope of the embodiments of the present invention.

Claims

1. A Nomad cluster Ascend NPU computing power scheduling plugin, characterized in that, Running as an executable file on the compute nodes of the Nomad cluster, including: The plugin initialization module is used to parse and load the plugin configuration file, complete the initialization of each module in the plugin, and after the initialization is successful, submit a registration request to the Nomad client and perform verification, and establish a valid communication link with the Nomad client. The Ascend driver adapter module is used to automatically detect the Ascend NPU and complete the Ascend NPU driver initialization, and establish the underlying communication link between the plug-in and the Ascend NPU device. The NPU identification and acquisition module is used to collect fixed resource information of NPU devices, load the device filtering processing in the plugin configuration parameters, and report the NPU resource data to the Nomad client to provide usable computing resources for subsequent task scheduling. The task scheduling and management module is used to verify the resource tasks of the allocated Ascend NPU devices, generate the ASCEND_VISIBLE_DEVICES environment variable required for the task, store the list of NPU device IDs that are allowed to be accessed in the variable, and work with the Nomad client to complete the scheduling and management of the target task. The logging and monitoring module is used to collect real-time operating status data of the Ascend NPU device and report it to the Nomad client, while also collecting the operating logs of each module in the plugin.

2. The plugin according to claim 1, characterized in that, The plugin initialization module is specifically used for: Implemented through the PluginInfo(), ConfigSchema(), and SetConfig() methods of the BasePlugin interface: The PluginInfo() method is used to get and return plugin information. The Nomad client matches and checks the supported version of the plugin information with its own version to ensure version compatibility. The ConfigSchema() method configures validation rules through HCL, controlling the type, value range, and default value of core configuration items; core configuration items include plugin enabled status, list of ignored devices, and scan cycle; The SetConfig() method is used to decode and load plugin configuration parameters, and complete the initialization configuration of plugin activation status, list of ignored devices, and scanning collection cycle.

3. The plug-in according to claim 2, characterized in that, The Ascend driver adapter module is also used for: When a plugin driver initialization failure is detected, the error information is recorded through the logging and monitoring module.

4. The plug-in according to claim 3, characterized in that, The NPU identification and acquisition module is specifically used for: The Fingerprint() method of the DevicePlugin interface is used to collect and report fixed NPU resources on the node, including: The Ascend CANN software stack interface is used to obtain a list of all Ascend NPU devices on the node. Devices that need to be ignored are filtered based on configuration items, and the obtained valid NPU device information is cached locally. After collecting the core indicator attributes, the data is converted and encapsulated into the Nomad standard format and reported to the Nomad client via the Fingerprint() method. The core indicator attributes include chip information, board-level information, HBM memory capacity, and driver version.

5. The plug-in according to claim 4, characterized in that, The task scheduling and management module is specifically used for: The `Reserve()` method of the `DevicePlugin` interface is used to implement task scheduling verification and the configuration of the environment required for task execution, including: When creating tasks with on-demand NPU computing resources in the Nomad cluster, the Nomad server uses the task scheduling management module to filter and schedule tasks to target nodes that meet the requirements based on the NPU resource information reported by Nomad clients across the entire domain. After receiving the task information, the task scheduling and management module will read the device ID in the local cache to verify the validity of the NPU device in the cluster task allocation. For invalid devices, it will return a save prompt and display detailed information. After successful verification, the system automatically generates environment variables and returns them to the Nomad client for use by the task.

6. The plug-in according to claim 2, characterized in that, The log and monitoring module is specifically used for: The Stats() method of the DevicePlugin interface is used to collect monitoring metrics for the NPU device, including: It runs as an independent coroutine, collecting NPU runtime data periodically with fixed configuration options. It also integrates the hwlog logging component to achieve log standardization and disk persistence management. Logs are persisted to a specified directory to fully record all events and exception information throughout the process. The system collects and transforms the monitoring metrics, encapsulates the data, and then reports it to the Nomad client via the Stats() method. When data is missing, it fills in the missing data with default values. The monitoring metrics are real-time data from the Ascend NPU, including device health status, power consumption, temperature, AI Core utilization, and memory usage.

7. A method for scheduling Ascend NPU computing power in a Nomad cluster, characterized in that, Based on any one of claims 1-6, the plugin includes: The Nomad main server calculates the target computing node based on the task request submitted by the user with the NPU resource quota. After receiving the assigned task, the Nomad client on the target compute node calls the plugin to verify whether the task is executable. If the verification is successful, the environment configuration returned by the plugin is generated and obtained. The Nomad client executes the corresponding task based on the environment variables returned by the plugin, and completes the allocation and management of the target computing node's NPU computing power resources for the task. Once a task is completed or actively terminated, the Nomad client destroys the corresponding local task, releasing the occupied NPU computing resources.

8. The method according to claim 7, characterized in that, Before computing power scheduling, the process also includes initializing the plugin and integrating it with the Nomad client, including: After the Nomad client starts and loads the plugin configuration file, if it detects a startup requirement for the plugin, it asynchronously starts the plugin from the default plugin directory via a coroutine using the plugin's executable file. After receiving and parsing the Nomad startup configuration parameters, the plugin initialization module verifies the plugin configuration parameters. If the parameter verification is valid, the logging and monitoring module is initialized. If the parameter verification is invalid, parameter configuration exception information is returned to the Nomad client. After the logging and monitoring module is initialized, the Ascend CANN software stack interface is called to initialize the Ascend driver adaptation module and perform a self-test. If the self-test passes, the plugin initialization module calls the PluginInfo() method to report its own data to the Nomad client and enters the ready state. If the self-test fails, the plugin initialization fails and returns detailed error information. The Nomad client restarts the plugin at fixed intervals.

9. The method according to claim 8, characterized in that, After the plugin is successfully initialized and enters the ready state, the process also includes monitoring the computing resources of the Ascend NPU based on the plugin, including: The NPU identification and acquisition module starts an independent coroutine based on the Fingerprint() interface, filters the specified devices by the device ID configured, and then collects the fixed resources of the node's NPU for the first time, reports the NPU resource data to the Nomad cluster and stores it in the local cache; The NPU identification and acquisition module continuously collects NPU resources based on the Fingerprint() interface at a fixed configuration period specified in the plugin configuration file. It compares the currently collected device information with the device information in the cache to determine if there has been a change in the fixed resource. If there is a change, it generates new data for reporting and updates the local device cache; if there is no change, it skips the reporting and update. The logging and monitoring module starts another independent coroutine based on the Stats() interface. It periodically collects runtime data of the Ascend NPU device in the cache according to the fingerprint_period parameter in the plugin configuration file, encapsulates it into a standard format, and then submits it to the Nomad client to report to the Nomad server for storage.

Images