A Kubernetes-based computing chip access and management method and device, and a data center

Abstract

The application discloses a Kubernetes-based computing chip access and management method and device and a data center, comprising: determining a unified capability descriptor of a heterogeneous computing chip in a computing node; the unified capability descriptor is used for describing device attributes and computing resource capabilities of the heterogeneous computing chip; based on the unified capability descriptor, registering computing resources corresponding to the heterogeneous computing chip to Kubernetes; obtaining runtime state information of the heterogeneous computing chip, and determining whether actual available capabilities of the heterogeneous computing chip change based on the runtime state information; in response to determining that the actual available capabilities change, updating the unified capability descriptor and resource information corresponding to the heterogeneous computing chip in Kubernetes based on the changed actual available capabilities. The application can realize unified access, management and dynamic updating of resource information of different types of heterogeneous computing chips, avoid the need for independent adaptation of different types of computing chips, and thus reduce the access and management complexity of the heterogeneous computing chips.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a method, apparatus, and data center for accessing and managing computing chips based on Kubernetes. Background Technology

[0002] With the development of artificial intelligence technology, the training and inference scale of large language models is constantly expanding. Data centers are typically equipped with various types of computing acceleration chips to meet the computing resource requirements of different computing tasks. Kubernetes, as an open-source cloud-native container orchestration and resource scheduling platform, can uniformly manage the computing resources in computing nodes.

[0003] Because different types of computing acceleration chips differ in their device management interfaces and resource query methods, in current technologies, each type of computing acceleration chip typically corresponds to its own independent set of access and management components, and provides corresponding resource information to Kubernetes according to its own device management methods. As the number of computing acceleration chip types increases, the number of components and adaptation logic that need to be maintained also increases, making the access and management of heterogeneous computing chips quite complex. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a Kubernetes-based method, device, and data center for accessing and managing computing chips. The aim is to reduce the complexity of accessing and managing heterogeneous computing chips by enabling access and management of different types of computing chips through a unified capability description method.

[0005] This invention discloses a Kubernetes-based method for accessing and managing computing chips, comprising: Determine the unified capability descriptor for heterogeneous computing chips in the compute node; the unified capability descriptor is used to describe the device attributes and computing resource capabilities of heterogeneous computing chips. Based on a unified capability descriptor, the computing resources corresponding to heterogeneous computing chips are registered with Kubernetes so that Kubernetes can uniformly manage and schedule the computing resources corresponding to heterogeneous computing chips. Obtain runtime state information of heterogeneous computing chips, and determine whether the actual available capabilities of heterogeneous computing chips have changed based on the runtime state information; In response to a change in the actual available capabilities, the unified capability descriptor and the resource information corresponding to the heterogeneous computing chip in Kubernetes are updated based on the changed actual available capabilities.

[0006] Preferably, determining a unified capability descriptor for heterogeneous computing chips in a computing node includes: In response to compute nodes connecting to Kubernetes, deploy general-purpose execution containers on the compute nodes to perform device detection and capability acquisition; Obtain device and capability information of heterogeneous computing chips in compute nodes through a general execution container; Based on device information and capability information, a unified capability descriptor is determined for heterogeneous computing chips in the computing node.

[0007] Preferably, the device information and capability information of heterogeneous computing chips in the computing node are obtained through a general execution container, including: Obtain device information of heterogeneous computing chips through at least one standard universal interface; Based on device information, characteristic information corresponding to heterogeneous computing chips is obtained; the characteristic information is used to infer the operating environment characteristics and hardware resource characteristics of heterogeneous computing chips. Based on the characteristics of the operating environment and hardware resources, the capability information of heterogeneous computing chips and the inference confidence level corresponding to the capability information are determined; the inference confidence level is used to characterize the reliability of the capability information.

[0008] Preferably, it further includes: When the inference confidence level is less than the confidence threshold, determine the manufacturer information of the heterogeneous computing chip; The system retrieves enhanced information about heterogeneous computing chips by calling the vendor device management executable file corresponding to the vendor information; this enhanced information is used to supplement or correct the capability information of heterogeneous computing chips. Enhanced capability information is obtained by supplementing or correcting the capability information based on the enhanced information.

[0009] Preferably, acquiring the runtime state information of the heterogeneous computing chip and determining, based on the runtime state information, that the actual available capability of the heterogeneous computing chip has changed includes: After the heterogeneous computing chip completes resource registration, the runtime status information of the heterogeneous computing chip is obtained according to a preset time period. Based on preset capability change detection rules, the runtime status information is analyzed to determine whether the actual available capabilities of the heterogeneous computing chip have changed.

[0010] Preferably, the runtime status information includes the chip temperature and actual operating frequency of the heterogeneous computing chip; Based on preset capability change detection rules, runtime state information is analyzed to determine whether the actual available capabilities of heterogeneous computing chips have changed, including: The analysis of the runtime state information based on preset capability change detection rules to determine whether the actual available capability of the heterogeneous computing chip has changed includes: Determine the first difference between the actual operating frequency information and the rated operating frequency, and the second difference between the chip temperature and the preset temperature; If the first difference is less than a preset difference and / or the second difference is greater than a preset temperature difference, it is determined that the actual usable capability of the heterogeneous computing chip has changed. If the first difference is equal to a preset difference and / or the second difference is less than or equal to the preset temperature difference, it is determined that the actual usable capability of the heterogeneous computing chip has not changed.

[0011] Preferably, in response to a change in the actual available capability, the unified capability descriptor and the resource information corresponding to the heterogeneous computing chip in Kubernetes are updated based on the changed actual available capability, including: When the actual available capacity of a heterogeneous computing chip changes, the performance coefficients used to characterize the actual available computing capacity of the heterogeneous computing chip are determined based on the actual operating frequency. The actual available computing power after the change is determined based on the performance coefficient, and the unified capability descriptor is updated based on the actual available computing power after the change, as well as the resource information corresponding to the heterogeneous computing chip in Kubernetes.

[0012] Preferably, the runtime status information also includes memory error status information and / or driver version information; In response to a change in the actual available capacity, the unified capability descriptor and the resource information corresponding to the heterogeneous computing chip in Kubernetes are updated based on the changed actual available capacity, including: When it is determined from memory error status information that the effective available memory capacity of the heterogeneous computing chip has changed, update the memory resource information corresponding to the heterogeneous computing chip. And / or, If the driver version of a heterogeneous computing chip changes based on the driver version information, the capability information of the heterogeneous computing chip is reacquired, and the unified capability descriptor is updated based on the reacquired capability information.

[0013] This invention discloses a Kubernetes-based computing chip access and management device, comprising: The general execution container, with its multi-layered adaptive probing submodule, is used to determine the unified capability descriptor of heterogeneous computing chips in a compute node; the unified capability descriptor is used to describe the device attributes and computing resource capabilities of the heterogeneous computing chips. The unified resource registration core module is used to register the computing resources corresponding to heterogeneous computing chips to Kubernetes based on unified capability descriptors, so that Kubernetes can uniformly manage and schedule the computing resources corresponding to heterogeneous computing chips. The monitoring integration module is used to acquire runtime status information of heterogeneous computing chips; The dynamic capability awareness submodule in the general execution container is used to determine changes in the actual available capabilities of heterogeneous computing chips based on runtime state information. The dynamic capability awareness submodule is also used to update the unified capability descriptor and the resource information corresponding to heterogeneous computing chips in Kubernetes based on the changed actual available capabilities in response to the determination that the actual available capabilities have changed.

[0014] This invention discloses a Kubernetes-based data center, comprising: Control nodes are used to manage computing resources in the data center; At least one computing node is communicatively connected to the control node. The computing node includes at least one heterogeneous computing chip and a general-purpose execution container. The general-purpose execution container is used to execute the Kubernetes-based computing chip access and management method.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention enables unified access and dynamic resource management for different types of heterogeneous computing chips, avoiding the need for independent adaptation for different types of computing chips, thereby reducing the complexity of accessing and managing heterogeneous computing chips.

[0016] This invention determines a unified capability descriptor for heterogeneous computing chips and registers the corresponding computing resources of these chips with Kubernetes based on this unified capability descriptor. This allows different types of heterogeneous computing chips to access Kubernetes using a unified resource description format, eliminating the need for different resource description methods for different types of chips, reducing the complexity of heterogeneous chip access, and facilitating unified management and scheduling of different types of computing resources by Kubernetes. Furthermore, by acquiring the runtime state information of the heterogeneous computing chips, it determines whether the actual available capacity of the chips has changed. When the actual available capacity changes, the unified capability descriptor and the resource information in Kubernetes are updated based on the changed actual available capacity, ensuring that the resource capacity managed by Kubernetes matches the actual capacity currently available to the heterogeneous computing chips. For example, if the actual available computing capacity of a heterogeneous computing chip is lower than its rated computing capacity due to a change in its runtime state, the computing resource capacity declared in Kubernetes can be reduced accordingly. This prevents Kubernetes from scheduling tasks based on the resource capacity before the change, which could lead to insufficient actual computing capacity, thereby improving the accuracy and reliability of resource scheduling.

[0017] Building upon this foundation, this invention employs a three-layer progressive detection architecture. It automatically discovers heterogeneous computing chips using standard universal interfaces and infers their capabilities based on runtime environment and hardware resource characteristics. When the confidence level of the capability inference is insufficient, it further supplements or corrects the device and capability information using a vendor-managed executable file. Thus, automatic detection and integration of new chips conforming to standard specifications can be achieved without prior knowledge of the chip type or pre-installed vendor-specific tools. For new chips whose capability information cannot be fully or accurately obtained through standard interfaces and capability inference, rapid adaptation can be achieved simply by adding a vendor-managed executable file, without recompiling the universal execution container or restarting the Kubernetes cluster. Compared to manually pre-configuring drivers or redeveloping adaptation programs for different chips, this approach reduces the workload of manual configuration and adaptation development during chip integration, lowers the cost of integrating and adapting new heterogeneous computing chips, and improves the flexibility of chip integration.

[0018] Furthermore, building upon device discovery via the standard universal interface, a capability inference layer combines runtime environment characteristics such as driver version, kernel module, and operating environment, along with hardware resource characteristics such as computing resources, storage resources, and interconnect resources, to infer the capabilities of heterogeneous computing chips. This allows for the acquisition of more comprehensive capability information without relying on vendor-specific management tools, providing a basis for generating unified capability descriptors and improving the consistency of capability descriptions for different types of heterogeneous computing chips. If the confidence level of the capability inference results is insufficient, enhanced information is obtained by calling the device management executable file corresponding to the chip manufacturer. This information is then supplemented or corrected to supplement or correct the aforementioned device and capability information. This further improves the accuracy and completeness of heterogeneous computing chip device and capability information while maintaining the automatic detection and capability inference mechanism of the standard universal interface, enhancing adaptability to heterogeneous computing chips from different manufacturers.

[0019] Additionally, this invention constructs a universal execution container that supports multiple processor architectures and provides image versions corresponding to different processor architectures within the same image. This enables the universal execution container to be deployed on computing nodes with different processor architectures, thereby avoiding the need to maintain independent images for different processor architectures, reducing the maintenance cost of container images, and improving the deployment flexibility of heterogeneous computing nodes.

[0020] Furthermore, this invention collects the runtime status of different types of heterogeneous computing chips by adopting a unified index naming method, so that the runtime status of different chips can be represented in a unified index form, thereby simplifying the configuration of monitoring indicators and the maintenance of alarm rules, and facilitating horizontal comparative analysis of the runtime status of different types of heterogeneous computing chips, thereby improving the efficiency of operation and maintenance management of heterogeneous computing resources. Attached Figure Description

[0021] Figure 1 This is a schematic diagram illustrating a scenario for Kubernetes-based computing chip access and management. Figure 2 This is a schematic diagram of a Kubernetes-based computing chip access and management device; Figure 3 This is a flowchart illustrating the Kubernetes-based approach to accessing and managing computing chips. Figure 4 This is a flowchart illustrating a method for determining capability descriptors; Figure 5 This is a flowchart illustrating a method for determining device and capability information of heterogeneous computing chips. Figure 6 This is a flowchart illustrating a method for determining device and capability information of heterogeneous computing chips. Figure 7This is a flowchart illustrating a process for determining whether actual available capacity has changed. Figure 8 This is a flowchart illustrating the process of dynamically updating heterogeneous computing chip resources. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] The present invention will now be described in further detail with reference to the accompanying drawings.

[0024] With the development of artificial intelligence technology, the training and inference scale of large language models is constantly expanding. Data centers are typically equipped with various types of computing acceleration chips to meet the computing resource requirements of different computing tasks. Kubernetes, as an open-source cloud-native container orchestration and resource scheduling platform, can uniformly manage the computing resources in computing nodes.

[0025] Based on this, embodiments of this application provide a Kubernetes-based computing chip access and management method, apparatus, and data center. The following, in conjunction with... Figure 1 An example is provided.

[0026] Figure 1 This is a schematic diagram illustrating a scenario for the access and management of computing chips based on Kubernetes.

[0027] like Figure 1As shown, the corresponding data center in this scenario includes a control node 11 and at least one compute node 12. The control node 11 communicates with at least one compute node 12 and manages and schedules the computing resources in the data center based on a Kubernetes cluster. The control node 11 is used for unified management and scheduling of the computing resources in the compute node 12. The compute node 12 includes at least one heterogeneous computing chip and a general-purpose execution container. The heterogeneous computing chip can be a computing acceleration chip of different types or from different manufacturers, such as a graphics processing unit (GPU) or a neural processing unit (NPU). The general-purpose execution container is used to perform operations related to the heterogeneous computing chip, such as device detection, capability acquisition, and dynamic capability awareness, and provides the obtained relevant information to Kubernetes to achieve unified access and dynamic management of the heterogeneous computing chip.

[0028] It should be noted that the control node 11 can be the control plane of the Kubernetes cluster, used to run the control components of the Kubernetes cluster to manage and schedule the compute nodes and their compute resources. This application does not make any specific limitations on this.

[0029] The following is combined with Figure 2 An exemplary illustration is provided of the software architecture for Kubernetes-based computing chip access and management devices.

[0030] Figure 2 This is a schematic diagram of a Kubernetes-based computing chip access and management device.

[0031] like Figure 2 As shown, the Kubernetes-based computing chip access and management device is applied to the data center. The device includes a scheduling module 21, a general execution container 22, a unified resource registration core module 23, and a monitoring integration module 24.

[0032] The scheduling module 21, also known as the DaemonSet scheduling module, is used to launch the general execution container 22 on the computing nodes in the data center. Specifically, the scheduling module 21 can deploy the corresponding general execution container 22 on each computing node through the DaemonSet mechanism, so that each computing node can run the general execution container 22 to perform operations such as device detection, capability acquisition, and dynamic capability awareness for the heterogeneous computing chips in that computing node.

[0033] In one implementation, the scheduling module 21 can provide the general execution container 22 with configurable device path grouping parameters `paths.groups` via startup parameters. These device path grouping parameters declare the device file paths that may exist on the compute nodes, including character device paths, block device paths, and / or DRM render node paths. By configuring the device paths, the device file layouts of different types or manufacturers of compute chips can be adapted without modifying the internal program code of the general execution container 22.

[0034] The general-purpose execution container 22 includes a multi-layer adaptive probing submodule 221 and a dynamic capability awareness submodule 222. The multi-layer adaptive probing submodule 221 is used to determine the unified capability descriptor of heterogeneous computing chips in the computing node. The unified capability descriptor is used to describe the device attributes and computing resource capabilities of the heterogeneous computing chips.

[0035] In one implementation, the multi-layer adaptive detection submodule 221 employs a layered, progressive detection approach to discover and acquire the capabilities of heterogeneous computing chips within a computing node. This layered, progressive detection approach includes a first layer of standard universal interface detection, a second layer of capability inference, and a third layer of enhanced detection using vendor device management interfaces. The multi-layer adaptive detection submodule 221 executes the corresponding detection operations in the order of the first, second, and third layers, and gradually establishes and refines the unified capability descriptors corresponding to the heterogeneous computing chips based on the detection results of each layer.

[0036] Specifically, the first layer is a standard universal interface detection layer, also known as a zero-configuration detection layer. This layer utilizes industry-standard interfaces followed by heterogeneous computing chips to perform initial detection of heterogeneous computing chips in computing nodes without prior knowledge of the chip type or the need for pre-installed vendor-specific tools. In one implementation, the standard universal interface used in the first layer of detection may include at least one of the following: PCIe interface, sysfs interface, OpenCL interface, and Vulkan interface. Through these standard universal interfaces, basic device information of heterogeneous computing chips can be obtained, and based on the obtained basic device information, initial chip discovery, device deduplication and merging, preliminary identification of manufacturer and model, and operating system device path location can be completed.

[0037] Furthermore, the basic device information obtained from the first-layer probing can also serve as input for subsequent capability inference and be used to generate device registration information, resource names, and scheduling tags corresponding to heterogeneous computing chips. For heterogeneous computing chips that conform to the aforementioned standard universal interface, device awareness can be directly achieved based on the first-layer probing without the need for pre-configuration of vendor-specific tools.

[0038] The second layer is the capability inference layer. The multi-layer adaptive detection submodule 221, based on the device information obtained from the first layer detection, acquires driver version information, kernel module information, PCIe link parameters, and runtime information corresponding to the heterogeneous computing chip, and infers the capability feature set of the heterogeneous computing chip based on the acquired information. In one embodiment, the precision format supported by the driver corresponding to the heterogeneous computing chip can be inferred by acquiring the kernel module version information and driver management information corresponding to the heterogeneous computing chip; the memory bandwidth and computing frequency configuration of the heterogeneous computing chip can be acquired or inferred by acquiring the device driver information corresponding to the heterogeneous computing chip; and the upper bound of the inter-device interconnect bandwidth corresponding to the heterogeneous computing chip can be estimated by combining the PCIe link rate and link channel number corresponding to the heterogeneous computing chip. For example, the interconnect bandwidth can be estimated based on PCIe Gen4 or Gen5 and x8 or x16 link parameters.

[0039] Based on the above information, the computing power, storage capacity, interconnection capabilities, and precision support capabilities of heterogeneous computing chips can be inferred, forming a capability feature vector. The capability information in the feature vector can be assigned a corresponding inference confidence level, which characterizes the reliability of the capability information obtained based on the current probe information.

[0040] The third layer is the vendor device management interface enhancement layer. When the detection results of the first and / or second layers can uniquely locate the vendor information of the heterogeneous computing chip, the multi-layer adaptive detection submodule 221 can call the device management executable file corresponding to the vendor based on the determined vendor information to obtain enhanced information about the heterogeneous computing chip. In one embodiment, the device management executable file is an executable program provided by the heterogeneous computing chip vendor for obtaining device information and operating status information. By calling the device management executable file corresponding to the vendor, enhanced information such as the serial number, unique device identifier, current power consumption, and device topology of the heterogeneous computing chip can be obtained, and the device information and capability characteristics obtained from the first and second layers can be supplemented and / or corrected based on the enhanced information.

[0041] Through the above three-layer progressive detection method, the multi-layer adaptive detection submodule 221 can first use the standard general interface to complete the basic discovery of heterogeneous computing chips without pre-configuration. On this basis, it can obtain richer capability information through capability inference. When the manufacturer information can be determined and the accuracy of capability description needs to be further improved, it can then obtain enhanced information through the manufacturer's device management interface, thereby taking into account the general access capabilities and capability description accuracy of different types of heterogeneous computing chips.

[0042] The dynamic capability awareness submodule 222 is used to continuously sense the operating status of the heterogeneous computing chip after the heterogeneous computing chip completes the initial resource registration, and determine whether the actual available capability of the heterogeneous computing chip has changed based on the operating status.

[0043] In one implementation, the dynamic capability sensing submodule 222 initiates a continuous sensing task after the heterogeneous computing chip completes its initial resource registration, and collects runtime telemetry data from the heterogeneous computing chip according to a preset time period. The preset time period can be configured according to actual application requirements, for example, it can be configured to 30 seconds.

[0044] The dynamic capability awareness submodule 222 analyzes runtime telemetry data based on preset capability change detection rules to determine whether the actual available capability of the heterogeneous computing chip has changed. If it is determined that the actual available capability has changed, the dynamic capability awareness submodule 222 triggers a resource description update and updates the unified capability descriptor and the resource information corresponding to the heterogeneous computing chip in Kubernetes based on the changed actual available capability.

[0045] In one implementation, the dynamic capability perception logic of the dynamic capability perception submodule 222 includes at least one of thermal downclocking perception and dynamic correction of computing capability, ECC error accumulation perception and dynamic correction of video memory capacity, and driving thermal update perception and dynamic expansion of capability feature set.

[0046] Among them, thermal throttling sensing is used to detect changes in the actual operating frequency of heterogeneous computing chips and dynamically correct the actual available computing power of heterogeneous computing chips based on changes in the actual operating frequency; ECC error accumulation sensing is used to determine whether the effective available capacity of video memory has changed based on the error status of heterogeneous computing chips and dynamically correct the corresponding video memory resource information; driver hot update sensing is used to detect changes in the driver version of heterogeneous computing chips and re-acquire the corresponding capability feature information when the driver version changes, so as to realize the dynamic expansion of the capability feature set.

[0047] The unified resource registration core module 23 is used to register computing resources corresponding to heterogeneous computing chips with Kubernetes based on unified capability descriptors, enabling Kubernetes to uniformly manage and schedule these resources. Specifically, the unified resource registration core module 23 obtains the unified capability descriptor determined by the multi-layer adaptive detection submodule and determines the computing resource information corresponding to the heterogeneous computing chip based on the unified capability descriptor. The unified resource registration core module 23 registers the computing resources as Kubernetes-recognizable resources through the Kubelet Device Plugin API and synchronously writes at least some of the capability information from the unified capability descriptor into the node label of the computing node.

[0048] In one implementation, computing resources can be registered using a uniform resource name, such as "gpu," or an abstract resource name configured according to actual application requirements. Node tags can use structured tags to describe the capability information of heterogeneous computing chips, including at least one of the following: "node.chip / vendor" for chip manufacturer, "node.chip / model" for chip model, "node.chip / precision" for precision support, "node.chip / memory-gb" for device memory capacity, and "node.chip / perf-ratio" for current performance coefficient. The current performance coefficient characterizes the degree of change in the heterogeneous computing chip's current actual available computing power relative to its rated computing power, and can be dynamically updated by the dynamic capability awareness submodule based on the runtime state of the heterogeneous computing chip.

[0049] Furthermore, the unified resource registration core module 23 can associate resource registration information with node tags, ensuring that the computing resource declarations in Kubernetes correspond to the capability metadata of heterogeneous computing chips. Users can use standard scheduling mechanisms such as the Kubernetes node selector and node affinity to select matching computing nodes based on the resource capabilities requested by the workload, thereby achieving unified scheduling of different types of heterogeneous computing chips.

[0050] Therefore, through the unified resource registration core module 23, different types of heterogeneous computing chips can be provided to Kubernetes in a unified resource form, and the specific capabilities of the chips can be described by node tags. This enables the unified management of computing resource declarations, capability metadata, and subsequent running status information, thereby achieving unified registration, management, and scheduling of heterogeneous computing resources without changing the existing orchestration and scheduling mechanisms of Kubernetes.

[0051] The monitoring integration module 24 is used to acquire runtime status information of heterogeneous computing chips and process the acquired runtime status information in a unified manner to provide runtime status data to the dynamic capability perception submodule, and at the same time provide a data foundation for the operation status monitoring, historical analysis and alarm of heterogeneous computing chips.

[0052] In one implementation, the monitoring integration module 24 can periodically acquire the node status of each computing node and the operating status of heterogeneous computing chips through the management service, and persist the acquired status information to the database for front-end status query and historical analysis. For example, the management service can scan the status of computing nodes according to a preset time period and write the scanned status information into the database.

[0053] Furthermore, the monitoring integration module 24 can collect operational metrics of the heterogeneous computing chip through an exporter. These metrics can include chip power consumption, memory usage, and chip utilization. For different types of heterogeneous computing chips, a unified naming convention can be used to describe the collected operational metrics, such as using "node_ <chip> utilization " Chip utilization is represented by "node_ <chip> _memory_used_mib”< / chip> It indicates the amount of video memory used, enabling the monitoring metrics of different types of computing chips to be managed in a unified data format, and supporting cross-chip metric aggregation, comparative analysis, and alarm rule reuse.

[0054] Furthermore, capability change events generated by the dynamic capability awareness submodule can be exposed through the monitoring integration module 24 in the form of custom monitoring metrics. For example, "chip_perf_ratio_current" can represent the current performance coefficient of the heterogeneous computing chip, and "chip_ecc_errors_total" can represent the cumulative number of error correcting code (ECC) errors that have occurred in the heterogeneous computing chip.

[0055] In this way, the monitoring integration module 24 can convert the operating status and capability change information of different types of heterogeneous computing chips into unified monitoring data, so that monitoring indicators, historical status and alarm information can be managed in a unified manner, thereby reducing the monitoring and maintenance complexity in multi-chip mixed deployment scenarios, and supporting horizontal comparative analysis of the operating status between different types of computing chips.

[0056] The following section, with reference to the accompanying diagram, explains the specific implementation process of the Kubernetes-based computing chip access and management method.

[0057] Figure 3 This is a flowchart illustrating the Kubernetes-based approach to accessing and managing computing chips.

[0058] like Figure 3 As shown, the Kubernetes-based computing chip access and management method includes the following steps: S1. Determine the unified capability descriptor for heterogeneous computing chips in the computing node.

[0059] Figure 4 This is a flowchart illustrating a method for determining capability descriptors.

[0060] like Figure 4 As shown, step S1 includes steps S11-S13.

[0061] S11: In response to compute nodes joining the Kubernetes cluster, deploy a general-purpose execution container on the compute node to perform device detection and capability acquisition.

[0062] Kubernetes is a container orchestration platform for automating the deployment, management, and scheduling of containerized applications. Kubernetes can organize multiple compute nodes in a data center and uniformly manage and schedule the computing resources provided by each compute node. A Kubernetes cluster, also known as a Kubernetes-based data center, is a resource management cluster composed of multiple communicating compute nodes and a control component for unified management of these compute nodes.

[0063] In step S11, if the compute node has successfully joined the Kubernetes cluster, the scheduling module in the Kubernetes cluster can deploy a general execution container on the compute node. The general execution container can be pre-packaged as a container image and deployed to the corresponding compute node through the container orchestration mechanism provided by Kubernetes. For example, a DaemonSet controller can create a corresponding general execution container for each compute node joining the cluster, so that each compute node runs a general execution container. The general execution container is used to perform device detection and capability acquisition operations for heterogeneous computing chips locally on the corresponding compute node.

[0064] For example, suppose a data center currently includes compute node 1, compute node 2, and compute node 3, and these three compute nodes have already joined a Kubernetes cluster. When a new compute node A is added to the data center, compute node A can first be added to the Kubernetes cluster. After compute node A is added to the cluster, the Kubernetes cluster can recognize compute node A as a new compute node and create and run a generic execution container on compute node A through the DaemonSet controller.

[0065] S12: Obtain device and capability information of heterogeneous computing chips in the compute node through a general execution container.

[0066] Figure 5 This is a flowchart illustrating a method for determining device information and capability information of heterogeneous computing chips.

[0067] like Figure 5 As shown, step S12 includes steps S121-S123.

[0068] S121: Obtain device information of heterogeneous computing chips through at least one standard universal interface.

[0069] In one implementation, a general-purpose execution container can perform device detection on heterogeneous computing chips in a compute node through a standard general-purpose interface. This standard general-purpose interface can be an industry-standard interface followed by the heterogeneous computing chips. The standard general-purpose interface may include at least one of the following: a Peripheral Component Interconnect (PCIe) configuration space interface, a system device file interface (Linux sysfs device file interface) provided by the Linux operating system, an Open Computing Language (OpenCL) runtime query interface, a physical device enumeration interface provided by the Vulkan graphics and computing interface, or an equivalent accelerator character device interface exposed by the operating system.

[0070] Different standard universal interfaces can obtain different device information. Device information may include one or more of the following: device manufacturer identifier, device model identifier, device type, device path, device node, bus address, driver binding information, visible computing interface, device unique identifier, and basic resource information directly provided by the corresponding standard universal interface.

[0071] For heterogeneous computing chips discovered through any standard universal interface, the device information is used to determine at least the device identity and device location information of the heterogeneous computing chip. Device identity information is used to determine the manufacturer, model, or device type of the heterogeneous computing chip, while device location information is used to determine the device location or access point of the heterogeneous computing chip within the computing node. Device location information may include one or more of the following: operating system device path, device node, bus address, or unique device identifier.

[0072] Specifically, when probing devices via the PCIe (Peripheral Component Interconnect) configuration space interface, device information such as device category, manufacturer identifier, model identifier, and bus address corresponding to the heterogeneous computing chip can be obtained to identify the computing accelerator device. When probing devices via the system device file interface provided by the Linux operating system, device information such as device node, device path, driver binding relationship, and bus address corresponding to the heterogeneous computing chip can be obtained. When probing devices via the OpenCL (Open Computing Language) runtime query interface, device information such as device name, manufacturer, number of compute units, device memory capacity, maximum workgroup size, and supported precision extensions corresponding to the heterogeneous computing chip can be obtained. When probing devices via the physical device enumeration interface provided by the Vulkan graphics and computing interface, device information such as manufacturer identifier, model identifier, device name, device type, driver version, and unique device identifier corresponding to the heterogeneous computing chip can be obtained.

[0073] In one implementation, when a heterogeneous computing chip can only be discovered through one standard universal interface, the universal execution container can obtain the corresponding device information based on the standard universal interface, and determine the device identity and device location information of the heterogeneous computing chip based on the obtained device information; when a heterogeneous computing chip can be discovered through multiple standard universal interfaces, the universal execution container can obtain the corresponding device information through multiple standard universal interfaces respectively, and associate, deduplicate and merge the obtained device information.

[0074] In this way, regardless of whether the heterogeneous computing chip can be discovered through one or more standard universal interfaces, device information corresponding to the heterogeneous computing chip can be formed, and the object and information acquisition entry point for subsequent capability information acquisition can be determined based on the device information.

[0075] S122: Based on device information, obtain the feature information corresponding to the heterogeneous computing chip.

[0076] In one implementation, the general execution container can determine the device access entry corresponding to the heterogeneous computing chip based on the device information obtained in S121, and further obtain the characteristic information corresponding to the heterogeneous computing chip based on the device access entry. The characteristic information is used to infer the operating environment characteristics and hardware resource characteristics of the heterogeneous computing chip. Among them, the operating environment characteristics are used to characterize the current software operating environment of the heterogeneous computing chip and the callable capabilities of the heterogeneous computing chip in the software operating environment; the hardware resource characteristics are used to characterize the computing resources, storage resources, and interconnect resources possessed by the heterogeneous computing chip.

[0077] Specifically, the general execution container can determine the entry point for obtaining driver, kernel module, runtime environment and hardware resource information corresponding to the heterogeneous computing chip based on at least one of the following device information: device manufacturer identifier, device model identifier, device type, device path, device node, bus address, driver binding information and visible computing interface.

[0078] In one implementation, the general execution container can determine the kernel module corresponding to the heterogeneous computing chip based on driver binding information or device location information, and obtain the version information of the kernel module and driver management information to determine the runtime environment characteristics such as the driver version, kernel module and precision format supported by the driver corresponding to the heterogeneous computing chip.

[0079] Furthermore, the general-purpose execution container can determine the device information acquisition entry point corresponding to the heterogeneous computing chip based on at least one of the following: device path, device node, or device type. It can then acquire hardware resource characteristics of the heterogeneous computing chip, such as memory bandwidth and computing frequency configuration, through this entry point. The general-purpose execution container can also determine the runtime environment corresponding to the heterogeneous computing chip based on the device information and acquire runtime environment characteristics, such as runtime interface type, runtime version, and supported device capabilities, through the runtime interface corresponding to that environment.

[0080] In one implementation, the runtime environment characteristics may include basic device identification characteristics and driver and runtime characteristics. The basic device identification characteristics may include a device vendor identifier (vendor), a device model identifier (model), a device type (device_type), and a driver name (driver_name), used to characterize the manufacturer, model, device type, and currently bound driver of the heterogeneous computing chip. The driver and runtime characteristics may include a driver version (driver_version), a kernel module (kernel_module), a runtime interface (runtime_api), and a runtime version (runtime_version), used to characterize the driver, kernel module, and runtime environment capable of calling the heterogeneous computing chip.

[0081] Furthermore, the general-purpose execution container can determine the hardware resource information acquisition entry point corresponding to the heterogeneous computing chip based on at least one of device path, device node, or device type, and obtain the hardware resource characteristics corresponding to the heterogeneous computing chip through the hardware resource information acquisition entry point. Specifically, the general-purpose execution container can obtain hardware resource characteristics such as memory bandwidth and computing frequency configuration of the heterogeneous computing chip through the device driver information corresponding to the heterogeneous computing chip.

[0082] The general-purpose execution container can obtain PCIe link information corresponding to heterogeneous computing chips based on device information. This PCIe link information can include the PCIe link rate and the number of PCIe link channels. The PCIe link rate can be Gen4 or Gen5, and the number of PCIe link channels can be x8 or x16. Based on the PCIe link rate and the number of PCIe link channels, the general-purpose execution container can estimate the upper bound of the interconnect bandwidth between the heterogeneous computing chip and the host or other heterogeneous computing chips, and incorporate this upper bound of interconnect bandwidth as part of the hardware resource characteristics.

[0083] In one implementation, hardware resource characteristics may include computing power characteristics, storage capacity characteristics, and interconnection capability characteristics. The computing power characteristics may include the number of computing units, maximum operating frequency, supported precision formats, and supported matrix or tensor acceleration capabilities, characterizing the parallel computing scale, frequency ceiling, supported precision formats, and matrix or tensor acceleration capabilities of the heterogeneous computing chip. The storage capacity characteristics may include total video memory capacity, video memory bandwidth, and storage media type, characterizing the available video memory capacity, video memory bandwidth, and storage media type of the heterogeneous computing chip. The interconnection capability characteristics may include PCIe link generation, number of PCIe link channels, interconnect type, and interconnect bandwidth, characterizing the data transfer capabilities between the heterogeneous computing chip and the host or other heterogeneous computing chips.

[0084] It should be noted that different types of heterogeneous computing chips and different device information acquisition entry points can obtain different feature information. The general execution container can select the corresponding information acquisition entry point that it can access based on the actual device information obtained by S121, and obtain one or more feature information.

[0085] S123: Based on the operating environment characteristics and hardware resource characteristics of heterogeneous computing chips, determine the capability information of heterogeneous computing chips and the inference confidence level corresponding to the capability information.

[0086] Capability information is used to characterize the computing, storage, and interconnect resources that heterogeneous computing chips can provide, as well as the resource access capabilities they support in the current operating environment. Inference confidence is used to characterize the reliability of the capability information.

[0087] In one implementation, the general-purpose execution container can analyze runtime environment characteristics and hardware resource characteristics, and determine the capability information corresponding to heterogeneous computing chips based on preset capability inference rules. Furthermore, the general-purpose execution container can determine the inference confidence level corresponding to the capability information based on the source, completeness, and consistency between feature information from different sources used to determine the capability information. A higher inference confidence level indicates that the capability information determined based on the feature information is more reliable.

[0088] In steps S121-S123 above, firstly, step S121 uses a standard universal interface to perform basic discovery of heterogeneous computing chips. Since different standard universal interfaces can obtain different device information, the corresponding device information can be obtained according to the interface used. Based on this, step S122 uses the existing device information, combined with operating environment information and hardware resource information, to further infer the capability information of the heterogeneous computing chip and the corresponding inference confidence level. When the inference confidence level reaches the preset confidence level, it indicates that the existing information can reliably determine the capability information, and there is no need to perform third-layer probing. A unified capability descriptor can be directly generated and unified resource registration can be performed. When the inference confidence level does not reach the preset confidence level, it indicates that the existing universal interface and system information are insufficient to reliably determine some capability information. At this time, if the manufacturer corresponding to the heterogeneous computing chip has been determined, step S123 can be used to call the manufacturer's dedicated System Management Interface (SMI) to obtain more accurate or complete information to supplement or correct the capability information. Therefore, third-layer probing, as an optional enhancement mechanism, is only executed when the aforementioned capability inference results are insufficient to meet the reliability requirements. The following will combine... Figure 6 Please provide an explanation.

[0089] Figure 6 This is a flowchart illustrating a method for determining device information and capability information of heterogeneous computing chips.

[0090] like Figure 6 As shown, step S12 includes steps S124-S126.

[0091] S124: Determine the manufacturer information of the heterogeneous computing chip when the inference confidence level is less than the confidence level threshold.

[0092] If the inference confidence level is less than the preset confidence level threshold, it indicates that the reliability of the capability information obtained based on the standard general interface, operating environment information, and hardware resource information has not met the preset requirements. In this case, the capability information of the heterogeneous computing chip can be further enhanced.

[0093] Specifically, the general execution container can determine the manufacturer information corresponding to the heterogeneous computing chip based on device information such as device manufacturer identifier, device model identifier, and device type obtained from the first layer, and runtime environment characteristics such as device basic identifier features, driver and runtime features obtained from the second layer.

[0094] In one implementation, when the relevant information obtained by the first or second layer can uniquely identify the manufacturer corresponding to the heterogeneous computing chip, the manufacturer information can be determined; when the obtained information cannot uniquely identify the manufacturer corresponding to the heterogeneous computing chip, the capability information obtained based on the aforementioned steps can be maintained without performing subsequent manufacturer device management executable file call operations.

[0095] S125: Call the vendor device management executable file corresponding to the vendor information to obtain enhanced information about heterogeneous computing chips.

[0096] Enhanced information is used to supplement or correct the capability information of heterogeneous computing chips. Vendor device management executables are executable programs provided by the manufacturers of heterogeneous computing chips or corresponding to heterogeneous computing chips, used for device management and information retrieval of heterogeneous computing chips.

[0097] After determining the vendor information, the general-purpose execution container can identify the corresponding vendor device management executable file based on a pre-defined mapping between vendor information and device management executable files. It then calls this executable file to obtain enhanced information about the heterogeneous computing chip. For example, the vendor device management executable file can be an executable program provided by the heterogeneous computing chip vendor to obtain device information and operational status information. For heterogeneous computing chips from different vendors, the container can call the device management executable file matched to the corresponding vendor to obtain the enhanced information for the heterogeneous computing chip.

[0098] In one implementation, the enhanced information may include one or more of the following: the precise serial number of the heterogeneous computing chip, the unique identifier of the device, the current power consumption, and the device topology information.

[0099] Among them, device topology information can be used to characterize the connection relationship between heterogeneous computing chips and hosts or other heterogeneous computing chips, current power consumption can be used to characterize the current power consumption status of heterogeneous computing chips, and precise serial number and device unique identifier can be used to further confirm the device identity information determined in the aforementioned steps.

[0100] It should be noted that the enhancement layer corresponding to the vendor device management executable file is an optional enhancement layer. When the vendor device management executable file corresponding to the vendor information is unavailable, or the vendor of the heterogeneous computing chip does not provide a callable device management executable file, it will not affect the basic detection, capability inference, and resource registration of heterogeneous computing chips based on the aforementioned standard general interface, operating environment characteristics, and hardware resource characteristics.

[0101] S126: Perform supplementary or corrective operations on the capability information based on the enhanced information to obtain enhanced capability information.

[0102] In one implementation, the general execution container can associate the enhancement information with the capability information determined in S123, and supplement the missing capability description content in the capability information based on the enhancement information, or correct the capability description content that is inconsistent with the enhancement information, so as to obtain enhanced capability information.

[0103] Specifically, when the enhanced information can provide device information or capability information that was not obtained in the aforementioned steps, the capability information can be supplemented based on the enhanced information; when there is a difference between the enhanced information and the capability information determined in the aforementioned steps, the corresponding capability information can be corrected based on the enhanced information.

[0104] For example, the capability information formed in the aforementioned steps can be supplemented or corrected based on the precise device unique identifier, current power consumption, and detailed topology information obtained from the manufacturer's device management executable file, thereby obtaining more complete or more accurate enhanced capability information.

[0105] S13: Based on device information and capability information, determine the unified capability descriptor of heterogeneous computing chips in the computing node.

[0106] In one implementation, the general execution container can summarize and encapsulate the device information, capability information, and inference confidence levels obtained in the aforementioned steps to generate a unified capability description structure corresponding to the heterogeneous computing chip. This unified capability description structure uses a unified data structure to describe heterogeneous computing chips of different types or from different manufacturers, enabling the device information and resource capabilities of different heterogeneous computing chips to be provided in a unified data format to the subsequent unified resource registration core module and scheduling module.

[0107] In one implementation, the unified capability description structure may include necessary fields and optional enhancement fields. The necessary fields, used to meet the device identification, resource registration, and basic scheduling requirements of heterogeneous computing chips, may include: a unique device identifier to distinguish different physical chips or logical devices; a vendor to determine the vendor category of the heterogeneous computing chip; a device model to distinguish different chip specifications from the same vendor; the number of computing units to describe the parallel computing scale of the heterogeneous computing chip; a precision support list to describe the computing precision supported by the heterogeneous computing chip, which may include FP32, FP16, BF16, INT8, etc.; total device memory capacity to describe the total video memory capacity or available device memory capacity of the heterogeneous computing chip, and used for resource capacity registration and scheduling constraint generation; PCIe link generation to describe the PCIe link generation between the heterogeneous computing chip and the host; the number of PCIe link channels to describe the number of PCIe link channels between the heterogeneous computing chip and the host and the corresponding link bandwidth limit; and a probe confidence score to characterize the reliability of device and capability information, with the probe confidence score ranging from 0 to 1. Optional enhancement fields are used to further describe the performance and topology characteristics of heterogeneous computing chips, provided that the relevant information is available. Optional enhancement fields may include at least one of the following: memory bandwidth (memory_bandwidth_gbps), interconnect type (interconnect_type), and interconnect bandwidth (interconnect_bandwidth_gbps). The unit for memory bandwidth can be GB / s, used for more refined performance modeling and scheduling optimization; the interconnect type describes the interconnection method between heterogeneous computing chips and other heterogeneous computing chips, such as PCIe (High-Speed ​​Interconnect), NVLink (Dedicated High-Speed ​​Interconnect Link), or vendor-defined interconnection methods.

[0108] In one implementation, optional enhancement fields can be obtained through a standard general interface or through a vendor device management executable corresponding to the heterogeneous computing chip.

[0109] In summary, through the above methods, regardless of the manufacturer or hardware architecture of the heterogeneous computing chip, its device identity and resource capabilities can be described based on a unified capability description structure, providing a standardized data foundation for the subsequent unified registration of the computing resources corresponding to the heterogeneous computing chip to Kubernetes.

[0110] S2. Based on the unified capability descriptor, register the computing resources corresponding to the heterogeneous computing chips to Kubernetes so that Kubernetes can uniformly manage and schedule the computing resources corresponding to the heterogeneous computing chips.

[0111] In one implementation, the resource registration component can register computing resources corresponding to heterogeneous computing chips as Kubernetes resources via the Kubelet DevicePlugin API, based on a unified capability descriptor. Computing resources can be registered using a unified resource name, such as "gpu," or a configurable abstract resource name. The generic execution container can synchronously write key capability fields from the unified capability descriptor into the node label of the compute node. The node label can include capability attributes such as vendor, model, precision support, device memory capacity, and current performance coefficients. Kubernetes can then use scheduling mechanisms such as `nodeSelector` or `nodeAffinity` based on the computing resources and node labels to select the corresponding compute node according to the resource capabilities requested by the workload, thereby achieving unified management and scheduling of computing resources corresponding to heterogeneous computing chips of different types or vendors.

[0112] S3. Obtain the runtime status information of the heterogeneous computing chip, and determine whether the actual available capabilities of the heterogeneous computing chip have changed based on the runtime status information.

[0113] Actual available capability is used to characterize the computing, storage, or other resource capabilities that a heterogeneous computing chip can actually provide in its current operating state.

[0114] Figure 7 This is a flowchart illustrating a process for determining whether actual available capacity has changed.

[0115] Step S3 includes steps S31-S32.

[0116] S31: After the heterogeneous computing chip completes resource registration, obtain the runtime status information of the heterogeneous computing chip according to the preset time period.

[0117] The preset time period refers to the time interval between two consecutive acquisitions of runtime status information of the heterogeneous computing chip. The preset time period can be configured according to the type of heterogeneous computing chip, the operating load, or monitoring requirements. For example, it can be set to 30 seconds, 1 minute, etc., and this application does not make specific limitations on it.

[0118] In one implementation, the runtime status information includes the chip temperature and actual operating frequency of the heterogeneous computing chip. The chip temperature refers to the temperature value of the heterogeneous computing chip in its current operating state, reflecting its current thermal state; the actual operating frequency refers to the operating frequency used by the heterogeneous computing chip during its current actual operation, reflecting its current actual computing state.

[0119] In another implementation, the runtime status information, in addition to chip temperature and actual operating frequency, may also include memory error status information and / or driver version information. The memory error status information reflects the memory error status of the heterogeneous computing chip, and the driver version information reflects the driver version currently used by the heterogeneous computing chip.

[0120] The runtime status information of heterogeneous computing chips can be periodically acquired by a monitoring integration module running on the compute nodes, and this information can be persisted to support runtime status queries and historical analysis. Simultaneously, runtime status indicators such as power consumption, device memory usage, and utilization of heterogeneous computing chips can be collected according to a unified indicator naming convention. For example, "node_" can be used. <chip>_utilization、node_ <chip>The system uses standardized indicator names such as "_memory_used_mib" to provide a unified data foundation for monitoring the operational status of different types of heterogeneous computing chips.

[0121] S32: Analyze runtime status information based on preset capability change detection rules to determine whether the actual available capability of the heterogeneous computing chip has changed.

[0122] Capability change detection rules refer to preset judgment rules that determine whether the actual available capability of a heterogeneous computing chip has changed based on its runtime state information. Capability change detection rules may include one or more of the following: comparison rules between runtime state information and a preset reference state, a change threshold corresponding to the runtime state information, and judgment conditions based on the change threshold to determine whether the actual available capability has changed.

[0123] Continue as Figure 7 As shown, step S32 includes steps S321-S323.

[0124] S321: Determine the first difference between the actual operating frequency information and the rated operating frequency, and the second difference between the chip temperature and the preset temperature.

[0125] The rated operating frequency characterizes the operating frequency of the heterogeneous computing chip under normal operating conditions, while the actual operating frequency characterizes the current operating frequency used by the heterogeneous computing chip. By determining a first difference between the two, the degree of change in the current operating frequency of the heterogeneous computing chip relative to its normal operating state can be obtained. In one embodiment, the difference between the rated operating frequency and the actual operating frequency can be used as the first difference. For example, when the actual operating frequency is lower than the rated operating frequency, the first difference can be used to characterize the degree of frequency reduction of the heterogeneous computing chip.

[0126] The preset temperature characterizes the reference temperature corresponding to the heterogeneous computing chip under normal operating conditions, while the chip temperature characterizes the current actual temperature of the heterogeneous computing chip. By determining the second difference between the chip temperature and the preset temperature, the degree of change in the current temperature of the heterogeneous computing chip relative to the reference temperature under normal operating conditions can be obtained. In one embodiment, the difference between the chip temperature and the preset temperature can be used as the second difference. For example, when the chip temperature is higher than the preset temperature, the second difference can be used to characterize the degree of temperature rise of the heterogeneous computing chip. When the temperature of the heterogeneous computing chip rises to a certain level, it may trigger the chip's temperature protection mechanism, causing the heterogeneous computing chip to reduce its actual operating frequency, thereby reducing its actual usable computing power. Therefore, the second difference can be used to determine whether the current temperature of the heterogeneous computing chip has reached a level that may affect its actual usable power.

[0127] S322: If the first difference is less than a preset difference and / or the second difference is greater than a preset temperature difference, it is determined that the actual usable capability of the heterogeneous computing chip has changed.

[0128] The preset difference is used to determine whether the change in the actual operating frequency meets the capability change condition. It can be a frequency difference preset based on the type of heterogeneous computing chip, the rated operating frequency, and the actual application requirements, or it can be a preset percentage of the rated operating frequency. For example, the preset difference can be set to 15% of the rated operating frequency, meaning that when the actual operating frequency drops by more than 15% compared to the rated operating frequency, it is determined that the change in the actual operating frequency meets the capability change condition.

[0129] When the first difference is less than the preset difference, it indicates that the actual operating frequency of the heterogeneous computing chip has decreased significantly compared to its rated operating frequency. At this point, it can be considered that the actual usable computing power of the heterogeneous computing chip has changed. For example, when a heterogeneous computing chip reduces its frequency due to excessive temperature, the actual operating frequency will decrease. When the degree of frequency reduction exceeds a preset threshold, it can be determined that its actual usable computing power has changed.

[0130] The preset temperature difference is used to determine whether the current temperature change of a heterogeneous computing chip has reached a level that may affect its actual usability. It can be a pre-set temperature difference value based on the type of heterogeneous computing chip, its normal operating temperature range, and actual application requirements. For example, the preset temperature difference can be set as a preset increment of a preset temperature. When the chip temperature is higher than the preset temperature and the difference between the two exceeds the preset temperature difference, it can be determined that the current temperature of the heterogeneous computing chip has reached a level that may affect its actual usability.

[0131] When the second temperature difference exceeds the preset temperature difference, it indicates that the current temperature of the heterogeneous computing chip has increased significantly compared to the reference temperature under normal operating conditions. At this point, it can be considered that the actual usable capability of the heterogeneous computing chip has changed. For example, when the heterogeneous computing chip operates under continuous high load, causing the chip temperature to rise and reach the temperature protection condition, the heterogeneous computing chip may reduce its actual operating frequency to control the temperature, thereby reducing its actual usable computing capability. Therefore, even if the actual operating frequency has not yet reached the corresponding frequency change threshold, it is possible to determine in advance, based on temperature changes, that the actual usable capability of the heterogeneous computing chip may have changed.

[0132] S323: If the first difference is less than a preset difference and / or the second difference is less than or equal to a preset temperature difference, it is determined that the actual usable capability of the heterogeneous computing chip has not changed.

[0133] When the first difference is less than the preset difference, it indicates that the change in the actual operating frequency compared to the rated operating frequency is still within the allowable range. Therefore, it can be determined that the actual available computing power of the heterogeneous computing chip has not changed, and there is no need to adjust the computing resource capability information.

[0134] When the second difference is less than or equal to the preset temperature difference, it indicates that the change in the current temperature of the heterogeneous computing chip compared to the reference temperature under normal operating conditions is still within the allowable range and has not reached the temperature change condition that may affect the actual usability of the heterogeneous computing chip. Therefore, it can be determined that the actual usability of the heterogeneous computing chip has not changed due to temperature changes, and there is no need to adjust the capability information of computing resources.

[0135] S4. In response to a change in the actual available capabilities, update the unified capability descriptor and the resource information corresponding to the heterogeneous computing chip in Kubernetes based on the changed actual available capabilities.

[0136] Optionally, changes in actual available capabilities may include one or more of the following: changes in the actual available computing power of heterogeneous computing chips, changes in effective available memory capacity, and changes in capability characteristics. For different types of changes in actual available capabilities, corresponding update methods can be used to update the unified capability descriptor and resource information in Kubernetes.

[0137] Specifically, when the actual available computing power changes, the performance coefficient can be determined based on the actual operating frequency of the heterogeneous computing chip, and the computing resource declaration corresponding to the heterogeneous computing chip can be adjusted according to the performance coefficient; when the effective available memory capacity changes, the corresponding memory resource information can be updated according to the changed effective available memory capacity; when the capability characteristics change, the capability information of the heterogeneous computing chip can be re-acquired, and the unified capability descriptor can be updated accordingly.

[0138] The following is combined with Figure 8 The resource update process under the different scenarios described above will be explained.

[0139] Figure 8 This is a flowchart illustrating the process of dynamically updating heterogeneous computing chip resources.

[0140] like Figure 8 As shown in (a), step S4 includes steps S41-S42.

[0141] S41: When it is determined that the actual available computing power of the heterogeneous computing chip changes, the performance coefficient used to characterize the actual available computing power of the heterogeneous computing chip is determined based on the actual operating frequency.

[0142] The performance coefficient characterizes the degree of change in the actual usable computing power of a heterogeneous computing chip relative to its rated computing power. The performance coefficient can be determined based on the ratio of the actual operating frequency to the rated operating frequency. When the actual operating frequency decreases compared to the rated operating frequency, the corresponding performance coefficient can be determined based on the proportional relationship between the two.

[0143] For example, when it is detected that the actual operating frequency drops by more than a preset ratio relative to the rated operating frequency, the performance coefficient can be determined based on the ratio between the actual operating frequency and the rated operating frequency.

[0144] S42: Determine the actual available computing power after the change based on the performance coefficient, and update the unified capability descriptor and resource information corresponding to heterogeneous computing chips in Kubernetes based on the actual available computing power after the change.

[0145] Specifically, the available computing resource declarations corresponding to heterogeneous computing chips can be adjusted proportionally based on performance coefficients, and the real-time computing power coefficients can be updated via Kubernetes node annotations, such as updating the "chip_perf_ratio" field, for reference during resource scheduling. When the temperature of the heterogeneous computing chip returns to normal and the actual operating frequency rebounds, the performance coefficients can be redefined, and the available computing resource declarations can be restored accordingly, thereby achieving dynamic adjustment of computing resources.

[0146] like Figure 8 As shown in (b), step S4 includes steps S43-S44.

[0147] S43: If the effective available memory capacity of the heterogeneous computing chip changes based on the memory error status information, update the memory resource information corresponding to the heterogeneous computing chip.

[0148] Memory error status information refers to information used to characterize whether there are errors in the device memory of heterogeneous computing chips and the degree of error. It can include information such as ECC error count and uncorrectable error count.

[0149] In one implementation, ECC error count information of heterogeneous computing chips can be acquired periodically, for example, by acquiring the ECC double-bit error count through the vendor device management interface (SMI) or the / sys / class / drm interface corresponding to the heterogeneous computing chip. If the accumulated number of uncorrectable errors exceeds a preset threshold, it can be determined that some device memory of the heterogeneous computing chip is abnormal, and the memory page corresponding to the row of video memory where the ECC error occurred is marked as unavailable.

[0150] Furthermore, the effective available memory capacity of the heterogeneous computing chip can be re-determined based on the memory pages marked as unavailable, and the memory resource information corresponding to the heterogeneous computing chip in Kubernetes can be updated based on the re-determined effective available memory capacity. This allows Kubernetes to schedule resources based on the updated effective available memory capacity, avoiding scheduling tasks with high requirements for device memory capacity to heterogeneous computing chips with memory anomalies.

[0151] S44: If it is determined that the driver version of the heterogeneous computing chip has changed based on the driver version information, the capability information of the heterogeneous computing chip is reacquired, and the unified capability descriptor is updated based on the reacquired capability information.

[0152] In one implementation, the driver version information corresponding to the heterogeneous computing chip can be continuously monitored to determine whether the driver version has changed. Specifically, the driver version information corresponding to the heterogeneous computing chip can be monitored through the file change monitoring mechanism provided by the operating system to obtain events indicating that the driver version has changed. When a driver version change is detected, such as a driver version upgrade, the aforementioned capability information acquisition process can be re-executed to re-acquire the capability information of the heterogeneous computing chip, and the unified capability descriptor can be updated based on the re-acquired capability information.

[0153] Specifically, the aforementioned capability inference and enhanced detection process for vendor device management executables can be re-executed, and the corresponding capability fields in the unified capability descriptor can be updated based on the re-acquired capability information. For example, when the updated driver adds support for FP8 calculation precision, the precision support list (precision_support) in the unified capability descriptor can be updated, and other relevant capability fields can be updated based on the re-acquired capability information.

[0154] Furthermore, the capability attributes corresponding to the updated unified capability descriptor can be synchronized to the node labels and monitoring system in Kubernetes, enabling Kubernetes to perform resource management and scheduling based on the updated capability attributes, and enabling the monitoring system to monitor the running status based on the updated capability characteristics.

[0155] It should be noted that steps S1-S4 above can be combined or implemented independently according to actual application requirements. Among them, steps S1-S2 are mainly used to realize unified access of heterogeneous computing chips, and steps S3-S4 are mainly used to realize dynamic management of heterogeneous computing chips.

[0156] In one implementation, steps S1-S4 can be executed sequentially after heterogeneous computing chips are connected to the same computing node, so as to achieve unified access and dynamic management of heterogeneous computing chips at the same time.

[0157] In another implementation, only steps S1-S2 can be executed. After completing device detection, capability acquisition, unified capability descriptor generation, and computing resource registration for heterogeneous computing chips, unified access to heterogeneous computing chips can be achieved without further executing steps S3-S4 for dynamic capability awareness and resource updates.

[0158] In another implementation, if the heterogeneous computing chip has completed unified access or already has corresponding resource registration information, steps S3-S4 can be executed only to continuously sense the operating status of the heterogeneous computing chip and dynamically update the corresponding resource information according to changes in actual available capabilities, so as to realize dynamic management of the heterogeneous computing chip.

[0159] Therefore, steps S1-S2 and S3-S4 can be executed in combination or as independent functional processes, thus forming a unified access implementation, a unified access and dynamic management implementation, and a dynamic management implementation, respectively.

[0160] In summary, the above methods enable unified access, management, and dynamic updating of resource information for different types of heterogeneous computing chips, reducing the need for independent adaptation for different types of computing chips and thus lowering the complexity of accessing and managing heterogeneous computing chips.

[0161] It should be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0162] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0163] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0164] For the purposes of this specification, "computer-readable medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0165] More specific examples of computer-readable media (a non-exhaustive list) include the following: electrical connections having one or more wires (electronic devices), portable computer disks (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CD-ROM).

[0166] Furthermore, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory. It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof.

[0167] In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc. The above embodiments are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of this application should be included within the scope of protection of this application.< / chip> < / chip> < / chip>

Claims

1. A method for accessing and managing computing chips based on Kubernetes, characterized in that, include: Determine a unified capability descriptor for heterogeneous computing chips in a compute node; The unified capability descriptor is used to describe the device attributes and computing resource capabilities of the heterogeneous computing chip; Based on the unified capability descriptor, the computing resources corresponding to the heterogeneous computing chip are registered with Kubernetes, so that Kubernetes can uniformly manage and schedule the computing resources corresponding to the heterogeneous computing chip. Obtain the runtime state information of the heterogeneous computing chip, and determine whether the actual available capability of the heterogeneous computing chip has changed based on the runtime state information; In response to determining that the actual available capability has changed, the unified capability descriptor and the resource information in Kubernetes corresponding to the heterogeneous computing chip are updated based on the changed actual available capability.

2. The Kubernetes-based computing chip access and management method according to claim 1, characterized in that, The unified capability descriptor for determining heterogeneous computing chips in a computing node includes: In response to the computing node connecting to Kubernetes, a general execution container for performing device detection and capability acquisition is deployed on the computing node; The device and capability information of the heterogeneous computing chips in the computing node are obtained through the general execution container. Based on the device information and capability information, a unified capability descriptor for heterogeneous computing chips in the computing node is determined.

3. The Kubernetes-based computing chip access and management method according to claim 2, characterized in that, The step of obtaining device information and capability information of heterogeneous computing chips in the computing node through the general execution container includes: The device information of the heterogeneous computing chip is obtained through at least one standard universal interface; Based on the device information, feature information corresponding to the heterogeneous computing chip is obtained; the feature information is used to infer the operating environment characteristics and hardware resource characteristics of the heterogeneous computing chip. Based on the operating environment characteristics and the hardware resource characteristics, the capability information of the heterogeneous computing chip and the inference confidence level corresponding to the capability information are determined; the inference confidence level is used to characterize the reliability of the capability information.

4. The Kubernetes-based computing chip access and management method according to claim 3, characterized in that, Also includes: If the inference confidence level is less than the confidence threshold, the manufacturer information of the heterogeneous computing chip is determined; The vendor device management executable file corresponding to the vendor information is invoked to obtain the enhanced information of the heterogeneous computing chip; The enhancement information is used to supplement or correct the capability information of the heterogeneous computing chip; Based on the enhanced information, supplementary or corrective operations are performed on the capability information to obtain enhanced capability information.

5. The method for accessing and managing Kubernetes-based computing chips according to any one of claims 1-4, characterized in that, The step of acquiring the runtime state information of the heterogeneous computing chip and determining whether the actual usable capability of the heterogeneous computing chip has changed based on the runtime state information includes: After the heterogeneous computing chip completes resource registration, the runtime status information of the heterogeneous computing chip is obtained according to a preset time period. The runtime state information is analyzed based on preset capability change detection rules to determine whether the actual available capability of the heterogeneous computing chip has changed.

6. The Kubernetes-based computing chip access and management method according to claim 5, characterized in that, The runtime status information includes the chip temperature and actual operating frequency of the heterogeneous computing chip; The analysis of the runtime state information based on preset capability change detection rules to determine whether the actual available capability of the heterogeneous computing chip has changed includes: Determine the first difference between the actual operating frequency information and the rated operating frequency, and the second difference between the chip temperature and the preset temperature; If the first difference is less than a preset difference and / or the second difference is greater than a preset temperature difference, it is determined that the actual usable capability of the heterogeneous computing chip has changed. If the first difference is equal to a preset difference and / or the second difference is less than or equal to the preset temperature difference, it is determined that the actual usable capability of the heterogeneous computing chip has not changed.

7. The Kubernetes-based computing chip access and management method according to claim 1, characterized in that, The step of responding to a determination that the actual available capability has changed, and updating the unified capability descriptor and the resource information in Kubernetes corresponding to the heterogeneous computing chip based on the changed actual available capability, includes: When it is determined that the actual available computing power of the heterogeneous computing chip changes, a performance coefficient is determined based on the actual operating frequency to characterize the actual available computing power of the heterogeneous computing chip. The actual available computing power after the change is determined based on the performance coefficient, and the unified capability descriptor and the resource information corresponding to the heterogeneous computing chip in Kubernetes are updated based on the actual available computing power after the change.

8. The Kubernetes-based computing chip access and management method according to claim 6, characterized in that, The runtime status information also includes memory error status information and / or driver version information; In response to determining that the actual available capability has changed, the unified capability descriptor and the resource information in Kubernetes corresponding to the heterogeneous computing chip are updated based on the changed actual available capability, including: If the effective available memory capacity of the heterogeneous computing chip changes based on the memory error status information, the memory resource information corresponding to the heterogeneous computing chip is updated. And / or, If it is determined that the driver version of the heterogeneous computing chip has changed based on the driver version information, the capability information of the heterogeneous computing chip is reacquired, and the unified capability descriptor is updated based on the reacquired capability information.

9. A Kubernetes-based computing chip access and management device, characterized in that, include: A general execution container, wherein a multi-layer adaptive probing submodule is used to determine a unified capability descriptor for heterogeneous computing chips in a computing node; The unified capability descriptor is used to describe the device attributes and computing resource capabilities of the heterogeneous computing chip; The unified resource registration core module is used to register the computing resources corresponding to the heterogeneous computing chip to Kubernetes based on the unified capability descriptor, so that Kubernetes can perform unified management and scheduling of the computing resources corresponding to the heterogeneous computing chip. A monitoring integration module is used to acquire runtime status information of the heterogeneous computing chip; The dynamic capability awareness submodule in the general execution container is used to determine, based on the runtime state information, that the actual available capability of the heterogeneous computing chip has changed. The dynamic capability awareness submodule is further configured to, in response to determining that the actual available capability has changed, update the unified capability descriptor and the resource information in Kubernetes corresponding to the heterogeneous computing chip based on the changed actual available capability.

10. A Kubernetes-based data center, characterized in that, include: A control node is used to manage the computing resources in the data center; At least one computing node is communicatively connected to the control node, the computing node comprising at least one heterogeneous computing chip and a general-purpose execution container; the general-purpose execution container is used to execute the Kubernetes-based computing chip access and management method as described in any one of claims 1-8.

Images