Heterogeneous cluster resource capacity detection method and system based on orchestration interface and file system state call

Abstract

The application provides a heterogeneous cluster resource capacity detection method and system based on an arrangement interface and a file system state call, and relates to the technical field of computer cluster resource management. The method comprises the following steps: a cluster management control end calls an arrangement interface to obtain the heterogeneous computing resource capacity of a cluster node in a ready state; file system mounting information of a running environment is analyzed to determine the mounting path of a target distributed file system, and a file system state call is performed to determine the original storage capacity; various capacities are aggregated, and the remaining allocatable amount is calculated in combination with the resource reservation ratio and the allocated quota. When the detection fails or times out, the root directory is detected in fallback, and a degraded state is marked. After cooling, the original mounting path is retried. The application does not need to deploy a capacity monitoring agent on each node, can shorten the detection link, and can improve the real-time performance and operation continuity of mixed resource capacity management.

Description

Technical Field

[0001] This invention relates to the field of computer cluster resource management technology, and more specifically, to a method and system for detecting the resource capacity of heterogeneous clusters based on orchestration interfaces and file system status calls. Background Technology

[0002] As the scale of computational tasks such as training artificial intelligence models increases, computing clusters used to perform these tasks are typically configured with different types of computing resources, such as central processing units (CPUs) and neural network processors, and provide shared storage for multiple computing nodes through a distributed file system. In such clusters, computing resources are generally configured on each computing node, while distributed storage resources are mounted to the corresponding runtime environment through network file systems, user-space file systems, or other storage drivers, thus forming a system architecture in which computing resources and storage resources are managed in different ways.

[0003] To implement resource allocation and task scheduling, the cluster management system needs to obtain capacity information for various resources. For node resources such as CPUs, memory, and heterogeneous accelerators, existing cluster orchestration components can typically provide the corresponding capacity data through node resource status interfaces. However, for distributed file systems that are mounted into the runtime environment, their capacity information is usually not provided uniformly by the orchestration interface as part of the node's computing resource status. Therefore, relying solely on node resource information obtained through the orchestration interface is insufficient to form a cluster capacity view that simultaneously covers heterogeneous computing resources and distributed storage resources.

[0004] One existing approach involves recording logical resource quotas allocated to different tenants or computing tasks in a management database and comparing the total allocated quotas with a pre-configured capacity limit to determine whether further resource allocation is possible. This method relies heavily on the logical quota data in the database. However, when the capacity of the distributed file system changes due to expansion, contraction, changes in storage device status, or changes in mount status, the pre-configured capacity limit may not reflect the file system's currently reported capacity in a timely manner, leading to a discrepancy between the logical quota data and the storage system's capacity information.

[0005] Another existing approach involves deploying monitoring agents or data acquisition components on cluster nodes. These agents periodically collect file system capacity information and report the results to a monitoring platform or time-series database via a network interface. The cluster management system then queries the corresponding data from the monitoring platform or time-series database. This method supplements storage capacity information not directly provided by the orchestration interface, but it requires deploying and maintaining additional monitoring components on relevant nodes, and involves processing steps such as data acquisition, network reporting, data storage, and management system querying. Since monitoring agents typically perform data collection according to a set cycle, and data needs to undergo network transmission and intermediate storage, there may be a time lag between the capacity data obtained by the cluster management system and the current state of the file system. Anomalies in the monitoring components, network communication, or intermediate data storage may also affect the acquisition of capacity data.

[0006] Furthermore, heterogeneous computing resources and distributed storage resources differ in data sources, units of measurement, and acquisition methods. Existing technologies typically collect and manage various types of resources separately. Without corresponding processing of capacity data from different sources, resource reservation configurations, and allocated quotas, cluster management systems struggle to obtain unified, multi-dimensional resource capacity results suitable for resource management.

[0007] Therefore, existing cluster resource capacity detection technologies still need to be improved in order to obtain capacity information of heterogeneous computing resources and distributed storage resources in the same resource management process, reduce the dependence on additional monitoring agents on the node side, and reduce the lag in capacity information caused by collection links and collection cycles. Summary of the Invention

[0008] The present invention aims to solve at least one of the aforementioned technical problems existing in the prior art.

[0009] To this end, the first aspect of the present invention provides a method for detecting the resource capacity of heterogeneous clusters based on orchestration interfaces and file system state calls.

[0010] A second aspect of the present invention provides a heterogeneous cluster resource capacity detection system based on orchestration interface and file system status calls.

[0011] This invention provides a method for detecting the resource capacity of heterogeneous clusters based on orchestration interfaces and file system status calls, applied to a cluster management control terminal. The target distributed file system is mounted in the runtime environment of the cluster management control terminal. The method includes: Call the orchestration interface of the cluster orchestration component to obtain the node resource status information of the cluster nodes in the ready state in the heterogeneous cluster, and summarize the heterogeneous computing resource capacity of the cluster nodes according to the resource type based on the node resource status information. Obtain the file system mount information of the running environment of the cluster management and control terminal, and determine the target mount path of the target distributed file system based on the file system mount information; For the target mount path, execute a file system status system call provided by the operating system to obtain the file system capacity parameters of the target distributed file system, and determine the original storage capacity of the target distributed file system based on the file system capacity parameters; The heterogeneous computing resource capacity and the original storage capacity are aggregated according to resource type to generate heterogeneous cluster resource capacity detection results containing capacity items of different resource types.

[0012] The heterogeneous cluster resource capacity detection method based on orchestration interface and file system state calls according to the above technical solution of the present invention may also have the following additional technical features: In the above technical solution, the step of obtaining the node resource status information of cluster nodes in a ready state in a heterogeneous cluster, and summarizing the heterogeneous computing resource capacity of the cluster nodes according to resource type based on the node resource status information, includes: Iterate through the cluster nodes in the ready state in the heterogeneous cluster; Read the capacity field from the resource status dictionary of each cluster node to obtain the number of CPU cores and the number of memory bytes; Identify the resource items corresponding to the preset heterogeneous computing resource identifier from the node resource status information, obtain the capacity of the heterogeneous acceleration processor, and summarize the central processing unit resources, memory resources and heterogeneous acceleration processor resources respectively.

[0013] In the above technical solution, obtaining the file system mount information of the operating environment of the cluster management and control terminal, and determining the target mount path of the target distributed file system based on the file system mount information, includes: Read the mount information file in the runtime environment; Parse the mount records in the mount information file and extract at least one of the file system type, source path and mount path corresponding to each mount record; Multiple candidate mounting paths were determined based on the extraction results; The mount information file includes at least one of / proc / self / mountinfo and / proc / mounts.

[0014] In the above technical solution, after determining multiple candidate mount paths based on the extraction results, the method further includes: The matching score of each candidate mount path is determined based on at least one of the following: the file system type corresponding to each candidate mount path, the preset keywords contained in the source path, the preset keywords contained in the mount path, and the path prompt information configured by the user. The candidate mounting paths are sorted according to the matching score, and the candidate mounting path with the highest matching score is determined as the target mounting path.

[0015] In the above technical solution, the target distributed file system is mounted to the container where the cluster management and control terminal is located through the storage volume mounting mechanism of the cluster orchestration component, and the target mount path is located within the mount namespace of the container; The file system status system call executed for the target mount path is routed by the operating system's virtual file system layer to the storage driver corresponding to the target distributed file system.

[0016] In the above technical solution, the file system status system call is the statfs call; The file system capacity parameters include the number of data blocks used to determine the original storage capacity and the unit block size corresponding to the number of data blocks. The original storage capacity is determined based on the product of the number of data blocks and the unit block size.

[0017] In the above technical solution, the step of executing a file system status system call provided by the operating system for the target mount path includes: In an independent asynchronous execution unit, the file system status system call is executed for the target mount path, and the execution result of the file system status system call is waited for a preset timeout period; If the execution result is not obtained within the preset timeout period, or if the file system status system call fails or does not return a valid file system capacity parameter, the probe is determined to be a probe failure, and the probe failure is classified according to the corresponding operating system error.

[0018] In the above technical solution, when the detection failure belongs to a preset anomaly type, the method further includes: The target path for executing the file system status system call is changed from the target mount path to the root directory of the runtime environment. The root file system capacity data corresponding to the root directory is obtained, and a degradation detection result containing the root file system capacity data and degradation status identifier is generated. After generating the degradation detection result, a cooling cycle is initiated. In the next detection after the cooling cycle ends, the file system status system call is re-executed for the target mount path. When the file system call for the target mount path is executed successfully and returns a valid file system capacity parameter, the downgrade status flag is cleared and the probing for the target mount path is resumed.

[0019] In the above technical solution, after generating the heterogeneous cluster resource capacity detection result, the method further includes: Obtain the resource reservation ratio corresponding to each resource type, and for each resource type, determine the safe allocable capacity of the resource type by multiplying the original detection capacity of the resource type by the value obtained by subtracting the corresponding resource reservation ratio. Obtain the allocated resource quotas corresponding to different resource types, and for each resource type, subtract the allocated resource quotas of that resource type from the safe allocable capacity of that resource type to obtain the remaining allocable amount of that resource type; The remaining allocable amounts of different resource types are aggregated into resource surplus status objects, and the resource surplus status objects are provided to the display module and / or the scheduling decision module. The resource types include at least two of the following: central processing unit resources, memory resources, heterogeneous accelerated processor resources, and storage resources.

[0020] This invention provides a heterogeneous cluster resource capacity detection system based on orchestration interface and file system state calls, applicable to the heterogeneous cluster resource capacity detection method based on orchestration interface and file system state calls as described in any of the above technical solutions. The system includes: The orchestration interface detection module is configured to call the orchestration interface of the cluster orchestration component to obtain the node resource status information of the cluster nodes in the ready state in the heterogeneous cluster, and to summarize the heterogeneous computing resource capacity of the cluster nodes according to the resource type based on the node resource status information. The mount path determination module is configured to obtain the file system mount information of the system's operating environment and determine the target mount path of the target distributed file system mounted in the operating environment based on the file system mount information. The system call detection module is configured to execute a file system status system call provided by the operating system for the target mount path, obtain the file system capacity parameters of the target distributed file system, and determine the original storage capacity of the target distributed file system based on the file system capacity parameters. The resource capacity aggregation module is configured to aggregate the heterogeneous computing resource capacity and the original storage capacity according to resource type, and generate heterogeneous cluster resource capacity detection results containing capacity items of different resource types.

[0021] In summary, due to the adoption of the above-mentioned technical features, the beneficial effects of the present invention are: This application obtains the computing resource capacity of cluster nodes, such as central processing units, memory, and heterogeneous accelerators, through an orchestration interface, and obtains the storage capacity of the target distributed file system through file system status system calls. This enables the generation of multi-dimensional capacity detection results, including computing resources and storage resources, in the same detection process, facilitating the cluster management and control end to uniformly grasp the capacity status of different types of resources.

[0022] This application executes file system status system calls on the target distributed file system already mounted in the cluster management and control environment. The operating system's virtual file system layer routes these calls to the corresponding storage driver. This method eliminates the need to deploy monitoring agents on each cluster node solely for obtaining storage capacity, reducing the number of node-side monitoring components and the resulting memory consumption, processor polling, and maintenance workload. Compared to capacity acquisition methods that involve periodic collection by monitoring agents, network reporting, intermediate database storage, and system queries, this application allows the cluster management and control end to directly obtain capacity parameters returned by the target distributed file system. This reduces intermediate steps in capacity data acquisition, shortens the capacity information acquisition chain, and reduces the impact of collection cycles and network transmission on the timeliness of capacity data.

[0023] This application identifies candidate mount paths based on the file system mount information of the operating environment, and determines the target mount path by combining the file system type, path information and user-configured path prompts. This reduces the dependence on fixed mount path configuration and adapts to capacity detection needs under different file system types and mount paths.

[0024] This application can process the detected raw capacity and the corresponding resource reservation ratio according to resource type to determine the safe allocable capacity for each resource type, and further combine it with the allocated quota of the same resource type to determine the remaining allocable amount. The resulting resource margin takes into account the currently detected capacity, resource reservation configuration, and allocated quota, which helps to improve the consistency between resource capacity calculation and actual cluster configuration, and reduces the risk of resource quota exceeding the set capacity limit.

[0025] When a system call to the file system status of the target distributed file system fails, times out, or fails to return valid capacity parameters, this application can switch to the root directory for degradation probing and output a degradation status identifier to distinguish between normal and degradation probing results. This avoids continuous blocking or abnormal exit of the capacity probing process due to a single storage probe anomaly. After entering the degradation state, this application can re-probe the original target mount path after the cooldown period ends, and clear the degradation state when valid capacity parameters are obtained, resuming capacity probing of the target distributed file system. This automatic recovery mechanism can reduce the continuous impact of temporary network anomalies or mount status anomalies on the subsequent capacity probing process, improving the operational stability of the capacity probing service.

[0026] Additional aspects and advantages of the invention will become apparent in the following description or may be learned by practice of the invention. Attached Figure Description

[0027] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a schematic diagram of a heterogeneous cluster resource capacity detection method based on orchestration interface and file system status calls according to an embodiment of the present invention. Detailed Implementation

[0028] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0029] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0030] The following reference Figure 1 This invention describes a method and system for detecting the resource capacity of heterogeneous clusters based on orchestration interfaces and file system state calls, according to some embodiments of the present invention.

[0031] This application employs different probing paths adapted to the corresponding resource management methods for heterogeneous computing resources and distributed storage resources within the same cluster management control terminal. Specifically, it obtains the heterogeneous computing resource capacity of cluster nodes through the orchestration interface provided by the cluster orchestration component, and obtains the storage capacity of the target distributed file system through file system status system calls provided by the operating system. Based on this, the capacity data of different resource types are aggregated according to their respective resource dimensions, and combined with the resource reservation ratio and allocated resource quotas, the safe allocable capacity and remaining allocable capacity for each resource type are determined. When the file system status system call for the target distributed file system fails or times out, a degradation rollback mechanism is used to obtain the root file system capacity data and set the degradation status. The original target mount path is then re-probing in the next probe after the cooldown period ends. This results in a multi-dimensional resource capacity probing result covering both heterogeneous computing resources and distributed storage resources.

[0032] The following is combined with Figure 1 The technical solution of this application is described in detail. Figure 1 It illustrates the relationships between the cluster management control terminal, orchestration interface probe branch, file system status call branch, safety margin calculation module, configuration database, resource quota database, and resource capacity probe results and resource margin status objects. Figure 1 The solid lines in the diagram represent the normal detection and capacity calculation process, while the dashed lines between the file system status call module and the degradation rollback module, as well as between the degradation rollback module and the final output result, represent the degradation processing process in case of detection failure or timeout.

[0033] This disclosure provides a method for detecting the resource capacity of a heterogeneous cluster based on orchestration interfaces and file system state calls. The method is applied to a cluster management control terminal, which is used to implement cluster resource capacity detection and scheduling control. The target distributed file system to be detected is mounted in the runtime environment of the cluster management control terminal, enabling processes in the cluster management control terminal to execute file system state system calls on the target distributed file system through accessible target mount paths within the runtime environment.

[0034] In a specific application scenario, a heterogeneous cluster can be an artificial intelligence (AI) model training cluster. Cluster nodes in a heterogeneous cluster can be configured with computing resources such as a central processing unit (CPU), memory, and a neural network processor (NPU). The CPU can adopt an ARM reduced instruction set architecture; the neural network processor is used to perform computational tasks suitable for accelerator processors. Data used for model training can be shared across nodes through a global distributed file system, which is connected to the corresponding runtime environment through mount points.

[0035] like Figure 1 As shown, the cluster management control terminal executes an orchestration interface probing branch and a file system status invocation branch. The orchestration interface probing branch is used to obtain the heterogeneous computing resource capacity of the cluster nodes, and the file system status invocation branch is used to obtain the raw storage capacity of the target distributed file system. In one specific embodiment, the two probing branches are executed in parallel in the same resource capacity probing task, and the probing results are aggregated after each branch has completed the probing of its corresponding resource capacity.

[0036] The heterogeneous computing resource capacity obtained from normal detection and the original storage capacity of the target distributed file system are input into the safety margin calculation module according to resource type. After processing, resource capacity detection results and resource margin status objects are generated. When the file system status call branch detection fails or times out and triggers degradation rollback, the degradation rollback module obtains the root file system capacity data and sets the degradation status flag, forming... Figure 1 The degradation detection results are shown by the dashed line in the middle, and these results are output to the resource capacity detection results and resource remaining status object. The root file system capacity data and the original storage capacity of the target distributed file system have different resource meanings. Therefore, the root file system capacity data is not used as the normal original storage capacity of the target distributed file system in the corresponding safe allocable capacity calculation.

[0037] Specifically, the method may include the following steps.

[0038] Step S101: Call the orchestration interface of the cluster orchestration component to obtain the node resource status information of the cluster nodes in the ready state in the heterogeneous cluster, and summarize the heterogeneous computing resource capacity of the cluster nodes according to the resource type based on the node resource status information.

[0039] The orchestration interface can be an application programming interface (API) provided by the cluster orchestration component. The cluster orchestration component is used to manage cluster nodes, containerized workloads, and corresponding resources. In one specific embodiment, the cluster orchestration component can adopt the Kubernetes container orchestration system, and the cluster management control terminal obtains cluster node and node resource status information through the cluster query interface provided by Kubernetes.

[0040] The cluster management control terminal filters the cluster nodes in the ready state based on the node status information returned by the cluster query interface, and reads the resource status dictionary of the cluster nodes in the ready state. The resource status dictionary may include a Status.Capacity field, and the cluster management control terminal parses the corresponding resource capacity from the Status.Capacity field.

[0041] In this embodiment, heterogeneous computing resources include central processing unit (CPU) resources, memory resources, and heterogeneous accelerator processor resources. The heterogeneous accelerator processor can be a neural network processor. Different resource types use units of measurement corresponding to their resource attributes; for example, CPU resources are represented by the number of cores, memory resources by the number of bytes, and neural network processor resources by the number of devices.

[0042] Step S102: Obtain the file system mount information of the cluster management control terminal running environment, and determine the target mount path of the target distributed file system based on the file system mount information.

[0043] File system mount information indicates the file systems, file system types, file system source paths, and mount paths that are mounted within the cluster management and control terminal's runtime environment. The cluster management and control terminal parses the file system mount information, identifies multiple candidate mount paths related to the target distributed file system, and selects the target mount path from these candidate paths.

[0044] Step S103: Execute a system call for the file system status provided by the operating system for the target mount path, obtain the file system capacity parameters of the target distributed file system, and determine the original storage capacity of the target distributed file system based on the file system capacity parameters.

[0045] The file system status system call is a system call provided by the operating system used to query the status information of the file system at a specified path. The cluster management control terminal executes the file system status system call for a target mounted path that is already mounted and accessible within the runtime environment. The operating system returns file system capacity parameters based on the file system corresponding to the target mounted path. These file system capacity parameters include the number of data blocks used to determine the original storage capacity and the corresponding block size. The cluster management control terminal determines the original storage capacity reported by the target distributed file system through the current mounted path based on the number of data blocks and the block size.

[0046] Step S104: Aggregate the heterogeneous computing resource capacity and the original storage capacity according to resource type to generate heterogeneous cluster resource capacity detection results including capacity items of different resource types.

[0047] During aggregation, CPU resources, memory resources, heterogeneous accelerator processor resources, and storage resources are stored as different resource dimensions. Capacity values ​​across different resource types are not added together. The aggregation organizes the capacity data of each resource type into a unified multi-dimensional resource capacity detection result, enabling subsequent reading, display, and processing of the corresponding capacity data according to resource type.

[0048] The following is in accordance with Figure 1The data flow shown further illustrates the steps described above.

[0049] For orchestration interface probing branches, such as Figure 1 As shown on the left, the cluster management control terminal calls the cluster query interface of the cluster orchestration component through the orchestration interface detection module. The cluster query interface returns cluster nodes and their status information. Based on the returned node status information, the cluster management control terminal filters the cluster nodes that are in the ready state and reads the resource status dictionary of each cluster node.

[0050] In one specific embodiment, the resource status dictionary is the Status.Capacity field in the Kubernetes node information. The status capacity field records the resource capacity reported by the corresponding cluster node. The cluster management control terminal parses the CPU capacity and memory capacity from the status capacity fields of each ready cluster node and accumulates them according to the corresponding resource type.

[0051] For example, suppose the number of cluster nodes in the ready state is N, where N is an integer greater than or equal to 1, and the number of CPU cores reported by the j-th cluster node is Cj, and the number of memory bytes is Mj. Then the original capacity C of the cluster CPU can be expressed as: C=ΣCj, j=1, 2,...,N; The original cluster memory capacity M can be represented as: M=ΣMj, j=1, 2,...,N.

[0052] The above summations are performed within the same resource type. The number of CPU cores and the number of memory bytes are not directly added together.

[0053] For neural network processor resources, the cluster management and control terminal extracts the corresponding number of neural network processors from the node resource status information of the cluster nodes in the ready state according to the preset heterogeneous chip label, and accumulates the number of neural network processors of each cluster node to obtain the original capacity of the neural network processors of the heterogeneous cluster.

[0054] Let Pj be the number of neural network processors in the j-th cluster node. Then the original capacity P of the cluster neural network processors can be expressed as: P=ΣPj, j=1, 2,...,N.

[0055] like Figure 1 As shown, the cluster orchestration component manages nodes 1, 2, and so on, up to node N. Each cluster node can be configured with a central processing unit (CPU), memory, and neural network processor. After obtaining the node resource status information of each node from the cluster orchestration component, the orchestration interface detection module summarizes the CPU, memory, and neural network processor capacities according to resource type, forming... Figure 1 The heterogeneous computing resource capacity is shown.

[0056] For acquiring mount information and identifying candidate mount paths, such as Figure 1 As shown on the right, the file system status call branch resides in the cluster management control terminal's runtime environment. The cluster management control terminal first reads the mount information file in this runtime environment.

[0057] In some embodiments, the mount information file includes at least one of / proc / self / mountinfo and / proc / mounts. The mount information file provides file system mount records that can be recognized within the cluster management control environment.

[0058] The cluster management control terminal parses the mount records in the mount information file line by line. For each mount record, it extracts at least one of the following: file system type, file system source path, and mount path. Based on the extracted file system type, file system source path, or mount path, the cluster management control terminal identifies mount records that match the characteristics of the distributed file system and determines the corresponding mount paths as candidate mount paths.

[0059] The file system type is used to represent the file system or file system protocol corresponding to the mount record. In one specific embodiment, the file system type used to identify candidate mount paths may include the JuiceFS distributed file system, the Network File System (NFS), the Common Internet File System (CIFS), and the Server Message Block version 3 (SMB3).

[0060] The file system source path indicates the source of the file system corresponding to the mount record, while the mount path indicates the access location of the corresponding file system in the cluster management and control environment. The cluster management and control end can determine whether a mount record corresponds to a candidate mount path based on the file system type, the file system source path, or whether there are preset distributed storage characteristics in the mount path.

[0061] In some embodiments, the mount path determination module determines the target mount path from multiple candidate mount paths using a matching scoring method. A matching score is assigned to each candidate mount path, and the score is increased based on the matching conditions satisfied by that candidate mount path.

[0062] Matching criteria may include at least one of the following: the file system type corresponding to the candidate mount path belongs to the preset distributed file system type; the file system source path contains preset keywords; the mount path contains preset keywords; the candidate mount path matches the path prompt information configured by the user.

[0063] In one specific embodiment, when the file system type is JuiceFS, 50 points are added to the current matching score of the corresponding candidate mount path. When the candidate mount path corresponds to other preset distributed file system types, or when the corresponding source path or mount path contains preset keywords, the matching score is added according to the corresponding matching conditions.

[0064] The cluster management control terminal also reads the path hint information configured by the user, namely the configHints configuration path hints. This path hint information is used to provide the mount path determination module with known paths or path keywords related to the target distributed file system. When a candidate mount path matches the path hint information, the matching score of that candidate mount path is increased.

[0065] After matching and scoring each candidate mount path, the mount path determination module sorts the candidate mount paths from highest to lowest matching score, and determines the candidate mount path with the highest matching score as the target mount path for the target distributed file system. This target mount path is then provided to the file system status invocation module for subsequent execution of file system status system calls.

[0066] In one specific embodiment, the target mount path can be / distributed_fs_mount. This path indicates the access location of the target distributed file system in the cluster management and control environment. The actual target mount path is determined based on the parsing and matching scoring results of the mount information file.

[0067] Regarding the mounting and system call paths of the target distributed file system, in some embodiments, the cluster management control terminal runs in a Kubernetes container orchestration environment, and the target distributed file system is mounted to the container where the cluster management control terminal is located through Kubernetes' standard storage volume mounting mechanism.

[0068] Specifically, the target distributed file system can be mounted to the Pod container group where the cluster management control terminal resides via a Persistent Volume Claim (PVC) or a HostPath type storage volume. After mounting is complete, the target mount path appears in the mount namespace of the container where the cluster management control terminal resides and can be identified through the mount records in / proc / self / mountinfo or / proc / mounts.

[0069] The cluster management control terminal does not need to obtain the file system capacity through the HostPID shared host process identifier namespace configuration or the HostNetwork shared host network namespace configuration, nor does it need to adopt privileged container mode to perform file system status queries. The cluster management control terminal initiates standard file system status system calls for target mount paths that have been mounted to their respective containers and are accessible within their runtime environment.

[0070] In the Linux operating system, the file system status system call can be the statfs file system status statistics call, which conforms to the Portable Operating System Interface (POSIX).

[0071] When the cluster management control terminal executes a statfs call for a target mount path, the operating system's Virtual File System (VFS) layer determines the corresponding file system based on the target mount path and routes the call to the storage driver corresponding to the target distributed file system. The storage driver can be a Filesystem in Userspace (FUSE) driver or a file system module within the operating system kernel.

[0072] Therefore, the file system status call module invokes a target mount path that is already mounted and accessible in the cluster management control environment. The cluster management control obtains the file system capacity parameters returned by the target distributed file system through this target mount path.

[0073] In some embodiments, to determine the initial storage capacity, the file system status invocation module performs a statfs call on the target mount path and receives file system status information returned by the operating system. This file system status information includes the number of data blocks used to determine the initial storage capacity of the target distributed file system, and the block size corresponding to that number of data blocks.

[0074] Let the number of data blocks returned by the target distributed file system be B, the block size be S, and the original storage capacity of the target distributed file system be Cstorage. Then: Cstorage = B × S.

[0075] The number of data blocks, B, and the unit block size, S, are corresponding capacity parameters returned by the file system status system call. The calculated raw storage capacity can be expressed in bytes so that it can be correlated with the storage resource reservation ratio and allocated storage quota in subsequent processing.

[0076] The original storage capacity in this application is the file system capacity reported by the target distributed file system through the target mount path and file system status system calls. It is not the sum of the physical media capacities obtained by separately detecting each physical storage medium in the distributed storage system, nor does it represent the actual physical space that the target distributed file system is not currently using.

[0077] For timeout and error handling of file system status calls, in some embodiments, to avoid file system mounting errors or network connection failures causing capacity probe requests to remain in a waiting state for an extended period, the file system status call module executes the statfs call in an independent coroutine. This independent coroutine constitutes an independent asynchronous execution unit for performing file system status system calls.

[0078] The cluster management control terminal sets a preset timeout for this file system status system call through a context timeout mechanism, and waits for the execution result within the preset timeout period. In one specific embodiment, the preset timeout period is 1.5 seconds.

[0079] If a valid file system capacity parameter is obtained within the preset timeout period, the call is considered a successful probe; if no result is obtained after the preset timeout period, the call is considered a probe timeout. After a probe timeout occurs, the upper-layer capacity probe process no longer waits for the return result of the current file system status system call and instead enters the corresponding error handling process.

[0080] In addition to probe timeouts, the cluster management control terminal also captures operating system errors returned by file system status system calls and categorizes them according to error type. Preset exception types can include input / output errors, ETIMEDOUT timeout errors, ENOTTON connection failure errors, and ECONNRESET connection reset errors. Errors resulting from a non-existent target path or access denied to the target path can also be identified as probe failures.

[0081] When a file system status system call fails, times out, or fails to return valid file system capacity parameters, the cluster management control terminal classifies the capacity probe as a failure. For probe failures belonging to preset exception types, a fallback process is triggered.

[0082] For downgrade rollback and automatic recovery, such as Figure 1 As shown, when the file system status call module fails to detect or times out, it enters the degradation rollback module through the exception handling path shown by the dashed line.

[0083] In some embodiments, the fallback module executes the root directory fallback logic, namely the fallback To Root logic, which switches the target path of the file system state system call from the target mount path of the target distributed file system to the root directory " / " of the cluster management control terminal runtime environment.

[0084] The downgrade rollback module performs a file system call to the root directory to obtain the root file system capacity data corresponding to the root directory. This root file system capacity data serves as backup capacity data in the downgraded state. When outputting the root file system capacity data, the downgrade rollback module also sets a downgrade status flag, forming a downgrade detection result that includes both the root file system capacity data and the downgrade status flag.

[0085] The resource meaning of the root file system capacity data differs from the original storage capacity of the target distributed file system. Therefore, the degradation detection result indicates that the current file system state call branch is in a degradation state, but does not mean that the root file system capacity can replace the target distributed file system capacity. Figure 1 The degradation detection results output by the degradation rollback module are directly entered into the resource capacity detection results and resource margin status object along the dotted path, and are not entered into the safety margin calculation module as the normal original storage capacity of the target distributed file system.

[0086] Degradation rollback is used to prevent the capacity probing process from continuously waiting or being interrupted because the target distributed file system cannot return capacity parameters normally. This allows the cluster management control terminal to output capacity probing results with a degraded status.

[0087] After generating the degradation detection results, the cluster management control terminal initiates a cooling-off period. During the cooling-off period, detection requests for storage capacity directly return the root file system capacity data and degradation status identifier, and no longer repeatedly initiate file system status system calls for abnormal target mount paths.

[0088] After the cooling-off period, the cluster management control terminal re-executes the file system status system call for the original target mount path during the next capacity probe. If the file system status system call does not return an error and returns valid file system capacity parameters, the degraded status flag is cleared, and normal capacity probe for the original target mount path resumes. The degraded status flag is only cleared if the file system status system call executed for the original target mount path does not return an error and returns valid file system capacity parameters; otherwise, the degraded status flag is not cleared.

[0089] For calculations of safe allocable capacity and resource reserves, such as Figure 1As shown in the lower section, the heterogeneous computing resource capacity obtained from normal detection and the original storage capacity of the target distributed file system are respectively input into the safety margin calculation module. The configuration database provides the safety margin calculation module with the resource reservation ratio corresponding to different resource types, and the resource quota database provides the safety margin calculation module with the allocated resource quota corresponding to different resource types.

[0090] In some embodiments, the configuration database stores the central processing unit (CPU) resource reservation ratio, memory resource reservation ratio, heterogeneous accelerator processor (HEAT) resource reservation ratio, and storage resource reservation ratio. Different resource types can have their own corresponding resource reservation ratios set.

[0091] Let Ci be the initial detection capacity of resource type i, ρi be the resource reservation ratio corresponding to resource type i, and Cisafe be the safe allocable capacity of resource type i. Then: Cisafe = Ci × (1 - ρi).

[0092] Here, resource type i can be a central processing unit resource, memory resource, neural network processor resource, or storage resource. The safety margin calculation module calculates separately for each resource type and does not merge capacity values ​​between resource types with different units of measurement.

[0093] For example, the secure allocable capacity of the CPU is calculated based on the original capacity of the CPU and the CPU resource reservation ratio; the secure allocable capacity of memory is calculated based on the original capacity of memory and the memory resource reservation ratio; the secure allocable capacity of the neural network processor is calculated based on the original capacity of the neural network processor and the neural network processor resource reservation ratio; and the secure allocable capacity of storage is calculated based on the original storage capacity of the target distributed file system and the storage resource reservation ratio.

[0094] The resource quota database records the allocated resource quotas for each resource type that have been distributed to the multi-tenant environment. The allocated resource quotas include allocated CPU quotas, allocated memory quotas, allocated neural network processor quotas, and allocated storage quotas.

[0095] Let Ai be the allocated resource quota for resource type i, and Ri be the remaining allocable amount for resource type i. Then: Ri = Cisafe - Ai.

[0096] Therefore, the remaining allocatable amount of a CPU is obtained by subtracting the allocated CPU quota from the secure allocatable amount of a CPU; the remaining allocatable amount of memory is obtained by subtracting the allocated memory quota from the secure allocatable amount of memory; the remaining allocatable amount of a neural network processor is obtained by subtracting the allocated neural network processor quota from the secure allocatable amount of a neural network processor; and the remaining allocatable amount of storage is obtained by subtracting the allocated storage quota from the secure allocatable amount of storage.

[0097] In the above calculations, the initial probe capacity, resource reservation ratio, and allocated resource quota are corresponded to the same resource type. The CPU capacity is not subtracted from the storage quota, and the memory capacity is not subtracted from the neural network processor quota, thus ensuring that the capacity data in each resource dimension has a consistent measurement meaning.

[0098] In some embodiments, the safety margin calculation module aggregates the remaining allocable resources of different resource types according to resource type to generate a resource margin status object. The resource margin status object includes resource capacity items corresponding to central processing unit resources, memory resources, neural network processor resources, and storage resources.

[0099] like Figure 1 As shown, the resource reserve status object can be provided to the front-end interface to display the capacity and remaining allocable amount of different resource types; it can also be provided to the scheduling decision module as data input for subsequent resource allocation or task scheduling. The safety reserve calculation module outputs a data set of different resource dimensions, without converting central processing unit, memory, neural network processor, and storage resources into a single capacity value.

[0100] For degradation detection results, the resource capacity detection results and resource remaining status objects include a degradation status identifier. The front-end interface or scheduling decision module can identify, based on the degradation status identifier, that the currently output storage capacity data is the root file system capacity data, rather than the normal original storage capacity of the target distributed file system.

[0101] In one specific embodiment, after the cluster management control terminal initiates a heterogeneous cluster resource capacity probe task, it calls the cluster query interface of the cluster orchestration component through the probe branch of the orchestration interface, and reads / proc / self / mountinfo and / or / proc / mounts through the file system status call branch. The two probe branches can be executed in parallel.

[0102] In the orchestration interface probing branch, the cluster management control terminal filters cluster nodes in a ready state based on the node status information returned by the cluster query interface, reads the Status.Capacity field of each cluster node, obtains the number of CPU cores and memory bytes, and obtains the number of neural network processors based on the preset heterogeneous chip tag identifier. Subsequently, the number of CPU cores, memory bytes, and neural network processors are summarized according to resource type to obtain the heterogeneous computing resource capacity.

[0103] In the file system status call branch, the cluster management control terminal parses the mount records in the mount information file line by line, extracts the file system type, file system source path and mount path, scores and sorts the candidate mount paths according to the file system type, path keywords and configHints configuration path hints, and determines the candidate mount path with the highest matching score as the target mount path.

[0104] The file system status call module executes a statfs call against the target mount path in a separate coroutine. If a valid file system capacity parameter is obtained within a preset timeout period, the original storage capacity of the target distributed file system is determined by the product of the number of data blocks and the size of each block.

[0105] The heterogeneous computing resource capacity obtained from the orchestration interface probe branch and the raw storage capacity obtained from the file system status call branch are respectively input into the safety margin calculation module. The safety margin calculation module reads the resource reservation ratio from the configuration database and the allocated resource quota from the resource quota database, calculates the safe allocable capacity and the remaining allocable amount according to the resource type, and forms the resource capacity probe results and resource margin status objects.

[0106] If the file system status call module fails, times out, or fails to return valid capacity parameters, and the corresponding detection failure falls under a preset exception type, the degradation rollback module switches the detection target to the root directory, obtains the root file system capacity data, and generates a degradation detection result including the root file system capacity data and a degradation status identifier. The degradation detection result follows... Figure 1 The dashed path shown outputs to the resource capacity detection results and resource balance status objects.

[0107] After the cooling-off period ends, the statfs call is re-executed against the original target mount path in the next capacity probe; if the call does not return an error and returns a valid file system capacity parameter, the downgrade status flag is cleared and normal probe is resumed.

[0108] This embodiment obtains the heterogeneous computing resource capacity by calling the orchestration interface of the cluster orchestration component through the cluster management control terminal, and obtains the distributed file system capacity by executing file system status system calls on the mounted paths through the cluster management control terminal. Therefore, compared with the node-side monitoring agent method, this embodiment does not require the deployment of a resident monitoring agent for storage capacity collection on each cluster node.

[0109] When using a third-party monitoring agent to obtain capacity data, the process typically involves several steps: periodic collection by the monitoring agent, exposure of the interface via Hypertext Transfer Protocol (HTTP), retrieval by the monitoring center, writing to a time-series database, and querying by the cluster management control terminal. In this embodiment, the file system status call branch is executed directly by the cluster management control terminal using a statfs call targeting the target mount path.

[0110] In a specific engineering estimation example, the memory usage of a typical monitoring agent process can range from 50MB to 200MB, and the typical monitoring collection cycle can range from 15 seconds to 60 seconds. Under normal operating conditions and target distributed file system status, the execution time of a statfs call can typically be in the range of microseconds to milliseconds, for example, less than 5 milliseconds. The above values ​​are estimates of runtime overhead and query latency. Actual values ​​are affected by factors such as operating system, file system type, network status, cluster size, and hardware configuration, and are not considered necessary conditions for implementing the technical solution of this application.

[0111] Other embodiments of this disclosure also provide a heterogeneous cluster resource capacity detection system based on orchestration interface and file system state calls. See further... Figure 1 The system is located in the cluster management control terminal, which is used to implement resource capacity detection and scheduling control. The system includes an orchestration interface detection module, a mounting path determination module, a system call detection module, and a resource capacity aggregation module. The system call detection module is located in... Figure 1 The middle section shows the file system status call module.

[0112] The orchestration interface detection module is used to call the orchestration interface of the cluster orchestration component to obtain the node resource status information of the cluster nodes in the ready state in the heterogeneous cluster, and to summarize the heterogeneous computing resource capacity of the cluster nodes according to the resource type based on the node resource status information.

[0113] Specifically, the orchestration interface detection module traverses the cluster nodes in the ready state, reads the capacity field in the resource status dictionary of each cluster node, and obtains the number of CPU cores and memory bytes. At the same time, it obtains the heterogeneous accelerator capacity according to the preset heterogeneous computing resource identifier, and summarizes the CPU resources, memory resources and heterogeneous accelerator resources respectively.

[0114] The mount path determination module is used to obtain the file system mount information of the system operating environment, parse the candidate mount paths according to the file system mount information, and determine the target mount path of the target distributed file system from multiple candidate mount paths.

[0115] Specifically, the mount path determination module reads at least one of / proc / self / mountinfo and / proc / mounts, parses the file system type, file system source path and mount path in the corresponding mount record, determines the matching score of the candidate mount path based on at least one of the file system type, source path keyword, mount path keyword and configHints configuration path hints, and selects the candidate mount path with the highest matching score as the target mount path.

[0116] The system call detection module is used to execute a file system call provided by the operating system for the target mount path, obtain the file system capacity parameters of the target distributed file system, and determine the original storage capacity of the target distributed file system based on the file system capacity parameters.

[0117] In one specific embodiment, the system call detection module executes a statfs call in an independent coroutine and waits for the call result within a preset timeout period. The product of the number of data blocks returned by the statfs call and the size of the unit block is used to determine the original storage capacity of the target distributed file system.

[0118] The resource capacity aggregation module aggregates heterogeneous computing resource capacity and raw storage capacity according to resource type, generating heterogeneous cluster resource capacity detection results that include capacity items for different resource types. The aggregation maintains the individual resource dimensions of the central processing unit, memory, heterogeneous accelerator processors, and storage resources, and does not directly add up resource capacities of different units of measurement.

[0119] In some embodiments, the system further includes a degradation rollback module. The degradation rollback module is used to switch the probe path to the root directory, obtain the root file system capacity data, generate a degradation probe result including the root file system capacity data and a degradation status identifier, and re-probe the original target mount path in the next probe after the cooling period ends when the file system status system call fails, times out, or does not return a valid capacity parameter, and the corresponding probe failure belongs to a preset exception type.

[0120] In some embodiments, the system further includes a safety margin calculation module. The safety margin calculation module receives the heterogeneous computing resource capacity obtained from the orchestration interface probe branch and the raw storage capacity obtained from the file system status call branch normal probe, respectively, and obtains the resource reservation ratio corresponding to each resource type from the configuration database and the allocated resource quota corresponding to each resource type from the resource quota database.

[0121] For each resource type, the safety margin calculation module determines the safe allocable capacity of that resource type by multiplying the original detection capacity of that resource type by a value obtained by subtracting the corresponding resource reservation ratio; then, it subtracts the allocated resource quota of that resource type from the safe allocable capacity of that resource type to obtain the remaining allocable amount of that resource type.

[0122] The safety margin calculation module aggregates the remaining allocable resources of different resource types into a resource margin status object, and provides the resource margin status object to the front-end interface and / or scheduling decision module. When the file system status call branch is in a degraded state, the system outputs the degrade detection result, including root file system capacity data and degraded status identifier, to the resource capacity detection result and the resource margin status object, so that the front-end interface or scheduling decision module can distinguish between normal storage capacity detection results and degraded detection results.

[0123] The above-mentioned functional modules are used to perform the corresponding steps in the heterogeneous cluster resource capacity detection method described in the above embodiments. The input data, processing procedures and output data of each functional module are described in the corresponding descriptions in the method embodiments.

[0124] In this specification, the illustrative expressions of the terms used do not necessarily refer to the same embodiments or examples. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0125] Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention shall be included within the scope of protection of this invention.

Claims

1. A method for detecting the resource capacity of a heterogeneous cluster based on orchestration interfaces and file system state calls, characterized in that, The method, applied to a cluster management and control terminal where a target distributed file system is mounted in the operating environment of the cluster management and control terminal, includes: Call the orchestration interface of the cluster orchestration component to obtain the node resource status information of the cluster nodes in the ready state in the heterogeneous cluster, and summarize the heterogeneous computing resource capacity of the cluster nodes according to the resource type based on the node resource status information. Obtain the file system mount information of the running environment of the cluster management and control terminal, and determine the target mount path of the target distributed file system based on the file system mount information; For the target mount path, execute a file system status system call provided by the operating system to obtain the file system capacity parameters of the target distributed file system, and determine the original storage capacity of the target distributed file system based on the file system capacity parameters; and The heterogeneous computing resource capacity and the original storage capacity are aggregated according to resource type to generate heterogeneous cluster resource capacity detection results containing capacity items of different resource types.

2. The heterogeneous cluster resource capacity detection method based on orchestration interface and file system state calls according to claim 1, characterized in that, The step of obtaining the node resource status information of cluster nodes in a ready state in a heterogeneous cluster, and summarizing the heterogeneous computing resource capacity of the cluster nodes according to resource type based on the node resource status information, includes: Iterate through the cluster nodes in the ready state in the heterogeneous cluster; Read the capacity field from the resource status dictionary of each cluster node to obtain the number of CPU cores and memory bytes; and Identify the resource items corresponding to the preset heterogeneous computing resource identifier from the node resource status information, obtain the capacity of the heterogeneous acceleration processor, and summarize the central processing unit resources, memory resources and heterogeneous acceleration processor resources respectively.

3. The heterogeneous cluster resource capacity detection method based on orchestration interface and file system state calls according to claim 1, characterized in that, The step of obtaining the file system mount information of the operating environment of the cluster management and control terminal, and determining the target mount path of the target distributed file system based on the file system mount information, includes: Read the mount information file in the runtime environment; Parse the mount records in the mount information file, and extract at least one of the following: file system type, source path, and mount path corresponding to each mount record; and Multiple candidate mounting paths were determined based on the extraction results; The mount information file includes at least one of / proc / self / mountinfo and / proc / mounts.

4. The heterogeneous cluster resource capacity detection method based on orchestration interface and file system state calls according to claim 3, characterized in that, After determining multiple candidate mount paths based on the extraction results, the process also includes: The matching score of each candidate mount path is determined based on at least one of the following: the file system type corresponding to each candidate mount path, the preset keywords contained in the source path, the preset keywords contained in the mount path, and the path prompt information configured by the user. The candidate mounting paths are sorted according to the matching score, and the candidate mounting path with the highest matching score is determined as the target mounting path.

5. The heterogeneous cluster resource capacity detection method based on orchestration interface and file system state calls according to claim 1, characterized in that, The target distributed file system is mounted to the container where the cluster management and control terminal is located through the storage volume mounting mechanism of the cluster orchestration component, and the target mount path is located within the mount namespace of the container; The file system status system call executed for the target mount path is routed by the operating system's virtual file system layer to the storage driver corresponding to the target distributed file system.

6. The heterogeneous cluster resource capacity detection method based on orchestration interface and file system state calls according to claim 1, characterized in that, The file system status system call is the statfs call; The file system capacity parameters include the number of data blocks used to determine the original storage capacity and the unit block size corresponding to the number of data blocks. The original storage capacity is determined based on the product of the number of data blocks and the unit block size.

7. The heterogeneous cluster resource capacity detection method based on orchestration interface and file system state calls according to claim 6, characterized in that, The execution of file system status system calls provided by the operating system for the target mount path includes: In an independent asynchronous execution unit, the file system status system call is executed for the target mount path, and the execution result of the file system status system call is waited for a preset timeout period; If the execution result is not obtained within the preset timeout period, or if the file system status system call fails or does not return a valid file system capacity parameter, the probe is determined to be a probe failure, and the probe failure is classified according to the corresponding operating system error.

8. The heterogeneous cluster resource capacity detection method based on orchestration interface and file system state calls according to claim 7, characterized in that, When the detection failure belongs to a preset anomaly type, the method further includes: The target path for executing the file system status system call is changed from the target mount path to the root directory of the runtime environment. The root file system capacity data corresponding to the root directory is obtained, and a degradation detection result containing the root file system capacity data and degradation status identifier is generated. After generating the degradation detection results, a cooling cycle is initiated. In the next detection after the cooling cycle ends, the file system state system call is re-executed for the target mount path; and When the file system call for the target mount path is executed successfully and returns a valid file system capacity parameter, the downgrade status flag is cleared and the probing for the target mount path is resumed.

9. The heterogeneous cluster resource capacity detection method based on orchestration interface and file system state calls according to claim 1, characterized in that, After generating the heterogeneous cluster resource capacity detection results, the method further includes: Obtain the resource reservation ratio corresponding to each resource type, and for each resource type, determine the safe allocable capacity of the resource type by multiplying the original detection capacity of the resource type by the value obtained by subtracting the corresponding resource reservation ratio. Obtain the allocated resource quotas for each resource type, and for each resource type, subtract the allocated resource quotas from the safe allocable capacity of that resource type to obtain the remaining allocable capacity for that resource type; and The remaining allocable amounts of different resource types are aggregated into resource surplus status objects, and the resource surplus status objects are provided to the display module and / or the scheduling decision module. The resource types include at least two of the following: central processing unit resources, memory resources, heterogeneous accelerated processor resources, and storage resources.

10. A heterogeneous cluster resource capacity detection system based on orchestration interface and file system state calls, characterized in that, The system, which is applied to the heterogeneous cluster resource capacity detection method based on orchestration interface and file system state calls as described in any one of claims 1 to 9, comprises: The orchestration interface detection module is configured to call the orchestration interface of the cluster orchestration component to obtain the node resource status information of the cluster nodes in the ready state in the heterogeneous cluster, and to summarize the heterogeneous computing resource capacity of the cluster nodes according to the resource type based on the node resource status information. The mount path determination module is configured to obtain the file system mount information of the system's operating environment and determine the target mount path of the target distributed file system mounted in the operating environment based on the file system mount information. The system call detection module is configured to execute a file system call provided by the operating system for the target mount path, obtain the file system capacity parameters of the target distributed file system, and determine the original storage capacity of the target distributed file system based on the file system capacity parameters; and The resource capacity aggregation module is configured to aggregate the heterogeneous computing resource capacity and the original storage capacity according to resource type, and generate heterogeneous cluster resource capacity detection results containing capacity items of different resource types.

Images