Test scheduling method for health monitoring, electronic device, and storage medium

Abstract

The present disclosure relates to a test scheduling method for health monitoring, an electronic device and a storage medium. The method comprises: constructing a plurality of test tasks according to a plurality of test items and a plurality of nodes in a computing power facility to be monitored; wherein each test task is composed of a test item and a node; for each test task, obtaining a test index value of the corresponding test task based on a preset index set; wherein the test index value represents the test priority of the corresponding test task; and performing scheduling execution of the test task according to the test index value of each test task. Through the method of the present disclosure, unnecessary test resource occupation can be reduced while effectively monitoring, and the overall test efficiency can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computing power facility testing, and more particularly, to a test scheduling method for health monitoring, an electronic device, and a non-transitory computer-readable storage medium. BACKGROUND

[0002] Computing power facilities are hardware and software resource systems capable of performing computing tasks, which are widely used in artificial intelligence, big data, high-performance computing, cloud computing, and distributed systems. To ensure the stability, reliability, and sustainable operation of computing power facilities, it is crucial to continuously and efficiently monitor their health. However, current mainstream health monitoring methods mainly rely on periodic polling detection or fixed threshold alarm strategies, both of which have obvious limitations.

[0003] On the one hand, traditional periodic polling methods usually perform full-quantity testing on all nodes in the computing power facility at fixed time intervals. Although this method can comprehensively cover system states, it often causes significant computing power and resource overhead due to high testing frequency and wide coverage, especially in large-scale computing clusters, where polling costs cannot be ignored. On the other hand, fixed threshold alarm strategies trigger alarms or tests when hardware indicators such as temperature, video memory usage, and power consumption exceed preset thresholds. This method reduces unnecessary testing overhead to some extent, but it is difficult to detect potential faults or performance degradation in advance. This often leads to faults or abnormalities developing to a serious stage when the alarm is actually triggered, increasing the risk of system operation and reducing the initiative and foresight of health management. SUMMARY

[0004] One purpose of the embodiments of the present disclosure is to provide a new technical solution for health monitoring of computing power facilities, achieving proactive scheduling and efficient execution of test tasks.

[0005] According to a first aspect of the present disclosure, a test scheduling method for health monitoring is provided, which comprises:

[0006] According to a set of multiple test items and multiple nodes being monitored in the computing power facility, multiple test tasks are constructed; wherein each test task is composed of a test item and a node;

[0007] For each test task, a test index value of the corresponding test task is obtained based on a preset index set; wherein the test index value represents the test priority of the corresponding test task;

[0008] According to the test index value of each test task, the scheduling and execution of the test task are performed.

[0009] Optionally, the preset indicator set includes a failure probability indicator of a test task, and determining the failure probability indicator value of the test task includes:

[0010] obtaining associated data of the test task; wherein the associated data includes data associated with at least one of nodes and test items in the test task;

[0011] generating a feature vector of the test task based on the associated data;

[0012] calling a prediction model to predict the failure probability indicator value of the test task based on the feature vector, wherein the prediction model is trained to establish a mapping relationship from the feature vector of the test task to the failure probability of the test task.

[0013] Optionally, the feature vector includes a structured feature and a time sequence feature, and the calling of the prediction model to predict the failure probability indicator value of the test task based on the feature vector includes:

[0014] calling a first prediction model to generate a first failure probability based on the structured feature;

[0015] calling a second prediction model to generate a second failure probability based on the time sequence feature;

[0016] fusing the first failure probability and the second failure probability to obtain the failure probability indicator value of the test task.

[0017] Optionally, the plurality of test tasks includes a cold start task satisfying a cold start condition, the cold start condition including that at least one of test items and nodes does not exist historical test data; and the generating of the feature vector of the cold start task based on the associated data of the cold start task includes:

[0018] calling at least one of a feature extraction model and a knowledge graph to generate the feature vector of the cold start task based on the associated data of the cold start task;

[0019] wherein the feature extraction model is trained to generate at least part of the feature of the cold start task based on the associated data of the cold start task, and the knowledge graph is constructed based on an association relationship between node hardware configurations, test items and fault information, and a target path in the knowledge graph matching the cold start task is used to generate at least part of the feature of the cold start task.

[0020] Optionally, the preset index set comprises at least two types of indexes among a first type of index, a second type of index and a third type of index; the first type of index is used to evaluate a contribution degree of the combination of the test item and the node to the test priority, the second type of index is used to evaluate a contribution degree of an inherent attribute of the test item to the test priority, and the third type of index is used to evaluate a contribution degree of an inherent attribute of the node to the test priority.

[0021] Optionally, the first type of index comprises at least part of indexes of a failure probability index of the test task and a timeliness index of the test task, and the timeliness index represents a time length from a current time to a time of a last test success; or,

[0022] The second type of index comprises at least part of indexes of a business influence index of the test item and a resource cost index of the test item; the business influence index represents an influence degree of a node capability verified by the test item on the business, and the resource cost index represents a proportion of resource consumption of executing the test item; or,

[0023] The third type of index comprises a reliability attenuation index of the node, and the reliability attenuation index is related to a service time length of the node.

[0024] Optionally, the obtaining, for each test task, a test index value of the corresponding test task based on the preset index set comprises:

[0025] respectively determining a single index value of the test task for each index in the preset index set;

[0026] determining a weight of each index in the preset index set; wherein the weight of at least part of indexes in the preset index set is obtained based on a weight probability distribution of a corresponding index, and the weight probability distribution of the corresponding index is updated based on historical test data;

[0027] obtaining the test index value of the test task based on the single index value of each index and the weight of each index.

[0028] Optionally, the scheduling and executing of the test task according to the test index value of each test task comprises:

[0029] distributing and executing the test task according to the test index value of each test task and a set scheduling condition as a constraint.

[0030] Optionally, after the scheduling and executing of the test task, the method further comprises:

[0031] obtaining test data of executing the test task;

[0032] update an adaptive parameter related to a test indicator value of a test task obtained, and / or update an adaptive parameter related to scheduling, based on the test data.

[0033] According to a second aspect of the present disclosure, an electronic device is also provided, which comprises:

[0034] at least one processor; and

[0035] a memory connected with the at least one processor in communication;

[0036] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to the first aspect of the present disclosure.

[0037] According to a third aspect of the present disclosure, a non-transitory computer readable storage medium is also provided, which stores computer instructions for causing the computer to perform the method according to the first aspect of the present disclosure.

[0038] The embodiments of the present disclosure generate test tasks by combining test items and nodes of a computing power facility, and evaluate the test priority of each test task based on a preset indicator set, so as to perform scheduling of the test tasks based on the test priority. On the one hand, the embodiments of the present disclosure can realize continuous monitoring of the health status of the computing power facility by performing active testing on the nodes in the computing power facility, thereby effectively reducing the operation risk. On the other hand, by dynamically evaluating the priority of the test tasks and scheduling based thereon, unnecessary resource occupation can be reduced while ensuring the health monitoring effect, thereby improving the overall test efficiency.

[0039] The features and advantages of embodiments of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of embodiments of the present disclosure.

[0041] Figure 1 is a schematic diagram of a structure of a computing power facility provided by the present disclosure;

[0042] Figure 2 is a flowchart of a test scheduling method according to some embodiments;

[0043] Figure 3 is a flowchart of predicting a failure probability of a test task according to some embodiments;

[0044] Figure 4 is a flowchart of a test scheduling method according to some embodiments;

[0045] Figure 5 is a schematic diagram of a health monitoring system according to some embodiments;

[0046] Figure 6 is a schematic diagram of a hardware structure of an electronic device according to some embodiments. DETAILED DESCRIPTION

[0047] Various exemplary embodiments of the present specification will now be described in detail with reference to the accompanying drawings.

[0048] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the embodiments of the present specification, its application or uses.

[0049] It should be noted that similar reference numbers and letters refer to similar items in the following drawings, and therefore, once an item is defined in one drawing, it need not be discussed further in subsequent drawings.

[0050] The present disclosure relates to a technical solution for health monitoring of nodes in a computing power facility. As shown in Figure 1 , the computing power facility includes a plurality of nodes, denoted as node n1, node n2, …, node nX, where X is a positive integer. These nodes are mainly computing nodes with computing power, each of which contains at least one type of processor, such as at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a neural processing unit (NPU). In addition, each node is usually also configured with a communication interface for realizing data communication between nodes, such as an Ethernet interface, a fiber communication interface, or other high-speed interconnection interface. In some nodes, a local memory can be further included for storing temporary or long-term data required for node operation. For example, as shown in Figure 1 , node n2 includes at least one processor 201, at least one memory 202, and a communication interface 203. When node n2 contains multiple processors, these processors can be of the same type or of different types to adapt to different computing task requirements.

[0051] In some embodiments, the computing power facility can also include a shared storage system, and the storage nodes in the storage system can be shared by some or all nodes in the computing power facility, for providing centralized or distributed storage services.

[0052] In the embodiments of the present disclosure, the computing power facility further comprises a health monitoring system 100 for monitoring and evaluating the health status of the nodes in the computing power facility. The health monitoring system 100 can be deployed in a flexible manner, for example, it can be deployed on one or more specific nodes in the computing power cluster, it can be deployed independently of the computing power cluster, such as being deployed on a dedicated monitoring node or edge device, or it can also be deployed in a hybrid mode, that is, part of the functions are deployed in the nodes of the computing power cluster, such as data collection functions, and part of the functions are deployed in the cloud, such as the cloud being responsible for health scoring and task scheduling, etc.

[0053] As shown in the figure, the health monitoring system 100 can include at least one processor 101 and at least one memory 102. The processor 101 is configured to execute a computer program, which can be written based on a plurality of instruction set architectures; the memory 102 is configured to store the executable program, and the type thereof can include Read-Only Memory (ROM), Random Access Memory (RAM), hard disk, etc. non-volatile storage medium. In addition, the health monitoring system 100 can also include a communication interface 103 for realizing data communication between itself and the nodes inside the computing power facility, as well as with external systems, supporting the collection, transmission of monitoring data and feedback of analysis results.

[0054] The following will be described in combination with Figure 1 a schematic computing power facility to illustrate various embodiments of the present disclosure.

[0055] <First embodiment>

[0056] Figure 2 A flowchart of a test scheduling method for health monitoring according to some embodiments is shown. The method can be implemented by the health monitoring system 100 in the computing power facility. As shown in Figure 1 , the method of the present embodiment can include the following steps S210 to S230. Figure 2

[0057] Step S210, according to a plurality of test items and a plurality of nodes in the computing power facility, a plurality of test tasks are constructed.

[0058] ​In this embodiment, the test set for health monitoring can be set for the computing power facility according to the business field and structural characteristics of the computing power facility. The test set includes a plurality of test items, and the test item is an executable detection unit designed for a possible failure mode or performance bottleneck of the computing power facility. The test set includes, for example, a test item for monitoring the correctness of the computing function of a node, a test item for monitoring whether the Peripheral Component Interconnect Express (PCIe) link of the node is normal, a test item for monitoring whether the temperature reading function of the node is normal, a test item for performing GPU memory diagnosis test, a test item for stress testing the GPU, a test item for monitoring network bandwidth and delay, a disk I / O throughput test item, and the like, which are not limited herein.

[0059] In this embodiment, a plurality of test tasks can be constructed by combining the test items in the test set and the nodes of the computing power facility that need to be monitored, that is, each test task is composed of a test item in the test set and a node of the computing power facility. The node in this embodiment refers to an entity in the computing power facility that needs to be monitored. These nodes can be nodes with computing power in the computing power cluster, such as servers equipped with GPUs or CPUs. These nodes can also include other types of nodes in the computing power facility, such as storage nodes, and the like, which are not limited herein.

[0060] Each node can be combined with the test items in the test set that are related to the function of the node to obtain a set of test tasks that need to be scheduled and executed.

[0061] For example, node n1 is combined with test item t1 and test item t2 to construct first test task T1(n1, t1) and second test task T2(n1, t2); node n2 is combined with test item t1, test item t2, and test item t3 to construct third test task T3(n2, t1), fourth test task T4(n2, t2), and fifth test task T5(n2, t3), and the like.

[0062] In step S220, a test index value of each test task is obtained based on a preset index set.

[0063] The test index value of the test task represents the test priority of the corresponding test task. In step S220, the health monitoring system converts the state of each test task into a test index value quantified by the same standard, thereby providing accurate and objective decision basis for subsequent test task scheduling and realizing active testing of the node.

[0064] In this embodiment, the preset index set includes at least one index, and the index in the preset index set is used to evaluate the urgency of the test task.

[0065] In some examples, the preset index set is a multi-dimensional index set including indexes of multiple dimensions to evaluate the test priority of the test task from different dimensions, so as to avoid the limitation of a single dimension and improve the effectiveness and robustness of the test index value.

[0066] In these examples, the preset index set can include at least two of the first type of index, the second type of index, and the third type of index, wherein the first type of index is used to evaluate the real-time risk of the combination of the test item and the node, focusing on the specific state of the test task at the current time; the second type of index is used to evaluate the contribution of the inherent attribute of the test item to the test priority, reflecting the inherent importance and cost of the test item; and the third type of index is used to evaluate the contribution of the inherent attribute of the node to the test priority.

[0067] In another example, the preset index set can only include the above-mentioned first type of index, that is, the test priority of the test task is evaluated by the first type of index.

[0068] For example, the first type of index can include a failure probability index of the test task T , where the test task T is composed of any node n and any test item t. The failure probability index represents the probability that the node n cannot pass the test item t in the current state. From the dimension of failure probability, the larger the failure probability index value , the higher the health risk of the node n, and the higher the test priority or urgency of the test task T .

[0069] For another example, the first type of index can include a timeliness index of the test task T , which represents the length of time from the current time to the last time when the test task T was successfully tested. From the dimension of timeliness, the larger the timeliness index value , the longer the test item of the node has been neglected, the uncertainty increases, and the higher the test priority of the corresponding test task T .

[0070] For example, the second type of index can include a business impact index of the test item t. The business impact index represents the influence degree of the node capability verified by the test item t on business continuity. From the dimension of business impact, the larger the business impact index value The larger the value, the higher the test priority of the corresponding test task. For example, the test item verifying the network link has a greater service impact and a higher priority than the test item verifying the temperature sensor. Here, a corresponding service impact index value can be configured for each test item based on the importance of the node capability corresponding to each test item in the test set. For example, the service impact index value of the PCIe link test item can be set to 0.8, while the service impact index value of the temperature reading test item can be set to 0.2, etc.

[0071] For example, the second type of metric could include the resource cost metric for test item t. The resource cost metric represents the amount of resources consumed in executing the test item. This metric covers resources including computing resources, and may further include storage resources and / or network resources. Resource cost metric values... The priority of a test can be determined based on its resource consumption percentage and execution time. Resource cost metrics negatively impact test priority; from a resource cost perspective, resource cost metric values... The larger the value, the higher the execution cost, and its scheduling priority will be appropriately reduced when resources are limited.

[0072] The third category of indicators includes, for example, the reliability degradation index of node n, which is related to the service life of node n. Related, service duration This can be measured in days; in this case, the service duration is the number of days between the current time and the timestamp of node n going online. Reliability degradation metric value. This can be modeled as a function of node service life, such as an exponential function. From the perspective of reliability degradation metrics, the longer the service life, the greater the probability of hardware aging in the node, and the higher the reliability degradation metric value. The larger the value, the higher the priority of the corresponding test task.

[0073] For example: reliability degradation index value ,in, The set attenuation coefficient.

[0074] In an example where the preset metric set includes multiple metrics, test task T Test index value It can be based on test task T The weighted summation method for each indicator's individual value is determined. For example, the preset indicator set includes the aforementioned failure probability indicator, business impact indicator, resource cost indicator, reliability degradation indicator, and timeliness indicator, with the following weights for each indicator: , , Test Task T The test metric value can be expressed as:

[0075]

[0076] Through this weighted model, the health monitoring system can detect test tasks that are "high-risk, have high business impact, have controllable resource costs, and have not been tested for a long time," and prioritize the execution of these test tasks in the test scheduling. This ensures that limited test resources are invested in the areas that can best improve the overall reliability of the computing facilities, thereby maximizing operational efficiency and system stability.

[0077] In some examples, a fixed weight can be set for each metric in a pre-defined set of metrics.

[0078] In other examples, to further enhance the adaptability and long-term optimization of the health monitoring system, the system supports dynamic adjustment and online learning of the weights of each indicator in a preset indicator set during the monitoring process. This approach is more adaptable to complex and ever-changing operational scenarios, and by driving weight optimization through data feedback, it makes the guidance of test indicator values ​​for test scheduling more accurate.

[0079] In these examples, step S220, which involves obtaining the test metric value for each test task based on a preset metric set, may further include the following steps S221 to S223:

[0080] Step S221: For each test task, determine the individual indicator value of each indicator in the preset indicator set for the test task.

[0081] For example, test task T Failure probability index value Timeliness index value Business impact index values These are all individual indicator values.

[0082] Step S222: Determine the weight of each indicator in the preset indicator set.

[0083] In this example, the weights of at least some indicators are obtained by sampling based on the weight probability distribution of the corresponding indicators, which is dynamically updated based on historical test data.

[0084] This example can employ a reinforcement learning approach based on exploration and exploitation, maintaining a weight probability distribution (such as a Beta distribution) for each indicator that requires dynamic adjustment. Before each round of scheduling decisions, the health monitoring system samples a set of target weights from these weight probability distributions. Through weight sampling, it can utilize the currently optimal weight configuration presented by the weight probability distribution while retaining a certain degree of exploration capability to try potentially better weight configurations.

[0085] In this example, a Bayesian optimization algorithm (such as Thompson Sampling), an Upper Confidence Bound (UCB) algorithm, or a Softmax strategy can be used to manage and update the weight probability distribution.

[0086] Taking the set of preset indicators including the above-mentioned failure probability indicator, the service impact indicator, the resource cost indicator, the reliability attenuation indicator, and the timeliness indicator as an example, the health monitoring system maintains a weight probability distribution for the failure probability indicator and the service impact indicator, and uses a fixed initial weight for other indicators 、 In one scheduling round, the initial weight of the failure probability indicator in this scheduling round is sampled based on the weight distribution probability of the failure probability indicator , and the initial weight of the service impact indicator in this scheduling round is sampled based on the weight distribution probability of the service impact indicator . Then, the initial weight , the initial weight , and the initial weight 、 are normalized so that the sum of the weights of all indicators is equal to 1, and the target weight of each indicator in this scheduling round is obtained.

[0087] In step S223, the test indicator value of the test task is obtained based on the single indicator value of each indicator and the weight of each indicator.

[0088] In step S230, the scheduling and execution of the test task are performed according to the test indicator value of each test task.

[0089] The health monitoring system 100 can distribute and execute the test tasks according to the test indicator values of all test tasks calculated in step S220, to ensure that test tasks with high test indicator values can be executed preferentially in this scheduling round.

[0090] To adapt to the continuous changes of the state of the computing power facility, the health monitoring system 100 can reevaluate the test indicator values of all test tasks at fixed time intervals or based on event triggering, and update the scheduling strategy accordingly, so that the scheduling strategy can respond to the state changes of the computing power facility in real time, and realize persistent and accurate scheduling.

[0091] In some examples, the health monitoring system can select a subset of test tasks to be executed in this round from all test tasks according to a preset strategy. The preset strategy can be to select test tasks whose test indicator values exceed a certain threshold, or test tasks with top K test indicator values, etc.

[0092] For the filtered subset of test tasks, the test tasks can be arranged in descending order according to their test indicator values to generate an ordered test task execution sequence, so that test tasks with higher test indicator values are executed preferentially. For example, Celery can be used as a distributed task queue framework, and a priority queue can be implemented by combining Redis Sorted Set (ZSET). The test tasks are stored in ZSET as scores according to their test indicator values, and the health monitoring system obtains the tasks from high to low according to the scores to ensure the preferential execution of high-priority tasks.

[0093] In some examples, a scheduling condition for constraining the scheduling strategy can also be set, such as limiting the maximum number of concurrent test tasks, limiting the upper limit of GPU memory occupation, limiting network bandwidth usage, limiting CPU utilization of the node, etc. When scheduling test tasks, the health monitoring system distributes the test tasks based on the test indicator values of the test tasks and the scheduling condition as a constraint to ensure that the resource consumption of the node in executing the test items minimizes the impact on the normal business operation of the computing power facility. For example, the health monitoring system can preferentially attempt to assign test tasks with high test indicator values and low resource consumption (such as GPU memory) to the corresponding nodes.

[0094] In some examples, to achieve the best balance between exploration and utilization of existing test indicator values, the health monitoring system can use reinforcement learning algorithms such as Thompson sampling to drive the scheduling and execution of test tasks. In this regard, the health monitoring system can select test tasks with the highest expected rewards, i.e., the highest test indicator values, on the one hand, and randomly select some test tasks with a certain probability on the other hand to collect new test data and avoid falling into local optimization. On this basis, the health monitoring system can use the Bayesian method to update the scheduling strategy based on the reward feedback obtained after the two aspects of scheduling and execution, to achieve continuous optimization of the scheduling strategy.

[0095] According to steps S210 to S230, in the present embodiment, by constructing a test task set composed of nodes in the computing power facility and test items in the test set, and evaluating the priority of each test task based on the preset indicator set, the health monitoring system can accurately identify the combination of high-risk nodes and key test items, and then actively and targetedly monitor and test the nodes in the computing power facility. The method of the present embodiment is beneficial to discovering potential problems in time before failure occurs or at the early stage of performance degradation, significantly reduces the risk of system operation, and improves overall reliability and stability.

[0096] On the other hand, this embodiment evaluates the priority of test tasks and implements intelligent scheduling and task distribution based on priority scores, thereby enabling the priority execution of high-value test tasks under limited resource conditions. This strategy not only ensures the effectiveness and coverage of health monitoring, but also effectively avoids the ineffective occupation of computing resources by low-priority or redundant tests, thus improving the utilization efficiency of test resources and reducing the overall system overhead.

[0097] <Second Embodiment>

[0098] In this embodiment, the preset indicator set includes at least the failure probability indicator of the test task. The health monitoring system predicts the failure probability indicator value of each test task by calling a pre-trained prediction model.

[0099] In this embodiment, as Figure 3 As shown, determining the failure probability index value for any test task may include the following steps S321 to S323:

[0100] Step S321: Obtain the associated data of the test task.

[0101] In this embodiment, the associated data of the test task includes data associated with at least one of the nodes and test items in the test task. To ensure data privacy and transmission security, all associated data can be transmitted using an encrypted channel.

[0102] The associated data for a test task may include at least some of the following: hardware status data of the corresponding node, historical test data related to the test task, and business load data of the corresponding node.

[0103] Historical test data related to the test task includes historical test data of nodes within the test task for test items within the test task. Historical test data related to the test task may also include at least one of the following: historical test data of the corresponding node for other test items, and historical test data of the corresponding test item on other nodes.

[0104] The node's hardware status data can include hardware configuration data and real-time status data. Hardware configuration data includes, for example, at least some of the static attributes such as GPU model, CPU model, driver version, and firmware version. Real-time status data includes, for example, GPU status data and / or CPU status data. GPU status data can include GPU memory usage, temperature, power consumption, ECC error count, etc., while CPU status data can include CPU load rate, PCIe bandwidth, InfiniBand / RoCE link latency, etc.

[0105] The historical test data is used to record the historical execution of the test tasks. The health monitoring system can maintain time series data of the test from multiple dimensions. For example, time series data is maintained from the dimensions of test task combined with test item and node, node, and test item, respectively, to support quick access to historical test data related to the test task as needed. Each data point of the time series data can include test execution timestamp, test result (pass or fail), performance indicator value obtained by the test, and other test data.

[0106] The business load data reflects the current business pressure carried by the node and can be obtained through the Kubernetes API. The business load data includes, for example, the type of task (such as model training, model inference) running on the node, the resource request specification, and the like. The business load data helps the health monitoring system to evaluate the impact of the node capability verified by the test item on the running task.

[0107] In step S322, a feature vector of the test task is generated based on the association data.

[0108] Through step S322, the original association data can be converted into a set of structured and computable feature vectors, providing high-quality input for the prediction model.

[0109] In some examples, the association data can be converted into three types of features, namely static attribute features, time series features, and context features.

[0110] The static attribute features describe the relatively stable inherent attributes of the node or test item. The static attribute features include, for example, hardware model, driver version, firmware version, node service time, and the like. The static attribute features are mainly derived from the hardware status data.

[0111] The time series features are dynamic trends and statistical features extracted from the historical test data, including, for example, at least one of the following:

[0112] Failure rate sliding window: the failure rate of the test task in the last M historical tests is calculated;

[0113] Performance degradation trend: for example, using methods such as Exponentially Weighted Moving Average (EWMA) or linear fitting, the decline slope of the key performance indicator (such as the algorithmic throughput of half-precision floating-point FP16) is calculated;

[0114] Abnormal detection feature: for example, using statistical methods such as Z-score and IQR (interquartile range), abnormal patterns (such as sudden increase in GPU occupancy) exhibited by the node in the historical test are identified.

[0115] The context features introduce task-related environmental and management information, derived from historical test data, business load data, for example, including at least one of the following:

[0116] Same-batch node behavior: calculating the average failure rate of nodes of the same model or the same batch when performing the same test item;

[0117] Task priority context: whether the current node is executing a high-priority business task;

[0118] Associated test item results: test result features of the node in the associated test item in the test task, wherein the associated test item is another test item that has an association with the test item in the test task, for example, the health status of the PCIe link affects the multi-card communication test, so the PCIe link test is associated with the multi-card communication test.

[0119] To construct a complete and predictive feature vector, it is usually required that the nodes and test items involved in the test task have prior historical test data. However, in actual scenarios, some test tasks may be cold start tasks, that is, at least one of the nodes and test items in the test task has not accumulated enough historical test data, so as to provide sufficient associated data for feature extraction. Such cold start tasks will affect the completeness of the feature vector and the accuracy of the prediction result to some extent due to the limited associated data.

[0120] To improve the prediction effectiveness of cold start tasks, in some examples, the health monitoring system 100 can call at least one of a pre-trained feature extraction module and a knowledge graph to generate a feature vector for the cold start task based on the current associated data of the cold start task, mainly the hardware state data of the node.

[0121] In this example, the feature extraction model is trained to generate a representative feature representation based on the available associated data (mainly hardware state data) of the node in the cold start task.

[0122] In this example, the knowledge graph is constructed based on the association between node hardware configuration, test items, and fault information. By searching for a target path matching the cold start task in the knowledge graph, the fault information on the target path is extracted as at least part of the features.

[0123] For example, for a cold start task, generating its feature vector can include the following steps: calling the feature extraction model to generate the first part of the features of the cold start task based on the associated data of the cold start task; searching for a target path matching the cold start task in the knowledge graph, and taking the fault information on the target path as the second part of the features of the cold start task; and fusing the first part of the features and the second part of the features to form the feature vector of the cold start task.

[0124] At step S323, the prediction model is invoked to predict the failure probability indicator value of the corresponding test task based on the feature vector.

[0125] In this embodiment, the prediction model is trained to establish a mapping relationship from the feature vector of the test task to the failure probability of the test task.

[0126] The prediction model can be trained based on the sample set. For real samples, the number of positive samples that pass the test will be much larger than the number of negative samples that fail the test. In order to cope with the problem of unbalanced positive and negative samples, a weighted loss function can be used in the model training stage to reduce the influence weight of the majority of positive samples and improve the recognition ability of the model for the minority class of negative samples.

[0127] In the example where the feature vector includes time sequence features and structured features, in order to make full use of the information of different types of features, a model fusion strategy can be used, that is, a first prediction model good at processing structured features such as XGBoost, LightGBM, etc. and a second prediction model good at processing time sequence features such as LSTM, Transformer, etc. can be used together to complete the prediction of the failure probability of the test task. Among them, the structured feature refers to a data feature that has no inherent time sequence or sequence dependency. These features can be independent, discrete or static attributes, and their values do not depend on the order of occurrence of other features, for example, including the above static attribute features. Structured features can be stored in table form, with each row representing a sample and each column representing a feature, etc. While the time sequence feature is a feature extracted from a series of data points arranged in time sequence, and there is a sequence dependency between these data points, which can be used to capture patterns, trends, periodicity and dynamic changes of data in the time dimension.

[0128] In this example, invoking the prediction model to predict the failure probability indicator value of the test task based on the feature vector can further include the following steps: invoking the first prediction model to generate a first failure probability based on the structured feature; invoking the second prediction model to generate a second failure probability based on the time sequence feature; and fusing the first failure probability and the second failure probability to obtain the failure probability indicator value of the test task.

[0129] In this example, the first failure probability and the second failure probability can be fused by weighted summation, and the first prediction model and the second prediction model can have the same weight or different weights, for example, the weight of the first prediction model is set to be greater than the weight of the second prediction model. In this example, the maximum value fusion strategy can also be used to obtain the failure probability indicator value of the test task, which is not limited here.

[0130] According to steps S321 to S323, the embodiment uses a pre-trained prediction model to evaluate the failure probability of a test task, which can reduce the dependence on expert experience, and thus adapt to complex and variable operating environments, and provide reliable and objective risk quantification basis for scheduling execution.

[0131] In addition, those skilled in the art can understand that, in another embodiment, the health monitoring system 100 can also establish a mapping between the correlation data and the failure correlation based on an expert knowledge base.

[0132] <Third embodiment>

[0133] In the embodiment, by constructing a closed loop of health monitoring, the execution result of the test task is fed back to the system, which is used to dynamically optimize the priority evaluation strategy and / or the scheduling strategy, thereby significantly improving the long-term stability and adaptability of the health monitoring system.

[0134] As shown in Figure 4 , the test scheduling method of the embodiment can include the following steps S410 to S450:

[0135] Step S410, according to a plurality of test items and a plurality of nodes in the computing power facility, a plurality of test tasks are constructed.

[0136] Step S420, for each test task, a test index value of the corresponding test task is obtained based on a preset index set.

[0137] Step S430, according to the test index value of each test task, the scheduling execution of the test task is performed.

[0138] Step S440, test data of the test task is obtained.

[0139] After the test task is executed, the health monitoring system obtains the actual test data of the task, including the test result of whether the test passes, the performance index value of the tested node, the execution time consumption, abnormal information, etc. These test data reflect the real health condition of the node in the current state, and can provide objective basis for system optimization.

[0140] Step S450, based on the test data, an adaptive parameter related to the test index value of the test task is updated, and / or an adaptive parameter related to scheduling is updated.

[0141] The health monitoring system 100 uses the test data obtained in step S440 to dynamically update and optimize the adaptive parameters that affect the calculation of the test index value or the decision of the task scheduling.

[0142] In some examples where individual metric values ​​are obtained based on models, such as predicting the failure probability metric value of a test task using a prediction model, the adaptive parameters related to obtaining the test metric value of the test task may include model parameters used for metric value calculation, and may also include model parameters of the feature extraction model for the cold start task.

[0143] In an example where the metric weights can be dynamically adjusted, the adaptive parameters related to obtaining the test metric values ​​for the test task can include distribution parameters of the weight probability distribution, such as parameters defining the Beta distribution.

[0144] When the scheduling strategy for a test task is built based on reinforcement learning, heuristic algorithms, etc., the adaptive parameters related to scheduling may include exploration and exploitation strategy parameters, task ranking threshold parameters, and parameters used in scheduling conditions.

[0145] like Figure 5 As shown, in some examples, a health monitoring system may include a data acquisition module for collecting associated data, a feature engineering module for generating feature vectors based on associated data, a risk prediction module for predicting the failure probability of test tasks based on feature vectors, a priority scoring module for prioritizing based on a set of predicted indicators, a scheduling module for executing scheduling strategies, a test execution module for executing test tasks, and a feedback loop module for adaptive parameter updates based on test data. The feedback loop module can be used to update the parameters of the feature engineering module, the risk prediction module, the priority scoring module, the scheduling module, etc.

[0146] <Fourth Embodiment>

[0147] This disclosure also provides an electronic device, such as... Figure 6 As shown, the electronic device 600 includes at least one processor 610 and at least one memory 620, the memory 620 being used to store computer program instructions, which, when executed by the processor 610, cause the electronic device 600 to perform a test scheduling method according to any embodiment of the present disclosure.

[0148] The electronic device 600 deploys the aforementioned health monitoring system to execute the test scheduling method; it can be a server or other types of equipment.

[0149] This disclosure also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the memory management methods or compilation methods described in the foregoing embodiments of this disclosure. Optionally, the computer-readable storage medium may be a non-transitory storage medium, but is not limited thereto; it may also be a temporary storage medium.

[0150] Various embodiments described in this specification can be implemented in any of a variety of ways. Each embodiment is primarily described in the context of a different implementation, but the same or similar elements of each embodiment can be used in other embodiments as well.

[0151] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out operations described herein for any of the methods described above.

[0152] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a magnetically encoded device such as magnetic strip cards, an optically encoded device such as a compact disc (CD) or DVD, and / or any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0153] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise one or more of a copper transmission cable, an optical transmission cable, a wireless transmission, routers, firewalls, switches, gateway computers and edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0154] Computer readable program instructions for carrying out operations of the present disclosure can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or source or object code, in any combination of one or more programming languages, including an object-oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present embodiments.

[0155] Various aspects of the present disclosure can be described in the general context of methods, apparatuses, and computer program products, according to embodiments of the present disclosure. It should be understood that the various aspects of the present disclosure can be implemented in combination with other aspects or in the alternative (and that one or more aspects of the present disclosure can be implemented by themselves), which can be claimed in the present and future applications. Generally, the order of the steps recited in the methods of aspects of the present disclosure can be interchanged, in other aspects of the present disclosure, unless a specific dependence of steps on one another is expressly stated.

[0156] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0157] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0158] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0159] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative, and not restrictive, of the disclosed embodiments. Many modifications and variations of the disclosed embodiments are possible in light of the above teachings. It is therefore to be understood that within the scope of the disclosed embodiments, modifications and variations of the disclosed embodiments can be practiced. It is also to be understood that the specific order or hierarchy of steps in the processes disclosed is an illustration of exemplary processes. Based upon the description and illustrations provided herein, those skilled in the art will understand that changes can be made to the order of steps in the processes and that many of the individual steps can be modified or eliminated. Additionally, the description and illustrations provided herein are not meant to limit the scope of the disclosed embodiments. The scope of the disclosed embodiments is limited only by the claims.

Claims

1. A test scheduling method for health monitoring, characterized in that, include: Multiple test tasks are constructed based on multiple set test items and multiple monitored nodes in the computing power facility; each test task consists of a test item and a node. For each test task, a test indicator value is obtained based on a preset indicator set; wherein, the test indicator value represents the test priority of the corresponding test task, and the larger the test indicator value, the higher the test priority and urgency of the corresponding test task. The preset indicator set includes a first type of indicator, which is used to evaluate the contribution of the combination of test items and nodes to the test priority. Based on the test metric values ​​of each test task, the test tasks are scheduled and executed so that test tasks with high test metric values ​​can be executed first in this round of scheduling, thereby reducing the overall test overhead.

2. The method according to claim 1, characterized in that, The first type of metric includes the failure probability metric of the test task. Determining the failure probability metric value of the test task includes: Obtain the associated data of the test task; wherein, the associated data includes data associated with at least one of the nodes and test items in the test task; Generate the feature vector of the test task based on the associated data; The prediction model is invoked to predict the failure probability index value of the test task based on the feature vector, wherein the prediction model is trained to establish a mapping relationship from the feature vector of the test task to the failure probability of the test task.

3. The method according to claim 2, characterized in that, The feature vector includes structured features and temporal features. The invocation prediction model predicts the failure probability index value of the test task based on the feature vector, including: The first prediction model is invoked to generate a first failure probability based on the structured features; The second prediction model is invoked to generate a second failure probability based on the aforementioned temporal features; By combining the first failure probability and the second failure probability, the failure probability index value of the test task is obtained.

4. The method according to claim 2, characterized in that, The multiple test tasks include cold start tasks that meet cold start conditions, where at least one of the test items and nodes has no historical test data; a feature vector for each cold start task is generated based on its associated data. include: Invoke at least one of the feature extraction model and the knowledge graph to generate the feature vector of the cold start task based on the associated data of the cold start task; The feature extraction model is trained to generate at least some features of the cold start task based on the associated data of the cold start task; the knowledge graph is constructed based on the association between node hardware configuration, test items and fault information, and the target path in the knowledge graph that matches the cold start task is used to generate at least some features of the cold start task.

5. The method according to claim 1, characterized in that, The first category of indicators includes at least some of the indicators in the test task failure probability indicator and the test task timeliness indicator, wherein the timeliness indicator represents the length of time since the most recent successful test.

6. The method according to any one of claims 1 to 5, characterized in that, The preset indicator set also includes at least one of the second type of indicators and the third type of indicators; the second type of indicators is used to evaluate the contribution of the inherent attributes of the test item to the test priority, and the third type of indicators is used to evaluate the contribution of the inherent attributes of the node to the test priority.

7. The method according to claim 6, characterized in that, The second category of indicators includes at least some of the business impact indicators and resource cost indicators of the test items; the business impact indicators represent the degree of impact of the node capabilities verified by the test item on the business, and the resource cost indicators represent the proportion of resources consumed in executing the test item; or, The third type of indicator includes the reliability degradation index of the node, which is related to the service life of the node.

8. The method according to any one of claims 1 to 5, characterized in that, For each of the aforementioned test tasks, obtaining the corresponding test indicator value based on a preset indicator set includes: Determine the individual indicator value for each indicator in the preset indicator set for the test task; Determine the weight of each indicator in the preset indicator set; wherein, the weights of at least some indicators in the preset indicator set are obtained by sampling based on the weight probability distribution of the corresponding indicators, and the weight probability distribution of the corresponding indicators is updated based on historical test data. The test index value of the test task is obtained based on the individual index value and weight of each index.

9. The method according to any one of claims 1 to 5, characterized in that, The step of scheduling and executing test tasks based on the test metric values ​​of each test task includes: Based on the test metric values ​​of each test task and constrained by the set scheduling conditions, the test tasks are distributed and executed.

10. The method according to any one of claims 1 to 5, characterized in that, After scheduling and executing the test task, the method further includes: Obtain test data from the executed test tasks; Based on the test data, update the adaptive parameters related to the test metric values ​​of the test task, and / or update the adaptive parameters related to scheduling.

11. An electronic device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

12. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1-10.

Images