Electronic device and performance detection method therefor, and medium

By matching with historical job performance data, the accuracy problem of job performance degradation detection in computer clusters is solved, accurate judgment and fault analysis of node performance degradation are achieved, and the accuracy and robustness of detection are improved.

WO2024244575A1PCT designated stage expired Publication Date: 2024-12-05HUAWEI TECH CO LTD

Patent Information

Application Number
PCT/CN2024/079582
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-30
Filing Date
2024-03-01
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

The prior art is difficult to accurately detect the deterioration of the job performance of nodes in computer clusters, especially when the job type and operation stage changes between nodes, resulting in misjudgment and inaccurate performance evaluation.

Method used

By collecting the performance data of the current job to match the performance data of the historical job, based on the matching degree of the job type, node type and job number, it is judged whether there is performance degradation, and distinguish between hardware failures and network failures in the detection results.

Benefits of technology

It improves the accuracy of operation performance detection, avoids misjudgment, can more accurately determine whether there is performance degradation in the current operation stage, and provides detailed fault analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024079582_05122024_PF_FP_ABST
    Figure CN2024079582_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The present application relates to the technical field of advanced computing devices, and particularly relates to an electronic device and a performance detection method therefor. The method comprises: a first computing node in a computer system executing a first operation; collecting first performance data of the first computing node executing the first operation; acquiring, from among historical operation performance data, first historical performance data which meets a first data condition with the first performance data, wherein the first data condition comprises: the first performance data having the same operation type as the first historical performance data; and on the basis of the degree of matching between the first historical performance data and the first performance data, determining whether the first computing node is subjected to performance degradation during the process of executing the first operation. Therefore, misjudging a node as being subjected to operation performance degradation is avoided, and operation performance data in the current running stage is compared with operation performance data in a historical running stage, such that it is possible to more accurately determine whether operation performance degradation occurs in the current running stage.
Need to check novelty before this filing date? Find Prior Art

Description

Electronic equipment and performance testing method and medium thereof

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 30, 2023, with application number 202310628502.8 and application name “Electronic device and its performance detection method and medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the technical field of advanced computing devices, and more particularly to an electronic device and a performance detection method and medium thereof. Background Art

[0003] A high-performance computer (HPC), also known as a supercomputer, is a high-performance computer device that supports multiple central processing units (CPUs) or graphics processing units (GPUs). In applications requiring high data processing volume, such as extreme weather forecasting and big data computing, multiple supercomputers (nodes 1 to N) can be connected via a network, as shown in Figure 1(a), to work together to form a computer cluster with enhanced performance, enabling faster processing of various tasks.

[0004] During the continuous operation of a computer cluster, the job performance of the computer cluster may degrade. Common job performance degradations may include: network congestion, CPU load imbalance, etc. Network congestion can mean that the network between nodes is busy and data transmission is slow; CPU load imbalance can mean that when a job is assigned to different nodes for execution, the CPU load of some nodes is too high, while the CPU load of other nodes is too low. The above-mentioned job performance degradations will lead to a decrease in the job performance of the computer cluster. Therefore, it is necessary to provide a detection method for detecting whether each node in the computer cluster has experienced job performance degradation.

[0005] Summary of the Invention

[0006] The present application provides an electronic device and a performance detection method and medium thereof.

[0007] In a first aspect, an embodiment of the present application provides a performance detection method, which is applied to a computing node in a computer system; and the performance detection method includes:

[0008] A first computing node in the computer system executes a first job;

[0009] Collecting first performance data of the first computing node executing the first job;

[0010] Acquire, from the historical job performance data, first historical performance data that satisfies a first data condition with the first performance data, wherein the first data condition includes: the first performance data and the first historical performance data have the same job type;

[0011] Based on the matching degree between the first historical performance data and the first performance data, it is determined whether performance degradation occurs during execution of the first job by the first computing node.

[0012] In the present application, the computer system may include a computer cluster, the computing nodes may include supercomputers (nodes) in the computer cluster, and the first computing node may be a supercomputer (node) in the computer cluster. The first job may be the current job running on the first computing node, and the first performance data may be the current job performance data, that is, newly acquired job performance data after the current running phase of the current job (first job) of the first computing node has completed. The historical job performance data may include job performance data of the computing nodes obtained by running jobs on each computing node in the computer cluster, stored on a common storage device in the computer cluster. The first historical performance data may be one of a plurality of historical job performance data. The first performance data and the historical job performance data may be described by a job performance profile, which includes, among other things, a job type. The first data condition may include matching the obtained job performance profile of the first performance data with the job performance profile of the historical job performance data, that is, matching, from the job performance profile of the historical job performance data, first historical performance data that occurs before the running time of the running phase of the first performance data and matches the job type of the first job of the first performance data. The degree of matching may be the numerical difference between the first performance data and the first historical performance data.

[0013] It can be seen that the above-mentioned method for detecting job performance degradation avoids the situation where the same node runs jobs of different job types in multiple running stages, and the numerical differences of the job performance data generated in multiple running stages are large, which leads to the node being misjudged as having job performance degradation; after a current running stage is completed, by looking for the job type, node information or job number corresponding to the current running stage and having been saved, the job performance data of the current running stage and the historical running stage are compared, which can more accurately determine whether job performance degradation has occurred in the current running stage.

[0014] In a possible implementation of the first aspect, the job type includes at least one of a storage job and a computing job.

[0015] In a possible implementation of the first aspect, satisfying the first data condition further includes:

[0016] The computing node that executes the job corresponding to the first historical performance data is of the same node type as the first computing node.

[0017] In the present application, the first data condition here also includes matching the obtained job performance portrait of the first performance data with the job performance portrait of the historical job performance data, that is, from the job performance portrait of the historical job performance data, matching the first historical performance data that matches the node information (node ​​type) of the first computing node corresponding to the first performance data.

[0018] In a possible implementation of the first aspect, satisfying the first data condition further includes:

[0019] The job corresponding to the first historical performance data has the same job number as the first job, where the job number is used to indicate that the first job is a repeatable job.

[0020] In the present application, the first data condition here also includes matching the job performance portrait of the obtained first performance data with the job performance portrait of the historical job performance data, that is, matching the first historical performance data corresponding to the first job that has been repeatedly executed from the job performance portrait of the historical job performance data.

[0021] In a possible implementation of the first aspect above, the first historical performance data indicates that no performance degradation occurs when the first computing node executes historical jobs in the corresponding historical time period.

[0022] In the present application, the first historical performance data may be job performance data obtained when the first computing node executes a historical job.

[0023] In a possible implementation of the first aspect, determining whether performance degradation occurs during execution of the first job by the first computing node based on a matching degree between the first historical performance data and the first performance data includes:

[0024] Calculating a matching degree between the first performance data and the first historical performance data;

[0025] When the matching degree satisfies a preset matching degree condition, it is determined that there is performance degradation in the process of the first computing node executing the first job, wherein the matching degree represents the data difference between the first performance data and the first historical performance data, and the matching degree satisfies the preset matching degree condition including: the absolute value of the matching degree is greater than the first preset matching degree or the absolute value of the matching degree is less than at least one of the second preset matching degree.

[0026] In the present application, the preset matching condition here may include that the first performance data is lower than the first historical performance data, and the numerical difference between the first performance data and the first historical performance data exceeds a preset numerical threshold.

[0027] In a possible implementation of the first aspect, the method further includes:

[0028] When performance degradation occurs during execution of the first job by the first computing node, a performance degradation detection result is generated based on the first job type of the first job, where the detection result includes at least one of a hardware failure and a network failure.

[0029] In a possible implementation of the first aspect above, the historical job performance data is stored in a public storage device of the computer system, wherein all computing nodes included in the computer system can access the historical job performance data.

[0030] In a possible implementation of the first aspect, the method further includes:

[0031] When it is determined that there is no performance degradation in the process of the first computing node executing the first job, the first historical performance data stored in the public storage device is updated using the first performance data.

[0032] In a possible implementation of the first aspect above, the public storage device stores historical New Year data in the form of a database or data file.

[0033] In a possible implementation of the first aspect, the method further includes:

[0034] The first computing node in the computer system executes the second job;

[0035] Collecting second performance data of the first computing node executing the second job;

[0036] When the second historical performance data that satisfies the first data condition with the second performance data is not obtained from the historical operation performance data, the second performance data is stored as the second historical performance data.

[0037] In the present application, if the second historical performance data is not matched, the second performance data will be stored as the second historical performance data in a public storage device. That is to say, when the first computing node obtains the second performance data when running the second job, there is no historical job performance data. The second performance data can be used as the historical job performance data for the first computing node to run the next same job or the same type of job.

[0038] In a second aspect, an embodiment of the present application provides an electronic device, including:

[0039] a memory for storing instructions to be executed by one or more processors of the electronic device, and

[0040] The processor is one of the processors of the electronic device, and is used to execute the performance detection method of the first aspect.

[0041] In a third aspect, an embodiment of the present application provides a computer program product, comprising: a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium containing a computer program code for executing the performance detection method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] FIG1( a ) is a schematic diagram of a computer cluster provided in an embodiment of the present application;

[0043] FIG1( b ) is a flow chart of a method for detecting operating performance degradation provided in an embodiment of the present application;

[0044] FIG2 is a schematic diagram showing the change in the number of executed instructions of a node over time during the process of a node running a job provided by an embodiment of the present application;

[0045] FIG3 is a system architecture diagram of a method for detecting nodes in a computer cluster provided in an embodiment of the present application;

[0046] FIG4 is a flow chart of a method for detecting operating performance degradation provided by an embodiment of the present application;

[0047] FIG5 is a schematic diagram of an operation type provided in an embodiment of the present application;

[0048] FIG6 is a schematic diagram of clustering nodes provided in an embodiment of the present application;

[0049] FIG7 is a schematic diagram of an operation performance portrait provided by an embodiment of the present application;

[0050] FIG8 is a schematic diagram of a system for detecting operating performance degradation provided by an embodiment of the present application;

[0051] FIG9 is a schematic diagram of the structure of a supercomputer (node) provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the objectives, embodiments and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0053] Figure 1(b) illustrates a flowchart of a method for detecting job performance degradation. The method for detecting job performance degradation illustrated in Figure 1(b) can be executed by one or more detection nodes, which can be a supercomputer in a computer cluster. Each detection node can be equipped with a functional module for executing the method. This functional module is configured to detect job performance data corresponding to all nodes in the computer cluster and determine whether any node is experiencing job performance degradation.

[0054] As shown in FIG1( b ), the method for detecting operation performance degradation may include the following steps.

[0055] S101: Detecting that a job is running.

[0056] S102: Read the log to obtain job performance data.

[0057] For example, the job performance data here may include: number of instructions executed per second, CPU usage, memory usage, etc.

[0058] S103: Determine whether the job performance data does not meet the performance threshold. For example, taking the number of instructions executed per second as an example, if the number of instructions executed per second is lower than the performance threshold, then execute step S105. If the job performance data is higher than or equal to the performance threshold, then execute step S104.

[0059] S104: Determine whether the operation duration is greater than a duration threshold.

[0060] Exemplarily, if the operation duration is higher than the duration threshold, step S105 is executed: determine whether a performance anomaly has occurred. Otherwise, the process may return to step S101, wait for a certain period of time, and then continue to perform the next detection of operation performance degradation. In other words, the method for detecting operation performance degradation shown in FIG1(b) may be executed periodically by the system. For example, when no anomaly is detected after executing steps S103 and S104, the detection may be terminated. After waiting for a certain period of time (for example, 5 minutes), step S101 is re-executed to restart monitoring. It will be understood that the numerical values ​​here are exemplary and are not limited to this in the embodiments of the present application.

[0061] S105: Determine whether a performance abnormality occurs.

[0062] Exemplarily, that is, it is determined that the operation performance has deteriorated.

[0063] For example, FIG2 illustrates a schematic diagram of the change in the number of instructions executed by a node over time during the process of a node running a job. It can be understood that the more instructions a node executes per unit time, the better the node's operating performance. By comparing the number of instructions executed by each node in a computer cluster, the operating performance of the entire computer cluster can be evaluated. As can be seen, FIG2 describes the process of a node running a job using time and the number of instructions executed as job performance data. The process of a node running a job can be divided into multiple continuous operating stages. Within each operating stage, the operating performance data of the node in that operating stage varies depending on the type of job run by the node. For example, taking stages 1 and 2 as an example, in stage 1, the node executes a low-load job, and in stage 2, the node executes a high-load job, so that the number of instructions of the node in stage 1 is lower than that in stage 2. The high load and low load here can be used to describe the number of floating-point operations or instructions that a node can complete per unit time, which can be expressed in terms of instructions executed per second (FLOPS).

[0064] The following describes a method for detecting the degradation of the operation performance of a node in combination with Figures 1 and 2. For example, the performance threshold of the number of instructions executed per second is set to 5 million / second. In stages 1 and 4 shown in Figure 2, the number of instructions executed per second by the node is 1 million / second respectively, and in stages 2, 3 and 5, the number of instructions executed per second by the node is 10 million / second respectively. That is to say, if the method for detecting the degradation of the operation performance shown in Figure 1 is adopted, that is, only the operation performance data of each stage from stage 1 to stage 5 is compared with the performance threshold to confirm that the node has operation performance degradation in the stage where the performance threshold is not met, then it is confirmed that the node shown in Figure 2 has operation performance degradation in stages 1 and 4.

[0065] It can be seen that if the job performance degradation detection method described in Figure 1 is used to detect job performance degradation on the job performance data in Figure 2, that is, if job performance degradation is detected only through threshold judgment, different thresholds need to be set for different types of nodes and different types of jobs, which increases the complexity of detecting job performance degradation. At the same time, if the association between the running stages that the same node has passed is not considered, that is, referring to the running time corresponding to the running stages of the same node and the job type of the job, if adjacent running stages correspond to jobs of different job types, resulting in a large difference in the job performance data corresponding to the running stages, detecting job performance degradation only through threshold judgment will result in inaccurate (misjudgment) problems, such as in stages 1 and 4 in Figure 2.

[0066] In order to solve the problem of inaccuracy in the detection process of job performance degradation, an embodiment of the present application proposes a method for detecting job performance degradation, which stores the job performance data of the node's historical operation stage (hereinafter referred to as historical job performance data). When detecting whether the current job performance data of the current operation stage is normal, after the current operation stage ends, historical job performance data before the operation time of the current operation stage and matching the job type, node information or job number of the current job performance data are matched from the historical job performance data. If the data difference between the current job performance data and the historical job performance data meets the performance degradation condition, it is determined that the node has job performance degradation in the current operation stage.

[0067] It's understandable that since the stored historical job performance data is normal, the difference between the current and historical job performance data can be used to determine whether a node has experienced job performance degradation. For example, if the job performance data is the number of instructions executed per second, if the number of instructions executed per second during the current run is significantly lower than the number of instructions executed per second during the historical run, then the node is determined to have experienced job performance degradation during the current run.

[0068] In some embodiments, if the node does not experience job performance degradation in the current operation phase, the current job performance data is used to update the historical job performance data for job performance degradation detection of the job performance data of the next operation phase. It can be understood that the job performance data, job type, node information (node ​​type), and running time or job number corresponding to the operation phase of each job of a node here can be described by the job performance portrait of the operation phase. The job number here is used to identify repeatable jobs. It is set when the job is first generated, and the job number can be used to match whether there is historical job performance data for the same job.

[0069] It can be seen that the above-mentioned method for detecting job performance degradation avoids the situation where the same node runs jobs of different job types in multiple running stages, and the numerical differences of the job performance data generated in multiple running stages are large, which leads to the node being misjudged as having job performance degradation; after a current running stage is completed, by looking for the job type, node information (node ​​type) or job number corresponding to the current running stage and having been saved, the job performance data of the current running stage and the historical running stage are compared, so as to more accurately determine whether job performance degradation occurs in the current running stage.

[0070] To more clearly and in detail describe the detection method provided in the embodiments of the present application, the following first describes the electronic device involved in implementing the method provided in the embodiments of the present application. The electronic device can be a computer, a supercomputer, or a computer cluster consisting of multiple computers or supercomputers. It is understood that the embodiments of the present application do not impose any restrictions on the specific type of electronic device.

[0071] FIG3 exemplarily shows a system architecture diagram of a method for detecting nodes in a computer cluster provided in an embodiment of the present application.

[0072] As shown in Figure 3, the system architecture 300 includes an online job feature construction module 301 and a job performance degradation perception module 302, wherein the online job feature construction module 301 also includes: a job performance feature extraction unit 3011 and a job performance feature merging unit 3012; the job performance degradation perception module 302 also includes: a job performance feature matching unit 3021, a contour detection unit 3022, an anomaly detection unit 3023 and an initial cause delimitation unit 3024.

[0073] The job performance feature extraction unit 3011 is used to collect information from each node in the computer cluster, including job performance data, job type, node information (node ​​type), and run time or job number. Using a feature description method, the collected information is characterized to extract a job performance profile. That is, a correspondence is established between job performance data, job type, node information (node ​​type), and run time or job number. This can also be called job performance profiling.

[0074] The job performance feature merging unit 3012 is used to merge the extracted job performance portrait, i.e., job performance data, job type, node information (node ​​type) and running time or job number, with the job performance portrait saved in the job performance feature library, that is, to update and replace the existing job performance portrait, for job performance feature matching during job performance degradation.

[0075] The job performance feature matching unit 3021 is configured to match historical job performance profiles from the job performance feature library that occurred before the current run phase and that match the current job performance profile. Specifically, the unit 3021 matches historical job performance profiles that occurred before the current run phase and that match the job type, node information (node ​​type), or job number in the current job performance profile.

[0076] The profile detection unit 3022 is used to determine whether the current job performance data in the current job performance portrait has job performance degradation through the historical job performance data in the historical job performance portrait matched by the job performance feature matching unit 3021. It can also be called obtaining the performance profile of the historical job performance data, regularly sensing and detecting the performance profile of the current job performance data, and determining whether the current job performance data has job performance degradation. In some embodiments, the profile detection unit 3022 can detect the current job performance data in a periodic manner, or it can uniformly detect the current job performance data after the current operation stage ends. If job performance degradation is detected, the abnormality detection unit 3023 is entered; otherwise, the current job performance portrait is used to update the historical job performance portrait, that is, the current job performance data is used to update the historical job performance data, which is used to detect job performance degradation of the job performance data of the next operation stage, which can also be called updating the performance baseline.

[0077] After the profile detection unit 3022 detects job performance degradation, the anomaly detection unit 3023 invokes an anomaly detection algorithm to perform anomaly detection. For example, it compares the current performance parameters of the current master node in the computer cluster with historical performance parameters to detect node performance degradation. If an anomaly occurs, the primary cause identification unit 3024 further determines whether the anomaly is a hardware failure.

[0078] The initial cause demarcation unit 3024 is used to analyze the root cause of the fault (network or hardware, etc.) and return the fault node information after the perception detection operation performance degradation occurs.

[0079] After introducing the system architecture of the detection method for nodes in a computer cluster, the following describes a method flow chart of the detection method of an embodiment of the present application through Figure 4. The method shown in Figure 4 can be implemented by the processor of one or more nodes (supercomputers) in the computer cluster executing relevant instructions. The detection method may include the following steps. It is understood that one or more nodes (supercomputers) here can be used as nodes specifically for performing job performance detection.

[0080] S401: Determine target job performance data.

[0081] For example, the target job performance data may be various job performance data of each supercomputer (node) in a computer cluster when running a job. The job performance data may include the number of instructions executed per second, CPU usage, memory usage, etc.

[0082] After the running phase of the job is completed or during the running phase of the job, feature extraction can be performed on the job performance data corresponding to the running phase to obtain target job performance data. In some embodiments, corresponding associated information can also be obtained, including basic features such as node information (node ​​type), job type, job number, and running phase or job number; the target job performance data can be the mean, mean square error, or covariance value of multiple job performance data in the entire running phase.

[0083] The target job performance data and associated information form a job performance profile, which is stored in the database and used as the basis for monitoring the job performance of the node (supercomputer) during the next operation phase. It is understood that the target job performance data can describe the job performance of the supercomputer (node) throughout the entire operation phase, and the supercomputer (node) can generate multiple job performance data at various time points or moments during the entire operation phase.

[0084] In some embodiments, in the process of determining the target job performance data, preliminary data screening can also be performed on the job performance data in the operation phase. The reason is that some job performance data are random, irregular or unstable. This type of job performance data cannot be used to detect job degradation of supercomputers (nodes), and it is necessary to screen job performance data with periodicity and stability. For example, in a short period of time, the change amplitude of the same type of job performance data exceeds the threshold. For example, taking the number of instructions executed per second as an example, within 10s, the data that switches between 5 million / second and 1 million / second for more than 10 times can be considered unstable data. It can be understood that the numerical values ​​here are all exemplary and are not limited in the embodiments of the present application.

[0085] In some embodiments, feature extraction may be performed on the job performance data after preliminary data screening as target job performance data. The process of feature extraction on the job performance data may include stage division, stage description, and stage organization.

[0086] Phase division is used to segment individual jobs. A job can be divided into multiple execution phases. That is, each execution phase corresponds to a job type. In some embodiments, based on the data characteristics of the job performance data corresponding to the job type, as shown in Figure 5, jobs are divided into three categories: computing jobs, storage (IO) jobs, and other jobs. These can also be referred to as phase descriptions: computing phase, IO phase, and other phases.

[0087] The stage description is used to add a characteristic description for each type of job, that is, to add corresponding job performance data to the job performance profile. For example, the job performance data for a compute job can be the number of instructions executed per second, while the job performance data for an IO job can be read and write IO performance, etc. Continuing with Figure 5, the vertical axis in Figure 5 can be expressed in instructions executed per second (mflops), that is, using mflops as the performance metric for evaluating jobs. Therefore, for the compute stage, the vertical axis in Figure 5 is larger than that for the IO stage, that is, more prominent on the vertical axis. If the vertical axis is expressed in IO bandwidth, the vertical axis value for the IO stage would be more prominent.

[0088] Phase sorting is used to sort and organize all job types, delete interfering job types, and merge duplicate job types.

[0089] In some embodiments, the job performance data may be filtered based on the job type. For example, the job performance data of the supercomputer (node) running the startup job or the warm-up job may be filtered out.

[0090] In some embodiments, it is also necessary to cluster and count the job types. Since each supercomputer (node) in the computer cluster has multiple nodes running jobs at the same time, that is, each node runs a high-concurrency job, but there are performance differences between each node. As shown in Figure 6, a clustering algorithm can be used, such as the k-means algorithm, to screen out nodes of different subclusters (clusters) (such as type 1, type 2, and type 3, etc.) and count the overall characteristics of nodes of the same type, and the job performance portrait of the running phase of the job corresponding to each node can be recorded by feature grouping. For example: for a computing job, the job performance portrait of the running phase of the computing job can record the number of instructions executed per second, etc.

[0091] S402: Storing the target job performance data as historical job performance data in a database.

[0092] Exemplarily, the target job performance data here may be the job performance data determined in step S401. Compared with the target job performance data, the historical job performance data here may have the same corresponding node information (node ​​type), job type or job number, and the time point of the running phase of the historical job performance data is earlier than the time point of the running phase of the target job performance data. The database here may be a public storage device accessible to all supercomputers (nodes) in the cluster, or a public storage device accessible to nodes in the cluster specifically used to perform job performance detection. In some embodiments, the database here may also be a public storage device, for example, it may be a data file.

[0093] In some embodiments, both the target job performance data and the historical job performance data are stored using job performance profiles. The job performance profile corresponding to the target job performance data can be matched with the job performance profiles of the historical job performance data in the database. If a match is successful, the target job performance data is used to update the historical job performance data in the database. It is understood that the matching here can be to match the job performance profiles of the historical job performance data with historical job performance data that is before the run time of the run phase of the target job performance data and that matches the job type and node information (node ​​type) or job number.

[0094] It can be seen that by using the job performance profile of job performance data, the job performance data and nodes during the running phase of the node running job can be described in more detail. The target job performance data replaces and updates the historical job performance data. During the operation of the computer cluster, the job performance profile can be continuously generated and updated, making the data basis for job degradation more accurate and timely.

[0095] S403: Acquire current job performance data, and match historical job performance data corresponding to the current job performance data.

[0096] For example, the current job performance data here can be newly acquired job performance data of a node in a computer cluster, that is, the job performance data of the first computing node after the current running phase of the current job (first job) has completed. In some embodiments, the newly acquired current job performance data can also be subjected to preliminary data screening and feature extraction to obtain a job performance profile corresponding to the current job performance data.

[0097] In some embodiments, the obtained job performance profile of the current job performance data is matched with the job performance profile of the historical job performance data in the database, that is, from the job performance profile of the historical job performance data, the historical job performance data that matches the job type of the current job in the current running stage and / or the node information (node ​​type) and / or job number of the node corresponding to the current running stage before the running time of the current running stage is matched. That is, the historical job performance data corresponding to the current job performance data that meets the first data condition is matched. The historical job performance data that matches the node information (node ​​type) of the node corresponding to the current running stage here, that is, the node information (node ​​type) between the node that executes the job corresponding to the current job performance data and the node that executes the job corresponding to the historical job performance data is matched. From the job performance profile of the historical job performance data here, the historical job performance data that matches the job number is matched, that is, the historical job performance data of the repeatedly executed current job (first job) is matched.

[0098] It can be understood that the amount of data of the current job performance data is large. For example, multiple supercomputers (nodes) in a computer cluster are running jobs concurrently, that is, multiple current jobs are running at the same time. The data that meets a preset proportion in the current running stage corresponding to the current job can be selected to determine the current job performance data, for example, 20% of the data, and the current job performance data is matched with the historical job performance data in the database. The matching process can include: determining the job performance profile corresponding to the current job performance data, that is, the job performance profile can include node information (node ​​type), job type or job number, etc. in addition to the previous job performance data. From the job performance profile of the historical job performance data, match the historical job performance data that is before the running time of the running stage of the current job performance data and matches the job type and node information (node ​​type) or job number.

[0099] S404: Determine whether the match is successful.

[0100] Exemplarily, if historical job performance data corresponding to the current job performance data is matched, step S405 is executed. If the match is not successful, step S402 can be executed to store the current job performance data as historical job performance data in the database. That is to say, the supercomputer (node) runs the job in the current running stage to obtain the current job performance data. If there is no historical job performance data, the current job performance data can be used as the historical job performance data for the supercomputer (node) to run the next same job or the same type of job.

[0101] S405: Perform operation performance data detection.

[0102] Exemplarily, the job performance data detection here can also be called profile detection, that is, by comparing the current job performance data and the historical job performance data in the job performance portrait, if there is a numerical difference (matching degree) between the current job performance data and the historical job performance data, or the numerical difference meets the preset difference condition, it can be considered that the supercomputer (node) has a performance difference when executing the job, for example: taking the number of instructions executed per second as an example, the current job performance data and the historical job performance data are 5 million / second and 5.1 million / second or 5.1 million / second and 5 million / second respectively, and the preset difference condition is that the numerical difference between the current job performance data and the historical job performance data is less than 50,000 / second. It is necessary to perform perception anomaly detection, that is, to further determine whether the numerical difference between the current job performance data and the historical job performance data meets the preset matching degree condition. It can be understood that the numerical values ​​here are all exemplary and are not limited in the embodiments of the present application.

[0103] S406: Perform anomaly detection.

[0104] For example, anomaly detection can continue to be performed on the supercomputer (node) using current job performance data and historical job performance data. In some embodiments, anomaly detection can be performed using historical job performance data that has the same node information (node ​​type), job type, and job number as the current job performance data. Alternatively, historical job performance data that has the same node information (node ​​type) and job type as the current job performance data can be used, that is, job performance data generated by the same supercomputer (node) running the same type of job.

[0105] It is understandable that in some embodiments, if no perception abnormality occurs, step S405 is continued to be executed regularly to perform operation performance data detection.

[0106] S407: Determine whether an abnormality occurs.

[0107] Exemplarily, if an exception occurs, step S408 is executed; otherwise, step S402 is executed to store the current job performance data as historical job performance data in the database.

[0108] In some embodiments, taking the current job performance data as the number of instructions executed per second as an example, if the numerical difference between the current job performance data and the historical job performance data satisfies a preset matching condition, i.e., the current job performance data is lower than the historical job performance data, and the numerical difference exceeds a preset numerical threshold, then it can be determined that an anomaly has occurred, i.e., job degradation has occurred. For example, continuing with the example of the number of instructions executed per second, if the current job performance data and the historical job performance data are 5 million and 5.1 million instructions per second, respectively, and the preset numerical threshold is 50,000 instructions per second, and the current job performance data has decreased compared to the historical job performance data, then it can be determined that the preset matching condition has been met, i.e., job degradation has occurred.

[0109] In another embodiment, taking the current job performance data as an example of an I / O (Input / Output) read / write rate, if the I / O read / write rate slows down, it can also be directly determined that the job is degraded.

[0110] S408: Return the detection result.

[0111] For example, when job degradation is determined, the corresponding detection results for the job degradation may also be returned. The detection results may include network issues, device hardware issues, and so on. The detection results may also be related to the job type. For example, the detection results for a computing job may include hardware failures, such as memory failures and processor failures; the detection results for a storage (IO) job may include network failures.

[0112] The detection results here can be determined by the following methods. For non-IO anomalies (network, computing read and write anomalies, etc.), the initial cause can be determined by the job performance data of the current running stage and the previous running stage in the job performance portrait. The initial cause determination here can be to determine the anomaly based on the preset job degradation judgment standard. For example, taking network anomalies as an example, the preset job degradation judgment standard corresponding to the initial cause determination may include: sending a connection request to the node within a preset time period (such as: 1s) but not receiving a response. For network and computing read and write anomalies, the network and computing read and write indicators of each node can also be compared, such as key performance indicators (KPIs), to screen out abnormal nodes and locate the root cause of the fault as the detection result. The key indicators here may include memory size, transmission bandwidth, etc.

[0113] In some embodiments, the data related to the key performance indicators (KPIs) described above may be based on network KPI analysis, network KPI monitoring, network KPI control notifications, and the like of supercomputers (nodes). Examples of KPI-related data include end-to-end (E2E) latency, network throughput, and network data rate. Examples of network KPI monitoring include: monitoring to determine the state of the network or network components corresponding to the node, such as indications of load and congestion levels; and monitoring predefined events (e.g., based on thresholds, conditions, and time). Examples of network KPI control notifications may include reporting to a network management node in a computer cluster. Network KPI control notifications may be used to help network management modify the network configuration of a node and adjust network resources.

[0114] In some embodiments, the aforementioned process for detecting network and computational read / write anomalies may also include integrating node information from all nodes; sequentially testing network latency and packet loss metrics to determine if a network anomaly exists; and using a k-means clustering similarity algorithm based on a distance matrix to filter out anomalous nodes. The detection results are then integrated.

[0115] Exemplarily, the job performance profile in step S401 is described by FIG7 . As shown in FIG7 , for example, the number of instructions executed per second for a computing job will be much higher than the number of instructions executed per second for a storage job. It is necessary to slice and classify the job performance data when the supercomputer (node) runs each job to obtain the job performance data corresponding to each type of job. Continuing to refer to FIG5 , the job performance profile may include: summary information used to describe the basic information of the job, including the job name (case_name), execution time (elapsed_time), number of nodes, etc. Basic information describing the node of the job, including the node name, etc. Job performance data (stage_info) corresponding to the job type, taking the computing job as an example, includes: number (num), number of instructions executed per second (flops), etc., wherein the number of instructions executed per second may also include numerical values ​​corresponding to the number of instructions executed per second, such as mean (mean), error rate (lower_limit), etc.

[0116] In some embodiments, the job performance profile may also include basic information about the master node (master_info) describing the job performance profile, including name, identification phase, end time of the identification phase, etc. For example, for read and write (I / O) jobs, this information includes I / O bandwidth, write latency, and I / O size ratio (large_io_ratio, small_io_ratio).

[0117] It can be seen that the detection method for job performance degradation using an autonomous description method as described in Figure 4 can convert the information corresponding to each historical job into a job performance portrait and store it in a database. That is, historical jobs are classified and stored according to the identification information corresponding to the historical jobs, such as node information (node ​​type), job type, job number, etc. For computer clusters running a large number of repetitive jobs, the historical jobs corresponding to the current job can be determined more quickly and accurately, thereby improving the efficiency of job management and improving the accuracy and robustness of anomaly detection.

[0118] In some embodiments, compared to the aforementioned method for detecting job performance degradation based on job performance profiles, FIG8 illustratively illustrates another scenario diagram of a system 800 for detecting job performance degradation. Referring to FIG8 , detection system 800 can be divided into two functional modules: a model training module 801 and an anomaly detection module 802.

[0119] In some embodiments, the model training module 801 is used to implement functions such as feature extraction, feature selection, and model training. First, the model training module 801 can obtain job performance data of known normal nodes and abnormal nodes. The model training module 801 can obtain job performance data within multiple continuous or non-continuous preset time periods in the form of a time series window. The time series window can include a normal resource usage window and an abnormal resource usage window. The normal resource usage window here can indicate that the job performance data used by the node to run the job in the time period corresponding to the time series window is normal. Relatively speaking, the abnormal resource usage window can indicate that the job performance data used by the node to run the job in the time period corresponding to the time series window is abnormal, for example: the CPU occupancy rate is too high, the memory occupancy rate is too high, etc.

[0120] The anomaly detection module 802 is used to monitor data at each node. That is, it extracts features from the data in real time, including abnormal and normal features. The features are input into the machine learning model, and the model output is used to determine whether an anomaly has occurred. If an anomaly occurs, an anomaly alert is issued.

[0121] As can be seen, the performance data used to train the model in detection system 800 shown in Figure 8 depends on the application's operating environment. Therefore, faults must be re-injected to detect anomalies based on the new environment. If the job executed by the computer cluster is updated, the model must be re-adapted, resulting in poor versatility: retraining is required for the new job. If the computer cluster is large, the hardware requirements of the computer equipment running detection system 800 are high, and real-time anomaly detection is expensive.

[0122] FIG9 is a schematic diagram of the structure of a supercomputer (node) provided in an embodiment of the present application. As shown in FIG9 , the supercomputer includes one or more processors 901, system memory 902, non-volatile memory (NVM) 903, communication interface 904, input / output (I / O) device 905, and system control logic 906 for coupling the processor 901, system memory 902, non-volatile memory 903, communication interface 904, and input / output (I / O) device 905. Among them:

[0123] The processor 901 may include one or more processing units, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a microprocessor (MCU), an AI (Artificial Intelligence) processor or a programmable logic device (FPGA) or a neural network processor (NPU), etc. The processing module or processing circuit may include one or more single-core or multi-core processors.

[0124] The system memory 902 is a volatile memory, such as a random-access memory (RAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0125] The non-volatile memory 903 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 903 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), a compact disc (CD), a digital versatile disc (DVD), a solid-state drive (SSD), etc. In some embodiments, the non-volatile memory 903 may also be a removable storage medium, such as a secure digital (SD) memory card.

[0126] In particular, the system memory 902 and the non-volatile memory 903 may respectively include a temporary copy and a permanent copy of the instruction 907. The instruction 907 may include: when executed by at least one of the processors 901, enabling the terminal device 1 to implement the data processing method provided in various embodiments of the present application.

[0127] The communication interface 904 may include a transceiver for providing a wired or wireless communication interface for the terminal device 1, thereby communicating with any other suitable device through one or more networks. In some embodiments, the communication interface 904 may be integrated into other components of the terminal device 1, for example, the communication interface 904 may be integrated into the processor 901. In some embodiments, the terminal device 1 can communicate with other devices through the communication interface 904. For example, the terminal device 1 can obtain a neural network model and a first lookup table and a second lookup table corresponding to the neural network model from other electronic devices through the communication interface 904.

[0128] The input / output (I / O) device 905 may include input devices such as a keyboard, a mouse, etc., and output devices such as a display, etc. A user may interact with the terminal device 1 through the input / output (I / O) device 905 .

[0129] The system control logic 906 may include any suitable interface controller to provide any suitable interface with other modules of the terminal device 1. For example, in some embodiments, the system control logic 906 may include one or more memory controllers to provide interfaces to the system memory 902 and the non-volatile memory 903.

[0130] It will be understood that the structure of the supercomputer shown in FIG9 is merely an example. In other embodiments, the supercomputer may include more or fewer components than shown, or may combine or separate certain components, or may have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0131] It should be understood that although the terms "first," "second," and the like may be used herein to describe various features, these features should not be limited by these terms. These terms are used merely to distinguish and should not be understood as indicating or implying relative importance. For example, a first feature may be referred to as a second feature, and similarly, a second feature may be referred to as a first feature, without departing from the scope of the exemplary embodiments.

[0132] Furthermore, various operations will be described as multiple, separate operations in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be construed to imply that the operations must be performed in order of description, and many of the operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can terminate when the described operations are completed, but can also have additional operations not included in the figures. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0133] References in the specification to "one embodiment," "an embodiment," "an illustrative embodiment," etc., indicate that the described embodiment may include a particular feature, structure, or property, but that every embodiment may or may not necessarily include the particular feature, structure, or property. Furthermore, these phrases are not necessarily referring to the same embodiment. Furthermore, while particular features may be described in conjunction with a specific embodiment, the knowledge of those skilled in the art may affect how those features may be combined with other embodiments, whether or not those embodiments are explicitly described.

[0134] Unless the context dictates otherwise, the terms "comprising," "having," and "including" are synonymous. The phrase "A / B" means "A or B." The phrase "A and / or B" means "(A), (B), or (A and B)."

[0135] As used herein, the term "module" may refer to, be part of, or include: memory (shared, dedicated, or group) for running one or more software or firmware programs, application-specific integrated circuits (ASICs), electronic circuits and / or processors (shared, dedicated, or group), combinational logic circuits, and / or other suitable components that provide the functionality.

[0136] In the accompanying drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order is not required. Rather, in some embodiments, these features may be illustrated in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular drawing does not mean that all embodiments need to include such features. In some embodiments, these features may not be included, or they may be combined with other features.

[0137] The embodiments of the present application are described in detail above with reference to the accompanying drawings. However, the application of the technical solution of the present application is not limited to the various applications mentioned in the embodiments of the present patent. Various structures and variations can be easily implemented with reference to the technical solution of the present application to achieve the various beneficial effects mentioned herein. Various changes made within the knowledge of ordinary technicians in this field without departing from the purpose of the present application should fall within the scope of coverage of the patent application.

Claims

1. A performance detection method, characterized in that: Applied to computing nodes in computer systems; and The method comprises: A first computing node in the computer system executes a first job; Collecting first performance data of the first computing node executing the first job; Acquire, from the historical job performance data, first historical performance data that satisfies a first data condition with the first performance data, wherein the first data condition includes: the first performance data and the first historical performance data have the same job type; Based on the matching degree between the first historical performance data and the first performance data, it is determined whether there is performance degradation in the process of the first computing node executing the first job.

2. The method according to claim 1, characterized in that The job type includes at least one of a storage job and a computing job.

3. The method according to claim 2, characterized in that The satisfying the first data condition further includes: The computing node that executes the job corresponding to the first historical performance data is of the same node type as the first computing node.

4. The method according to claim 3, characterized in that The satisfying the first data condition further includes: The job corresponding to the first historical performance data has the same job number as the first job, wherein the job number is used to indicate that the first job is a repeatable job.

5. The method according to claim 4, characterized in that The first historical performance data indicates that no performance degradation occurred when the first computing node executed the historical job in the corresponding historical time period.

6. The method according to claim 5, characterized in that Determining whether performance degradation occurs during execution of the first job by the first computing node based on a matching degree between the first historical performance data and the first performance data includes: Calculating the matching degree between the first performance data and the first historical performance data; When the matching degree satisfies a preset matching degree condition, it is determined that there is performance degradation in the process of the first computing node executing the first job, wherein the matching degree represents the data difference between the first performance data and the first historical performance data, and the matching degree satisfies the preset matching degree condition including: the absolute value of the matching degree is greater than the first preset matching degree or the absolute value of the matching degree is less than at least one of the second preset matching degree.

7. The method according to claim 6, characterized in that Also includes: In the case where performance degradation occurs during the execution of the first job by the first computing node, a detection result of the performance degradation is generated based on the first job type of the first job, wherein the detection result includes at least one of a hardware failure and a network failure.

8. The method according to claim 1, characterized in that The historical job performance data is stored in a public storage device of the computer system, wherein all computing nodes included in the computer system can access the historical job performance data.

9. The method according to claim 8, characterized in that Also includes: When it is determined that there is no performance degradation in the process of the first computing node executing the first job, the The first performance data updates the first historical performance data stored in the public storage device.

10. The method according to claim 9, characterized in that The public storage device stores the historical New Year data in the form of a database or a data file.

11. The method according to claim 1, characterized in that Also includes: A first computing node in the computer system executes a second job; Collecting second performance data of the first computing node executing the second job; When the second historical performance data that satisfies the first data condition with the second performance data is not obtained from the historical operation performance data, the second performance data is stored as the second historical performance data.

12. An electronic device, characterized in that: include: a memory for storing instructions to be executed by one or more processors of the electronic device, and The processor is one of the processors of the electronic device, and is used to execute the performance detection method described in any one of claims 1-11.

13. A computer program product, characterized in that include: A non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium contains a computer program code for executing the performance detection method described in any one of claims 1 to 11.

Citation Information

Patent Citations

  • Hadoop task scheduling method and device

    CN110618865A

  • Operation fault prediction method and device and electronic equipment

    CN112395167A

  • Large-scale cluster job anomaly detection method

    CN114116392A

  • Aperiodic data driven cognitive control system

    US11200989B1

Cited By

  • Enterprise management intelligent decision-making system integrated with big data analysis

    CN120122970A