Electronic equipment, performance detection method thereof and medium
By matching the historical job performance data, the operation performance deterioration of computer cluster nodes is accurately detected, and the problems of misjudgment and inaccurate evaluation in the prior art are solved, and the accuracy and robustness of the detection are improved.
Patent Information
- Application Number
- CN202310628502.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In computer clusters, it is difficult for the prior art to accurately detect deterioration in job performance, especially in the case of busy networks between nodes and unbalanced CPU load, resulting in misjudgment and inaccurate performance evaluation.
By collecting the performance data of the current job, matching it with the historical job performance data, and comparing it with the job performance portrait to determine whether there is performance deterioration, avoiding misjudgment caused by excessive differences in performance data in different job types and operation stages, and improving The accuracy of the detection.
Accurate detection of degradation in the performance of nodes of computer clusters is realized, misjudgment is reduced, detection accuracy and robustness is improved, and the performance of computer clusters can be managed and optimized more effectively.
Smart Images

Figure CN119065940A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of advanced computing devices, and more particularly to an electronic device and a performance detection method and medium thereof. Background Art
[0002] High Performance Computer (HPC), also known as supercomputer, is a high-performance computer device that can support multiple central processing units (CPU) or graphics processing units (GPU). In application scenarios with large data processing volume, such as extreme weather forecasting and big data computing, as shown in Figure 1(a), multiple supercomputers (nodes 1 to N) can be connected together through a network to work together to form a computer cluster with stronger operating performance, so as to process various tasks at a faster speed.
[0003] During the continuous operation of the computer cluster, the job performance of the computer cluster may deteriorate. Common job performance degradations may include: network congestion, CPU load imbalance, etc. Network congestion may be that the network between nodes is busy and data transmission is slow; CPU load imbalance may be that when a job is assigned to different nodes for execution, the CPU load of some nodes is too high, and the CPU load of other nodes is too low. The above job performance degradations will cause the job performance of the computer cluster to decrease. Therefore, it is necessary to provide a detection method for detecting whether each node in the computer cluster has job performance degradation. Summary of the invention
[0004] The present application provides an electronic device and a performance detection method and medium thereof.
[0005] In a first aspect, an embodiment of the present application provides a performance detection method, which is applied to a computing node in a computer system; and the performance detection method includes:
[0006] A first computing node in the computer system executes a first job;
[0007] Collecting first performance data of a first computing node executing a first job;
[0008] Acquire, from the historical operation performance data, first historical performance data that satisfies a first data condition with the first performance data, wherein the first data condition includes: the first performance data and the first historical performance data have the same operation type;
[0009] Based on the matching degree between the first historical performance data and the first performance data, it is determined whether there is performance degradation in the process of the first computing node executing the first job.
[0010] In the present application, the computer system here may include a computer cluster, the computing node may include a supercomputer (node) in the computer cluster, and the first computing node may be a supercomputer (node) in the computer cluster. The first job here may be the current job run by the first computing node, and the first performance data may be the current job performance data, that is, the job performance data newly acquired after the current running phase of the current job (first job) of the first computing node ends. The historical job performance data here may include the job performance data of the computing node obtained by running the job of each computing node in the computer cluster stored by the public storage device in the computer cluster. The first historical performance data may be one of a plurality of historical job performance data. The first performance data and the historical job performance data may be described by a job performance profile, and the job performance profile includes the job type and the like. The first data condition here includes matching the job performance profile of the obtained first performance data with the job performance profile of the historical job performance data, that is, matching the first historical performance data before the running time of the running phase of the first performance data and matching the job type of the first job of the first performance data from the job performance profile of the historical job performance data. The matching degree here may be the numerical difference between the first performance data and the first historical performance data.
[0011] It can be seen that the above-mentioned method for detecting job performance degradation avoids the situation where, when the same node runs jobs of different job types in multiple running stages, the numerical differences of job performance data generated in multiple running stages are large, and the node is misjudged as having job performance degradation; after a current running stage is completed, by looking up the job type, node information or job number corresponding to the current running stage and the same historical running stage that has been saved, the job performance data of the current running stage and the historical running stage are compared, so as to more accurately determine whether job performance degradation occurs in the current running stage.
[0012] In a possible implementation of the first aspect above, the job type includes at least one of a storage job and a computing job.
[0013] In a possible implementation of the first aspect, satisfying the first data condition further includes:
[0014] The computing node that executes the job corresponding to the first historical performance data is of the same node type as the first computing node.
[0015] In the present application, the first data condition here also includes matching the job performance portrait of the obtained first performance data with the job performance portrait of the historical job performance data, that is, matching the first historical performance data that matches the node information (node type) of the first computing node corresponding to the first performance data from the job performance portrait of the historical job performance data.
[0016] In a possible implementation of the first aspect, satisfying the first data condition further includes:
[0017] The job corresponding to the first historical performance data has the same job number as the first job, wherein the job number is used to indicate that the first job is a repeatable job.
[0018] In the present application, the first data condition here also includes matching the job performance portrait of the obtained first performance data with the job performance portrait of the historical job performance data, that is, matching the first historical performance data corresponding to the first job that has been repeatedly executed from the job performance portrait of the historical job performance data.
[0019] In a possible implementation of the first aspect above, the first historical performance data indicates that no performance degradation occurs when the first computing node executes the historical job in the corresponding historical time period.
[0020] In the present application, the first historical performance data may be job performance data obtained when the first computing node executes a historical job.
[0021] In a possible implementation of the first aspect, determining whether performance degradation occurs during execution of the first job by the first computing node based on a matching degree between the first historical performance data and the first performance data includes:
[0022] Calculating a matching degree between the first performance data and the first historical performance data;
[0023] When the matching degree satisfies a preset matching degree condition, it is determined that there is performance degradation in the process of the first computing node executing the first job, wherein the matching degree represents the data difference between the first performance data and the first historical performance data, and the matching degree satisfies the preset matching degree condition including: the absolute value of the matching degree is greater than the first preset matching degree or the absolute value of the matching degree is less than at least one of the second preset matching degree.
[0024] In the present application, the preset matching condition here may include that the first performance data is lower than the first historical performance data, and the numerical difference between the first performance data and the first historical performance data exceeds a preset numerical threshold.
[0025] In a possible implementation of the first aspect, the present invention further includes:
[0026] When performance degradation occurs during execution of a first job by a first computing node, a detection result of the performance degradation is generated based on a first job type of the first job, wherein the detection result includes at least one of a hardware failure and a network failure.
[0027] In a possible implementation of the first aspect above, the historical job performance data is stored in a public storage device of a computer system, wherein all computing nodes included in the computer system can access the historical job performance data.
[0028] In a possible implementation of the first aspect, the present invention further includes:
[0029] When it is determined that there is no performance degradation in the process of the first computing node executing the first job, the first performance data is used to update the first historical performance data stored in the public storage device.
[0030] In a possible implementation of the first aspect above, the public storage device stores historical New Year data in the form of a database or a data file.
[0031] In a possible implementation of the first aspect, the present invention further includes:
[0032] A first computing node in the computer system executes a second job;
[0033] Collecting second performance data of the first computing node executing the second job;
[0034] When the second historical performance data that satisfies the first data condition with the second performance data is not obtained from the historical operation performance data, the second performance data is stored as the second historical performance data.
[0035] In the present application, if the second historical performance data is not matched, the second performance data is stored as the second historical performance data in a public storage device. That is to say, when the first computing node obtains the second performance data when running the second job, there is no historical job performance data. The second performance data can be used as the historical job performance data for the first computing node to run the next same job or the same type of job.
[0036] In a second aspect, an embodiment of the present application provides an electronic device, including:
[0037] a memory for storing instructions to be executed by one or more processors of the electronic device, and
[0038] The processor is one of the processors of the electronic device and is used to execute the performance detection method of the first aspect.
[0039] In a third aspect, an embodiment of the present application provides a computer program product, including: a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium containing a computer program code for executing the performance detection method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] FIG1( a ) is a schematic diagram of a computer cluster provided in an embodiment of the present application;
[0041] FIG1( b ) is a flow chart of a method for detecting operating performance degradation provided in an embodiment of the present application;
[0042] Figure 2 A schematic diagram showing the change in the number of executed instructions of a node over time during the process of a node running a job provided in an embodiment of the present application;
[0043] Figure 3 A system architecture diagram of a detection method for a node of a computer cluster provided in an embodiment of the present application;
[0044] Figure 4 A method flow chart of a method for detecting operating performance degradation provided in an embodiment of the present application;
[0045] Figure 5 A schematic diagram of a type of operation provided in an embodiment of the present application;
[0046] Figure 6 A schematic diagram of clustering nodes provided in an embodiment of the present application;
[0047] Figure 7 A schematic diagram of a work performance portrait provided in an embodiment of the present application;
[0048] Figure 8 A schematic diagram of a system for detecting operating performance degradation provided in an embodiment of the present application;
[0049] Fig. 9 A schematic diagram of the structure of a supercomputer (node) provided in an embodiment of the present application. DETAILED DESCRIPTION
[0050] In order to make the objectives, embodiments and advantages of the present application more clear, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0051] FIG1(b) exemplarily shows a method flow chart of a method for detecting job performance degradation. The method for detecting job performance degradation shown in FIG1(b) can be run by one or more detection nodes, and the detection node can be a supercomputer in a computer cluster. A functional module for running the method for detecting job performance degradation can be set on the detection node, and the functional module is used to detect the job performance data corresponding to all nodes in the computer cluster and determine whether there is a problem of job performance degradation on any node.
[0052] As shown in FIG. 1( b ), the method for detecting degradation of operating performance may include the following steps.
[0053] S101: Detecting that a job is running.
[0054] S102: Read the log to obtain job performance data.
[0055] For example, the job performance data here may include: the number of instructions executed per second, CPU usage, memory usage, etc.
[0056] S103: Determine whether the job performance data does not meet the performance threshold. For example, taking the number of instructions executed per second as an example, if the number of instructions executed per second is lower than the performance threshold, execute step S105. If the job performance data is higher than or equal to the performance threshold, execute step S104.
[0057] S104: Determine whether the operation duration is greater than a duration threshold.
[0058] Exemplarily, if the operation duration is higher than the duration threshold, step S105 is executed: determine whether a performance anomaly has occurred. Otherwise, you can return to step S101, wait for a certain period of time, and continue to perform the next detection of operation performance degradation. In other words, the method for detecting operation performance degradation shown in Figure 1(b) can be executed periodically by the system. For example: when no abnormality is detected after executing step S103 and step S104, the detection can be ended. After waiting for a certain period of time (for example: 5 minutes), re-execute step S101 and restart monitoring. It can be understood that the numerical values here are exemplary and are not limited to this in the embodiments of the present application.
[0059] S105: Determine that a performance abnormality occurs.
[0060] Exemplarily, that is, it is determined that the operation performance degradation occurs.
[0061] For example, Figure 2The diagram shows the change of the number of executed instructions of a node over time during the process of a node running a job. It can be understood that the more instructions a node executes per unit time, the better the node's operating performance. By comparing the number of executed instructions of each node in the computer cluster, the operating performance of the entire computer cluster can be evaluated. Figure 2 The process of a node running a job is described by using time and the number of executed instructions as job performance data. The process of a node running a job can be divided into multiple continuous running stages. In each running stage, the job performance data of the node in the running stage is different according to the job type of the job run by the node. For example, taking stage 1 and stage 2 as examples, in stage 1, the node executes a low-load job, and in stage 2, the node executes a high-load job, so that the number of instructions of the node in stage 1 is lower than that in stage 2. The high load and low load here can be used to describe the number of floating-point operations or instructions that a node can complete in a unit of time, which can be expressed by the number of instructions executed per second (Floating Point Operations PerSecond, FLOPS).
[0062] Below is a combination of Figure 1 and Figure 2 Describe a method for detecting the degradation of node job performance. For example, set the performance threshold of the number of instructions executed per second to 5 million per second. Figure 2 In the stage 1 and stage 4 shown in FIG. 1 , the number of instructions executed per second by the node is 1 million per second, and in the stage 2, stage 3, and stage 5, the number of instructions executed per second by the node is 10 million per second. That is to say, if the method for detecting job performance degradation shown in FIG. 1 is adopted, that is, only the job performance data of each stage from stage 1 to stage 5 is compared with the performance threshold, and the method for confirming that the node has job performance degradation in the stage that does not meet the performance threshold is confirmed. Figure 2 The nodes shown have job performance degradation in phases 1 and 4.
[0063] It can be seen that if the detection method for performance degradation described in Figure 1 is used to Figure 2 In other words, only threshold judgment is used to detect job performance degradation. Different thresholds need to be set for different types of nodes and different types of jobs, which increases the complexity of detecting job performance degradation. At the same time, if the association between the running stages that the same node has passed is not considered, that is, referring to the running time corresponding to the running stage of the same node and the job type of the job, if adjacent running stages correspond to jobs of different job types, the job performance data corresponding to the running stages differ greatly. Detecting job performance degradation only through threshold judgment will cause inaccurate (misjudgment) problems, such as Figure 2 Phase 1 and Phase 4 of the
[0064] In order to solve the problem of inaccuracy in the detection process of job performance degradation, an embodiment of the present application proposes a method for detecting job performance degradation, which stores the job performance data of the node's historical operation stage (hereinafter referred to as historical job performance data), and when detecting whether the current job performance data of the current operation stage is normal, after the current operation stage ends, the historical job performance data before the running time of the current operation stage and matching the job type, node information or job number of the current job performance data are matched from the historical job performance data. If the data difference between the current job performance data and the historical job performance data meets the performance degradation condition, it is determined that the node has job performance degradation in the current operation stage.
[0065] It can be understood that since the stored historical job performance data is normal, it is possible to determine whether the node has experienced job performance degradation by looking at the data difference between the current job performance data and the historical job performance data. For example, if the job performance data is the number of instructions executed per second, if the number of instructions executed per second in the current operation phase is much lower than the number of instructions executed per second in the historical operation phase, it is determined that the node has experienced job performance degradation in the current operation phase.
[0066] In some embodiments, if the node does not experience job performance degradation in the current operation stage, the current job performance data is used to update the historical job performance data for job performance degradation detection of the job performance data of the next operation stage. It can be understood that the job performance data, job type, node information (node type), and running time or job number corresponding to the operation stage of each job of a node here can be described by the job performance portrait of the operation stage. The job number here is used to identify repeatable jobs. It is set when the job is generated for the first time. The job number can be used to match whether there is historical job performance data for the same job.
[0067] It can be seen that the above-mentioned method for detecting job performance degradation avoids the situation where the same node runs jobs of different job types in multiple running stages, and the numerical differences of job performance data generated in multiple running stages are large, which leads to the node being misjudged as having job performance degradation. After a current running stage is completed, by looking up the job type, node information (node type) or job number corresponding to the current running stage and the same historical running stage that has been saved, the job performance data of the current running stage and the historical running stage are compared, so as to more accurately determine whether job performance degradation occurs in the current running stage.
[0068] In order to more clearly and in detail introduce the detection method provided by the embodiment of the present application, the electronic device involved in implementing the method provided by the embodiment of the present application is first introduced below. The electronic device can be a computer, a supercomputer, and a computer cluster composed of multiple computers or supercomputers. It is understood that the embodiment of the present application does not impose any restrictions on the specific type of the electronic device.
[0069] Figure 3 The system architecture diagram of the detection method for nodes of a computer cluster provided in an embodiment of the present application is exemplarily shown.
[0070] like Figure 3 As shown, the system architecture 300 includes an online job feature construction module 301 and a job performance degradation perception module 302, wherein the online job feature construction module 301 also includes: a job performance feature extraction unit 3011 and a job performance feature merging unit 3012; the job performance degradation perception module 302 also includes: a job performance feature matching unit 3021, a contour detection unit 3022, an anomaly detection unit 3023 and an initial cause delimitation unit 3024.
[0071] The job performance feature extraction unit 3011 is used to collect information of each node of the computer cluster, including: job performance data, job type, node information (node type), and running time or job number, etc. Through the feature description method, the collected information is feature described to extract the job performance profile, that is, the corresponding relationship between the job performance data, job type, node information (node type), and running time or job number is established, which can also be called job performance profiling.
[0072] The job performance feature merging unit 3012 is used to merge the extracted job performance portrait, i.e., job performance data, job type, node information (node type) and running time or job number, with the job performance portrait saved in the job performance feature library, that is, to update and replace the existing job performance portrait, for job performance feature matching during job performance degradation.
[0073] The operation performance feature matching unit 3021 is used to match the historical operation performance profiles before the running time of the current running stage and matching the current operation performance profile from the operation performance feature library. That is, the historical operation performance profiles before the running time of the current running stage and matching the operation type and node information (node type) or operation number in the current operation performance profile are matched.
[0074] The profile detection unit 3022 is used to determine whether the current job performance data in the current job performance portrait has job performance degradation through the historical job performance data in the historical job performance portrait matched by the job performance feature matching unit 3021. It can also be called obtaining the performance profile of the historical job performance data, regularly sensing and detecting the performance profile of the current job performance data, and determining whether the current job performance data has job performance degradation. In some embodiments, the profile detection unit 3022 can detect the current job performance data in a periodic manner, or it can uniformly detect the current job performance data after the current operation stage ends. If the perception detects that the job performance is degraded, enter the abnormality detection unit 3023; otherwise, the current job performance portrait is used to update the historical job performance portrait, that is, the current job performance data is used to update the historical job performance data, which is used to detect job performance degradation of the job performance data of the next operation stage, which can also be called updating the performance baseline.
[0075] The anomaly detection unit 3023 is used to call the anomaly detection algorithm to perform anomaly detection after the profile detection unit 3022 senses and detects the performance degradation of the job. For example, the current performance parameters of the current master node of the computer cluster are compared with the historical performance parameters to sense the performance degradation of the node. If an anomaly occurs, the initial cause delimitation unit 3024 is further used to determine whether the anomaly is a hardware failure.
[0076] The initial cause delimiting unit 3024 is used to analyze the root cause of the fault (network or hardware, etc.) and return the fault node information after the perception detection operation performance degradation occurs.
[0077] After introducing the system architecture of the detection method for nodes in a computer cluster, Figure 4 A flow chart of the detection method of the embodiment of the present application is introduced, Figure 4 The method shown can be implemented by executing relevant instructions by a processor of one or more nodes (supercomputers) in a computer cluster. The detection method can include the following steps. It can be understood that one or more nodes (supercomputers) here can be used as nodes specifically used to perform job performance detection.
[0078] S401: Determine target job performance data.
[0079] For example, the target job performance data may be various job performance data of each supercomputer (node) in the computer cluster when running a job. The job performance data may include the number of instructions executed per second, CPU usage, memory usage, and the like.
[0080] After the running phase of the running job is completed or during the running phase of the running job, the target job performance data can be obtained by extracting features from the job performance data corresponding to the running phase. In some embodiments, the corresponding associated information can also be obtained, including basic features such as node information (node type), job type, job number, and running phase or job number; the target job performance data can be the mean, mean square error value or covariance value of multiple job performance data in the entire running phase, etc.
[0081] The target job performance data and related information form a job performance profile, which is used to be stored in the database as a basis for detecting the job performance of the node (supercomputer) in the next operation stage. It can be understood that the target job performance data can describe the job performance of the supercomputer (node) in the entire operation stage, and the supercomputer (node) can generate multiple job performance data at various time points or moments in the entire operation stage.
[0082] In some embodiments, in the process of determining the target job performance data, preliminary data screening can also be performed on the job performance data in the operation phase. The reason is that some job performance data is random, irregular or unstable. This type of job performance data cannot be used to detect job degradation of supercomputers (nodes), and it is necessary to screen job performance data with periodicity and stability. For example, in a short period of time, the change range of the same type of job performance data exceeds the threshold data. For example, taking the number of instructions executed per second as an example, within 10 seconds, the number of instructions executed per second switches between 5 million / second and 1 million / second for more than 10 times, which can be unstable data. It can be understood that the numerical values here are exemplary and are not limited to the embodiments of the present application.
[0083] In some embodiments, feature extraction may be performed on the job performance data after preliminary data screening as target job performance data. The process of feature extraction on the job performance data may include stage division, stage description, and stage organization.
[0084] Stage division is used to divide each job into multiple operation stages, that is, one operation stage can correspond to one job type. In some embodiments, according to the data characteristics of the job performance data corresponding to the job type, such as Figure 5 As shown, jobs are divided into three categories: computing jobs, storage (IO) jobs, and other jobs, which can also be called stage descriptions, computing stages, IO stages, and other stages.
[0085] The stage description is used to add feature descriptions for each type of job, that is, to add corresponding job performance data to the job performance profile. For example, the job performance data of a computing job can be the number of instructions executed per second, while the job performance data of an IO job can be the read and write IO performance, etc. Figure 5 , Figure 5 The vertical axis can be expressed in terms of instructions executed per second (mflops), that is, mflops is used as the performance indicator for evaluating the job. Therefore, for the calculation stage, Figure 5 The value of the ordinate in the IO stage is larger than that in the IO stage, that is, it is more obvious in the ordinate; if the ordinate is expressed in IO bandwidth, the value of the ordinate in the IO stage will be more obvious.
[0086] Phase sorting is used to sort and organize all job types, delete interfering job types, and merge duplicate job types.
[0087] In some embodiments, the job performance data may be filtered according to the job type. For example, the job performance data of the supercomputer (node) running the startup job or the warm-up job may be filtered out.
[0088] In some embodiments, it is also necessary to cluster and count the job types. Since each supercomputer (node) in a computer cluster has multiple nodes running jobs simultaneously, that is, each node runs high-concurrency jobs, but there are performance differences between the nodes, such as Figure 6 As shown, clustering algorithms, such as the k-means algorithm, can be used to filter out nodes of different sub-clusters (cluster types) (such as type 1, type 2, and type 3, etc.) and count the overall characteristics of nodes of the same type, and the content of the feature grouping can be recorded in the job performance portrait of the running stage of the job corresponding to each node. For example, for a computing job, the job performance portrait of the running stage of the computing job can record the number of instructions executed per second, etc.
[0089] S402: Storing the target operation performance data as historical operation performance data in a database.
[0090] Exemplarily, the target job performance data here may be the job performance data determined in step S401. Compared with the target job performance data, the node information (node type), job type or job number corresponding to the historical job performance data here may be the same, and the time point of the running phase of the historical job performance data is earlier than the time point of the running phase of the target job performance data. The database here may be a public storage device accessible to each supercomputer (node) in the cluster, or it may be a public storage device accessible to nodes in the cluster specifically used to perform job performance detection. In some embodiments, the database here may also be a public storage device, for example: it may be a data file.
[0091] In some embodiments, both the target job performance data and the historical job performance data are stored through job performance portraits, and the job performance portrait corresponding to the target job performance data can be matched with the job performance portrait of the historical job performance data in the database. If the match is successful, the target job performance data is used to update the historical job performance data in the database. It can be understood that the match here can be to match the historical job performance data that is before the running time of the running stage of the target job performance data and matches the job type and node information (node type) or job number from the job performance portrait of the historical job performance data.
[0092] It can be seen that by using the job performance profile of job performance data, the job performance data and nodes during the running phase of the node running job can be described in more detail. The target job performance data replaces and updates the historical job performance data. During the operation of the computer cluster, the job performance profile can be continuously generated and updated, making the data basis for job degradation more accurate and timely.
[0093] S403: Acquire current operation performance data, and match historical operation performance data corresponding to the current operation performance data.
[0094] Exemplarily, the current job performance data here may be the job performance data of a node in a computer cluster, that is, the job performance data of a node newly acquired after the current running phase of the current job (first job) of the first computing node ends. In some embodiments, the newly acquired current job performance data may also be subjected to preliminary data screening and feature extraction to obtain a job performance profile corresponding to the current job performance data.
[0095] In some embodiments, the obtained job performance portrait of the current job performance data is matched with the job performance portrait of the historical job performance data in the database, that is, the historical job performance data that matches the job type of the current job in the current running stage and / or the node information (node type) and / or job number of the node corresponding to the current running stage is matched from the job performance portrait of the historical job performance data. That is, the historical job performance data corresponding to the current job performance data that meets the first data condition is matched. The historical job performance data that matches the node information (node type) of the node corresponding to the current running stage here, that is, the node information (node type) between the node that executes the job corresponding to the current job performance data and the node that executes the job corresponding to the historical job performance data matches. Here, from the job performance portrait of the historical job performance data, the historical job performance data that matches the job number is matched, that is, the historical job performance data of the current job (first job) that is repeatedly executed is matched.
[0096] It can be understood that the data volume of the current job performance data is large, for example: multiple supercomputers (nodes) in a computer cluster are running jobs concurrently, that is, multiple current jobs are running at the same time, and the data that meets the preset proportion in the current operation phase corresponding to the current job can be selected to determine the current job performance data, for example: 20% of the data, and the current job performance data is matched with the historical job performance data in the database. The matching process can include: determining the job performance portrait corresponding to the current job performance data, that is, the job performance portrait can include node information (node type), job type or job number, etc. in addition to the previous job performance data. From the job performance portrait of the historical job performance data, match the historical job performance data before the running time of the running phase of the current job performance data and matching the job type and node information (node type) or job number.
[0097] S404: Determine whether the match is successful.
[0098] Exemplarily, if historical job performance data corresponding to the current job performance data is matched, step S405 is executed. If the match is not successful, step S402 can be executed to store the current job performance data as historical job performance data in the database. That is to say, the supercomputer (node) runs the job in the current running stage to obtain the current job performance data. If there is no historical job performance data, the current job performance data can be used as the historical job performance data for the supercomputer (node) to run the next same job or the same type of job.
[0099] S405: Perform operation performance data detection.
[0100] Exemplarily, the job performance data detection here can also be called profile detection, that is, by comparing the current job performance data and the historical job performance data in the job performance portrait, if there is a numerical difference (matching degree) between the current job performance data and the historical job performance data, or the numerical difference meets the preset difference condition, it can be considered that the supercomputer (node) has a performance difference when executing the job, for example: taking the number of instructions executed per second as an example, the current job performance data and the historical job performance data are 5 million / second and 5.1 million / second or 5.1 million / second and 5 million / second, respectively, and the preset difference condition is that the numerical difference between the current job performance data and the historical job performance data is less than 50,000 / second. It is necessary to perform perception anomaly detection, that is, to further determine whether the numerical difference between the current job performance data and the historical job performance data meets the preset matching degree condition. It can be understood that the numerical values here are all exemplary and are not limited in the embodiments of the present application.
[0101] S406: Perform anomaly detection.
[0102] Exemplarily, the supercomputer (node) may continue to be detected for anomalies using the current job performance data and historical job performance data. In some embodiments, anomaly detection may be performed using historical job performance data with the same node information (node type), job type, and job number as the current job performance data, or historical job performance data with the same node information (node type) and job type as the current job performance data, that is, job performance data generated by the same supercomputer (node) running the same type of job.
[0103] It can be understood that in some embodiments, if no perception abnormality occurs, step S405 is continued to be executed regularly to perform operation performance data detection.
[0104] S407: Determine whether an abnormality occurs.
[0105] Exemplarily, if an abnormality occurs, step S408 is executed; otherwise, step S402 is executed to store the current job performance data as historical job performance data in the database.
[0106] In some embodiments, taking the current job performance data as the number of instructions executed per second as an example, if the numerical difference between the current job performance data and the historical job performance data satisfies the preset matching condition, that is, the current job performance data is lower than the historical job performance data, and the numerical difference exceeds the preset numerical threshold, it can be determined that an abnormality has occurred, that is, job degradation has occurred. For example: continuing to take the number of instructions executed per second as an example, the current job performance data and the historical job performance data are 5 million / second and 5.1 million / second respectively, the preset numerical threshold is 50,000 / second, and the current job performance data has decreased compared to the historical job performance data, it can be determined that the preset matching condition is met, that is, job degradation has occurred.
[0107] In another embodiment, taking the current job performance data as the I / O (Input / Output) read / write rate as an example, if the I / O read / write rate slows down, it can also be directly determined that the job is degraded.
[0108] S408: Return the detection result.
[0109] Exemplarily, when it is determined that a job degradation occurs, the detection result corresponding to the job degradation may also be returned at the same time, and the detection result includes: network problem, device hardware problem, etc. The detection result may also be related to the job type. For example, the detection result of a computing job may include hardware failure, such as: memory failure, processor failure, etc.; the detection result of a storage (IO) job may include a network failure.
[0110] The detection results here can be determined by the following methods. For non-IO anomalies (network, computing read and write anomalies, etc.), the initial cause can be determined by the job performance data of the current running stage and the previous running stage in the job performance portrait. The initial cause determination here can be to determine the anomaly based on the preset job degradation judgment standard. For example, taking network anomalies as an example, the preset job degradation judgment standard corresponding to the initial cause determination may include: sending a connection request to the node within a preset time period (such as: 1s) but not receiving a response. For network and computing read and write anomalies, the network and computing read and write indicators of each node can also be compared, such as: key performance indicators (Key Performance Indicator, KPI), to screen out abnormal nodes and locate the root cause of the fault as the detection result. The key indicators here may include memory size, transmission bandwidth, etc.
[0111] In some embodiments, the data related to the above-mentioned key performance indicators (KPIs) can be based on network KPI analysis, network KPI monitoring, network KPI control notifications, etc. of supercomputers (nodes). Examples of KPI-related data include end-to-end (E2E) delay, network throughput, and network data rate. Examples of network KPI monitoring include: monitoring to determine the network or network component status corresponding to the node, such as indications of violations of load and congestion levels; and monitoring predefined events (e.g., based on thresholds, conditions, and time). Examples of network KPI control notifications can include reporting to a network management node in a computer cluster. Network KPI control notifications can be used to help network management modify the network configuration of a node and adjust network resources.
[0112] In some embodiments, the above-mentioned detection process of the corresponding network and computing read and write anomalies may also include integrating the node information of all nodes; sequentially detecting the network delay and packet loss indicators to determine whether there is a network anomaly. Using the k-means clustering similarity algorithm based on the distance matrix to filter out abnormal nodes. Integrate the detection results.
[0113] For example, by Figure 7 The job performance profile in step S401 is described as follows: Figure 7 For example, the number of instructions executed per second for computing jobs is much higher than the number of instructions executed per second for storage jobs. It is necessary to slice and classify the job performance data of each job when the supercomputer (node) runs each job to obtain the job performance data corresponding to each type of job. Figure 5 , the job performance profile may include: summary information used to describe the basic information of the job, including the job name (case_name), execution time (elapsed_time), number of nodes, etc. Describe the basic information of the node of the job, including the node name, etc. The job performance data (stage_info) corresponding to the job type, taking the computing job as an example, includes: number (num), number of instructions executed per second (flops), etc., where the number of instructions executed per second can also include the value corresponding to the number of instructions executed per second, such as: mean (mean), error rate (lower_limit), etc.
[0114] In some embodiments, the job performance profile may further include: basic information of the master node (master_info) describing the job performance profile, including: name, identification phase, end time of the identification phase, etc. Taking read and write (I / O) jobs as an example, it includes: I / O bandwidth, write latency, I / O size ratio (large_io_ratio, small_io_ratio), etc.
[0115] It can be seen that through Figure 4The method for detecting job performance degradation using an autonomous description method can convert the information corresponding to each historical job into a job performance portrait and store it in a database, that is, the historical jobs are classified and stored by the identification information corresponding to the historical jobs, such as node information (node type), job type, job number, etc. For a computer cluster running a large number of repetitive jobs, the historical job corresponding to the current job can be determined more quickly and accurately, thereby improving the efficiency of job management and the accuracy and robustness of anomaly detection.
[0116] In some embodiments, compared with the above-mentioned method for detecting operation performance degradation based on the above-mentioned operation performance profile, Figure 8 Another schematic diagram of a system 800 for detecting degradation of operating performance is also shown. Figure 8 , the detection system 800 can be divided into two functional modules: a model training module 801 and an anomaly detection module 802 .
[0117] In some embodiments, the model training module 801 is used to implement the functions of feature extraction, feature selection and model training. First, the model training module 801 can obtain the job performance data of known normal nodes and abnormal nodes. The model training module 801 can obtain the job performance data within multiple continuous or non-continuous preset time periods in the form of a time series window. The time series window may include a normal resource usage window and an abnormal resource usage window. The normal resource usage window here can indicate that the job performance data used by the node to run the job in the time period corresponding to the time series window is normal. Relatively speaking, the abnormal resource usage window can indicate that the job performance data used by the node to run the job in the time period corresponding to the time series window is abnormal, for example: the CPU occupancy rate is too high, the memory occupancy rate is too high, and so on.
[0118] The anomaly detection module 802 is used to monitor data at each node, that is, to extract features from the data in real time, including abnormal features and normal features. The features are input into the machine learning model, and whether an anomaly occurs is determined based on the model output results. When an anomaly occurs, an abnormal alarm is issued.
[0119] It can be seen that Figure 8 The performance data used for training the model in the detection system 800 shown depends on the operating environment of the application, so it is necessary to re-inject faults and perform anomaly injection based on the new environment. If the job executed by the computer cluster is updated, the model needs to be re-adapted, and the versatility is poor: retraining is required for new jobs. If the scale of the computer cluster is large, the hardware requirements of the computer equipment running the detection system 800 are high, and the overhead of real-time anomaly detection is high.
[0120] Fig. 9Schematic diagram of the structure of a supercomputer (node) provided in an embodiment of the present application. Fig. 9 As shown, the supercomputer includes one or more processors 901, system memory 902, non-volatile memory (NVM) 903, communication interface 904, input / output (I / O) device 905, and system control logic 906 for coupling the processor 901, system memory 902, non-volatile memory 903, communication interface 904 and input / output (I / O) device 905. Among them:
[0121] The processor 901 may include one or more processing units, for example, a central processing unit CPU (Central Processing Unit), an image processor GPU (Graphics Processing Unit), a digital signal processor DSP (Digital Signal Processor), a microprocessor MCU (Micro-programmed Control Unit), an AI (Artificial Intelligence) processor or a programmable logic device FPGA (Field Programmable Gate Array), a neural network processor (Neural-network Processing Unit, NPU), etc. The processing module or processing circuit may include one or more single-core or multi-core processors.
[0122] The system memory 902 is a volatile memory, such as a random-access memory (RAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), etc.
[0123] The non-volatile memory 903 may include one or more tangible, non-temporary computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 903 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), a compact disc (CD), a digital versatile disc (DVD), a solid-state drive (SSD), etc. In some embodiments, the non-volatile memory 903 may also be a removable storage medium, such as a secure digital (SD) memory card, etc.
[0124] In particular, the system memory 902 and the non-volatile memory 903 may respectively include: a temporary copy and a permanent copy of the instruction 907. The instruction 907 may include: when executed by at least one of the processors 901, the terminal device 1 implements the data processing method provided in each embodiment of the present application.
[0125] The communication interface 904 may include a transceiver for providing a wired or wireless communication interface for the terminal device 1, and then communicating with any other suitable device through one or more networks. In some embodiments, the communication interface 904 may be integrated into other components of the terminal device 1, for example, the communication interface 904 may be integrated into the processor 901. In some embodiments, the terminal device 1 may communicate with other devices through the communication interface 904, for example, the terminal device 1 may obtain a neural network model and a first lookup table and a second lookup table corresponding to the neural network model from other electronic devices through the communication interface 904.
[0126] The input / output (I / O) device 905 may include input devices such as a keyboard, a mouse, etc., and output devices such as a display, etc. A user may interact with the terminal device 1 through the input / output (I / O) device 905 .
[0127] The system control logic 906 may include any suitable interface controller to provide any suitable interface with other modules of the terminal device 1. For example, in some embodiments, the system control logic 906 may include one or more memory controllers to provide interfaces to the system memory 902 and the non-volatile memory 903.
[0128] Understandably, Fig. 9The structure of the supercomputer shown is only an example. In other embodiments, the supercomputer may include more or fewer components than shown, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0129] It should be understood that although the terms "first", "second", etc. may be used herein to describe various features, these features should not be limited by these terms. These terms are used only to distinguish and should not be understood as indicating or implying relative importance. For example, a first feature may be referred to as a second feature, and similarly a second feature may be referred to as a first feature without departing from the scope of the exemplary embodiments.
[0130] In addition, various operations will be described as multiple separate operations in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be interpreted as implying that these operations must rely on the order of description, and many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can also be rearranged. When the described operations are completed, the process can be terminated, but there can also be additional operations not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.
[0131] References in the specification to "one embodiment," "an embodiment," "an illustrative embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or property, but each embodiment may or may not necessarily include the particular feature, structure, or property. Moreover, these phrases are not necessarily directed to the same embodiment. In addition, when particular features are described in conjunction with a specific embodiment, the knowledge of those skilled in the art can affect the combination of these features with other embodiments, whether or not these embodiments are explicitly described.
[0132] Unless the context dictates otherwise, the terms "comprising," "having," and "including" are synonymous. The phrase "A / B" means "A or B." The phrase "A and / or B" means "(A), (B), or (A and B)."
[0133] As used herein, the term "module" may refer to, be part of, or include: a memory (shared, dedicated, or group) for running one or more software or firmware programs, an application-specific integrated circuit (ASIC), an electronic circuit and / or processor (shared, dedicated, or group), a combinational logic circuit, and / or other suitable components that provide the described functionality.
[0134] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order is not required. Instead, in some embodiments, these features may be described in a manner and / or order different from that shown in the illustrative drawings. In addition, the structural or method features included in a specific figure do not mean that all embodiments need to include such features. In some embodiments, these features may not be included, or these features may be combined with other features.
[0135] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the use of the technical solution of the present application is not limited to the various applications mentioned in the embodiments of the present patent. Various structures and variations can be easily implemented with reference to the technical solution of the present application to achieve the various beneficial effects mentioned herein. Various changes made within the knowledge of ordinary technicians in the field without departing from the purpose of the present application should all fall within the scope of the patent application.
Claims
1. A performance detection method, characterized in that: Applied to computing nodes in computer systems; and The method comprises: A first computing node in the computer system executes a first job; Collecting first performance data of the first computing node executing the first job; Acquire, from the historical job performance data, first historical performance data that satisfies a first data condition with the first performance data, wherein the first data condition includes: the first performance data and the first historical performance data have the same job type; Based on the matching degree between the first historical performance data and the first performance data, it is determined whether there is performance degradation in the process of the first computing node executing the first job.
2. The method according to claim 1, characterized in that The job type includes at least one of a storage job and a computing job.
3. The method according to claim 2, characterized in that The satisfying the first data condition further includes: The computing node that executes the job corresponding to the first historical performance data is of the same node type as the first computing node.
4. The method according to claim 3, characterized in that The satisfying the first data condition further includes: The job corresponding to the first historical performance data has the same job number as the first job, wherein the job number is used to indicate that the first job is a repeatable job.
5. The method according to claim 4, characterized in that The first historical performance data indicates that no performance degradation occurred when the first computing node executed the historical job in the corresponding historical time period.
6. The method according to claim 5, characterized in that Determining whether performance degradation occurs during execution of the first job by the first computing node based on a matching degree between the first historical performance data and the first performance data includes: Calculating the matching degree between the first performance data and the first historical performance data; When the matching degree satisfies a preset matching degree condition, it is determined that there is performance degradation in the process of the first computing node executing the first job, wherein the matching degree represents the data difference between the first performance data and the first historical performance data, and the matching degree satisfies the preset matching degree condition including: the absolute value of the matching degree is greater than the first preset matching degree or the absolute value of the matching degree is less than at least one of the second preset matching degree.
7. The method according to claim 6, characterized in that Also includes: In the case where performance degradation occurs during the execution of the first job by the first computing node, a detection result of the performance degradation is generated based on the first job type of the first job, wherein the detection result includes at least one of a hardware failure and a network failure.
8. The method according to claim 1, characterized in that The historical job performance data is stored in a public storage device of the computer system, wherein all computing nodes included in the computer system can access the historical job performance data.
9. The method according to claim 8, characterized in that Also includes: When it is determined that there is no performance degradation in the process of the first computing node executing the first job, the first performance data is used to update the first historical performance data stored in the public storage device.
10. The method according to claim 9, characterized in that The public storage device stores the historical New Year data in the form of a database or a data file.
11. The method according to claim 1, characterized in that: Also includes: A first computing node in the computer system executes a second job; Collecting second performance data of the first computing node executing the second job; When the second historical performance data that satisfies the first data condition with the second performance data is not obtained from the historical operation performance data, the second performance data is stored as the second historical performance data.
12. An electronic device, characterized in that: include: a memory for storing instructions to be executed by one or more processors of the electronic device, and The processor is one of the processors of the electronic device, and is used to execute the performance detection method described in any one of claims 1-11.
13. A computer program product, characterized in that include: A non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium contains a computer program code for executing the performance detection method described in any one of claims 1 to 11.
Citation Information
Patent Citations
Hadoop task scheduling method and device
CN110618865A
Operation fault prediction method and device and electronic equipment
CN112395167A
Large-scale cluster job anomaly detection method
CN114116392A
Aperiodic data driven cognitive control system
US11200989B1
