Large-scale Cluster Job Anomaly Detection Method

Abstract

A job anomaly detection method for large-scale clusters provided by the present invention relates to an anomaly detection method, including jointly obtaining job performance data from a database; screening the obtained job performance data to obtain the performance data of job I / O; performing secondary screening on the obtained performance data of job I / O, dividing it into several independent I / O stages to obtain I / O stage data; obtaining the total throughput, duration, peak bandwidth, and mode bandwidth of the I / O stage as the characteristic data of the I / O stage; clustering the characteristic data of the I / O stage with the historical characteristic data of the I / O stage to determine whether the characteristic data of the I / O stage is abnormal. By clustering the characteristic data of the I / O stage with the historical characteristic data of the I / O stage, the present invention can timely detect the performance anomalies of I / O, facilitate timely handling and solution according to the determined anomaly types, so as to optimize the I / O performance in real time, improve the utilization rate of system resources, and improve the I / O performance.

Description

Technical Field

[0001] The present invention relates to an anomaly detection method, and in particular to an anomaly detection method for large-scale cluster jobs. Background Art

[0002] With the rapid improvement of the computing power of large-scale clusters, for storage devices, neither the speed nor the capacity can keep up with this pace. At the same time, the I / O subsystem in large-scale clusters has a long access path and a high degree of competition for applications, resulting in difficulty in improving the overall resource utilization rate and poor application experience. Therefore, as the parallelism of large-scale applications continues to increase, I / O performance often replaces computing power as the performance bottleneck. Summary of the Invention

[0003] In view of the above technical problems, a large-scale cluster job anomaly detection method provided by the present invention can optimize I / O performance, improve system resource utilization rate, and improve I / O performance.

[0004] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] A large-scale cluster job anomaly detection method provided by the present invention includes

[0006] Jointly obtaining job performance data from a data repository;

[0007] Screening the obtained job performance data to obtain the performance data of job I / O;

[0008] Performing secondary screening on the obtained job I / O performance data, dividing it into several independent I / O stages, and obtaining I / O stage data;

[0009] Obtaining the total throughput, duration, bandwidth peak, and bandwidth mode of the I / O stage as the characteristic data of the I / O stage;

[0010] Clustering the characteristic data of the I / O stage with the historical I / O stage characteristic data to determine whether the characteristic data of the I / O stage is abnormal;

[0011] Among them, the job performance data includes data obtained from a job database and the performance data of computing nodes corresponding to the time from system-level data; the data obtained from the job database mainly includes the start and end times of the job and the computing nodes used; the performance data of computing nodes corresponding to the time obtained from system-level data mainly includes read / write bandwidth, read / write IOPS, read / write request distribution, and accessed files; the performance data includes the size of read / write bandwidth per second, the number of read / write operations per second, the number of files read / written per second, the number of processes performing I / O per second, the main accessed file list, the read / write request size distribution, the total I / O time, and the overall read / write continuity.

[0012] The job anomaly detection method for large-scale clusters provided by the present invention, preferably, after the step of "obtaining the total throughput, duration, bandwidth peak value, and bandwidth mode of the I / O phase as the characteristic data of the I / O phase", further includes adding the characteristic data of the I / O phase as new characteristic data to the historical I / O phase characteristic data.

[0013] The job anomaly detection method for large-scale clusters provided by the present invention, preferably, the "secondary screening of the obtained performance data of job I / O" includes: selecting a wavelet basis function, equally spaced sampling the bandwidth waveform signal of job I / O to obtain a sample point sequence corresponding to the signal; performing N-level discrete wavelet transform based on the sample point sequence to obtain wavelet expansion coefficients at N different scales; performing thresholding processing on the wavelet expansion coefficients by selecting corresponding thresholds and threshold rules to obtain the wavelet expansion coefficients at each level after thresholding processing; performing N-level reconstruction according to the wavelet expansion coefficients after thresholding processing and the unprocessed wavelet expansion coefficients to obtain a denoised signal; separating different I / O phases through the denoised signal.

[0014] The job anomaly detection method for large-scale clusters provided by the present invention, preferably, the step of "clustering the characteristic data of the I / O phase and the historical I / O phase characteristic data to determine whether the characteristic data of the I / O phase is abnormal" is specifically: using the DBSCAN algorithm to cluster the characteristic data of the I / O phase and the historical I / O phase characteristic data; if the characteristic data of a certain I / O phase cannot be classified into any category, it is determined that the characteristic data of the I / O phase is abnormal, and the I / O phase is marked as abnormal.

[0015] The job anomaly detection method for large-scale clusters provided by the present invention, preferably, after the step of "clustering the characteristic data of the I / O phase and the historical I / O phase characteristic data to determine whether the characteristic data of the I / O phase is abnormal", further includes: if there is an I / O phase marked as abnormal, summarizing the data related to the anomaly, and determining the type of anomaly for the summarized data.

[0016] The job anomaly detection method for large-scale clusters provided by the present invention, preferably, the data related to the anomaly includes the I / O mode; the I / O mode includes four modes: 1-1, N-1, N-M, and N-N;

[0017] When the number of processes is 1, the I / O mode is 1-1;

[0018] When r ≥ 0.5, the I / O mode is N-1;

[0019] When 0.2 ≤ r < 0.5, the I / O mode is N-M;

[0020] When r < 0.2, the I / O mode is N-N;

[0021] where r is the value obtained by dividing the number of processes by the number of files.

[0022] The job anomaly detection method for a large-scale cluster provided by the present invention. Preferably, the anomaly types include application I / O inefficiency anomaly types, inefficiency anomaly types caused by other application interferences, and inefficiency anomaly types caused by system service components; the application I / O inefficiency types mainly include inefficiency types caused by I / O mode, I / O request size, metadata overhead, and discontinuous access;

[0023] Among them, when the I / O mode is N-1 and 1-1, inefficiency anomaly types will be caused; dense small I / O requests will cause inefficiency anomaly types; excessive metadata overhead will cause inefficiency anomaly types.

[0024] The above technical solution has the following advantages or beneficial effects:

[0025] A job anomaly detection method for a large-scale cluster provided by the present invention includes jointly obtaining job performance data from a database; screening the obtained job performance data to obtain job I / O performance data; performing secondary screening on the obtained job I / O performance data to divide it into several independent I / O stages to obtain I / O stage data; obtaining the total throughput, duration, peak bandwidth, and mode bandwidth of the I / O stage as the characteristic data of the I / O stage; clustering the characteristic data of the I / O stage with the historical I / O stage characteristic data to determine whether the characteristic data of the I / O stage is abnormal. By clustering the characteristic data of the I / O stage with the historical I / O stage characteristic data, the present invention can timely detect the performance anomalies of I / O, facilitate timely handling and resolution according to the determined anomaly types, optimize the I / O performance in real time, improve the system resource utilization rate, and improve the I / O performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] By reading the detailed description of the non-restrictive embodiments with reference to the following drawings, the present invention and its features, shapes, and advantages will become more obvious. The same reference numerals indicate the same parts in all the drawings. The drawings are not deliberately drawn to scale, and the focus is on showing the gist of the present invention.

[0027] Figure 1 is a flowchart of a job anomaly detection method for a large-scale cluster provided in Embodiment 1 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] The present invention will be further described below with reference to the drawings and specific embodiments, but it is not limited to the present invention.

[0029] Embodiment 1:

[0030] As Figure 1 shown, a method for detecting job anomalies in a large-scale cluster provided by Embodiment 1 of the present invention includes

[0031] S101: Jointly obtain job performance data from the database;

[0032] S102: Screen the obtained job performance data to obtain the performance data of job I / O;

[0033] S103: Perform secondary screening on the obtained job I / O performance data, divide it into several independent I / O stages, and obtain I / O stage data;

[0034] S104: Obtain the total throughput, duration, peak bandwidth, and bandwidth mode of the I / O stage as the characteristic data of this I / O stage;

[0035] S105: Cluster the characteristic data of the I / O stage with the historical I / O stage characteristic data to determine whether the characteristic data of the I / O stage is abnormal;

[0036] Among them, the job performance data includes the data obtained from the job database and the performance data of the computing nodes within the corresponding time obtained from the system-level data; the data obtained from the job database mainly includes the start and end times of the job and the computing nodes used; the performance data of the computing nodes obtained from the system-level data within the corresponding time mainly includes read and write bandwidth, read and write IOPS, read and write request distribution, and accessed files; the performance data includes the size of read and write bandwidth per second, the number of read and write operations per second, the number of files read and written per second, the number of processes performing I / O per second, the list of main accessed files, the read and write request size distribution, the total I / O time, and the overall read and write continuity.

[0037] The total throughput is the sum of the read or written data volume of this I / O stage, the duration is the total time of the I / O stage (in seconds), and the peak bandwidth is the bandwidth corresponding to the second with the highest read and write bandwidth value in the I / O stage. Since the bandwidth at the start and end moments of an I / O stage may be very low, and usually the proportion of minimum and maximum values is very low, in order to avoid the excessive influence of extreme values on the mean, we select the mode instead of the mean as the last characteristic. We evenly divide the bandwidth interval of [0, peak bandwidth] into 10 sub-intervals, calculate the number of bandwidths per second in each sub-interval, select the sub-interval with the highest number as the calculation interval for the mode, and take the mean of the bandwidths within the range to obtain the bandwidth mode. In this way, four-dimensional data of [total throughput, duration, peak bandwidth, bandwidth mode] is obtained as the characteristics of an I / O stage, as shown in the following table:

[0038] I / O phase 1 Volume Time Peak Mode I / O phase 2 Volume Time Peak Mode … … … … … I / O phase n Volume Time Peak Mode

[0039] For a job, there are usually multiple I / O phases, which are manifested in the read / write bandwidth waveform diagram and consist of multiple waves. The shapes of the waves are disorderly, and sometimes it is very difficult to separate different waves. During actual operation, a continuous read / write operation of a job, that is, an I / O burst, consists of several small waves. In general waveform classification, this I / O operation segment will be sliced into multiple small waves, which destroys the meaning of an I / O burst and causes great interference to the accurate analysis of each I / O phase. Therefore, in this embodiment, "performing secondary screening on the obtained performance data of job I / O" includes: selecting a wavelet basis function, performing equally spaced sampling on the bandwidth waveform signal of job I / O to obtain a sample point sequence corresponding to the signal; performing N-level discrete wavelet transform based on the sample point sequence to obtain wavelet expansion coefficients at N different scales; performing thresholding processing on the wavelet expansion coefficients by selecting corresponding thresholds and threshold rules to obtain the wavelet expansion coefficients at each level after thresholding processing; performing N-level reconstruction according to the wavelet expansion coefficients after thresholding processing and the unprocessed wavelet expansion coefficients to obtain a denoised signal; through the denoised signal, smoothing the waveform diagram so as to separate different I / O phases.

[0040] After the step of "obtaining the total throughput, duration, bandwidth peak value, and bandwidth mode of the I / O phase as the characteristic data of the I / O phase", it further includes adding the characteristic data of the I / O phase as new characteristic data to the historical I / O phase characteristic data, and classifying jobs with the same application name and the same number of node cores used into one category. Only when the same application runs under the same scale can it possibly have similar I / O performance manifestations, thereby providing a basis for subsequent judgment on whether the new I / O characteristic data is abnormal.

[0041] Generally, if a job has an abnormality, then its bandwidth diagram will be different from other I / O phases. Therefore, it is hoped to classify similar I / O phases into one category to identify abnormal I / O phases. Generally speaking, when the amount of data read and written in the I / O phase is the same, when a performance abnormality occurs, the duration will be longer than normal performance, while the peak bandwidth and average bandwidth will be lower than normal performance. Therefore, [total throughput, duration, bandwidth peak value, bandwidth mode] of the I / O phase are selected as the feature vector for I / O phase clustering. Therefore, the step in this embodiment of "clustering the characteristic data of the I / O phase with the historical I / O phase characteristic data and judging whether the characteristic data of the I / O phase is abnormal" is specifically: using the DBSCAN algorithm to cluster the characteristic data of the I / O phase with the historical I / O phase characteristic data; if the characteristic data of a certain I / O phase cannot be classified into any category, it is judged that the characteristic data of this I / O phase is abnormal, and this I / O phase is marked as abnormal.

[0042] The specific method of clustering the feature data of the I / O phase and the historical I / O phase feature data using the DBSCAN algorithm is as follows: First, obtain all the I / O phases of a job (phase[0], phase[1], …, phase[n]), extract the features of each I / O phase, and then match the job scale, that is, query the I / O phases of historical jobs with the same application name and the number of node cores. After that, cluster each I / O phase phase[i] (0 ≤ i ≤ n) with the I / O phases of historical jobs in turn. If phase[i] cannot be classified into any category, it is considered that there may be a performance anomaly in this I / O phase, and it is marked as an abnormal I / O phase.

[0043] After an I / O phase is marked as abnormal, it is necessary to locate and handle the problem in a timely manner. Therefore, after the step of "clustering the feature data of the I / O phase and the historical I / O phase feature data to determine whether the feature data of the I / O phase is abnormal", it also includes: If there is an I / O phase marked as abnormal, summarize the data related to the anomaly, and determine the type of anomaly for the summarized data.

[0044] Among them, the data related to the anomaly includes the I / O mode; the I / O mode includes four modes: 1-1, N-1, N-M, and N-N;

[0045] When the number of processes is 1, the I / O mode is 1-1;

[0046] When r ≥ 0.5, the I / O mode is N-1;

[0047] When 0.2 ≤ r < 0.5, the I / O mode is N-M;

[0048] When r < 0.2, the I / O mode is N-N;

[0049] Among them, r is the value of the number of processes divided by the number of files.

[0050] The types of anomalies include application I / O inefficiency anomaly type, inefficiency anomaly type caused by interference from other applications, and inefficiency anomaly type caused by system service components. Specifically as follows:

[0051] ① Application I / O inefficiency exception types, that is, due to the application's own reasons, the I / O performance is reduced, mainly including inefficiency types caused by I / O mode, I / O request size, metadata overhead, and discontinuous access. When the I / O mode is N-1 and 1-1, the Lustre parallel file system cannot be fully utilized, resulting in reduced bandwidth. If more than 30% of the total number of I / O requests are less than 4KB, we consider the I / O requests to be small at this time, and intensive small I / O requests will slow down the application's I / O performance. Similarly, metadata operation-intensive will also affect I / O performance. Since metadata requests have a higher priority than ordinary file I / O requests, if the metadata operations are higher than a certain threshold, it is very likely that the application's I / O performance will be lower. In addition, by counting the continuity of application reads and writes, if there are discontinuous reads or discontinuous writes, it is also an important factor in I / O inefficiency.

[0052] ② Inefficiency exception types caused by application interference, that is, due to other applications sharing I / O node resources, resulting in performance interference. We query jobs that use the same proxy node and storage node. If there are cases of their own I / O inefficiency in these jobs, it is very likely to interfere with other jobs. For example, excessive metadata requests occupy the service capacity of the Lustre metadata server, causing the I / O performance of jobs using the same node to decrease. Therefore, jobs sharing nodes may also interfere with the application's I / O performance.

[0053] (3) Inefficiency exception types caused by system service components, that is, due to problems with system service components, the service capacity is reduced, resulting in a decrease in application performance. We found that although some nodes are still running and functioning normally, there are cases of performance degradation, that is, the nodes can provide services normally, but the performance cannot reach the peak. By querying the status of the nodes used by the jobs, if there are nodes with performance degradation, then the low efficiency of the system service components is called the I / O performance bottleneck of the application.

[0054] Dispose and solve in a timely manner according to the determined exception types to optimize the I / O performance in real time, improve the utilization rate of system resources, and improve the I / O performance.

[0055] Those skilled in the art should understand that those skilled in the art can implement the above-mentioned variation examples in combination with the prior art and the above embodiments, which will not be elaborated here. Such variation examples do not affect the essence of the present invention and will not be elaborated here.

[0056] The preferred embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and the devices and structures not described in detail should be understood to be implemented in a common manner in the art; any person skilled in the art can make many possible changes and modifications without departing from the technical solution of the present invention, or modify it into an equivalent embodiment with equivalent changes, which does not affect the essence of the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for detecting job anomalies in a large-scale cluster, characterized in that, including jointly obtaining job performance data from a database; screening the obtained job performance data to obtain the performance data of job I / O; performing secondary screening on the obtained performance data of job I / O, dividing it into several independent I / O stages, and obtaining I / O stage data; obtaining the total throughput, duration, peak bandwidth, and modal bandwidth of the I / O stage as the characteristic data of the I / O stage; clustering the characteristic data of the I / O stage with the historical characteristic data of the I / O stage to determine whether the characteristic data of the I / O stage is abnormal; wherein the job performance data includes the data obtained from the job database and the performance data of the computing nodes within the corresponding time obtained from the system-level data; the data obtained from the job database includes the start and end times of the job and the computing nodes used; the performance data of the computing nodes obtained from the system-level data within the corresponding time includes read / write bandwidth, read / write IOPS, read / write request distribution, and accessed files; the performance data includes the size of read / write bandwidth per second, the number of read / write operations per second, the number of files read / written per second, the number of processes performing I / O per second, the list of main accessed files, the read / write request size distribution, the total I / O time, and the overall read / write continuity; The "performing secondary screening on the obtained performance data of job I / O" includes: selecting a wavelet basis function, performing equally spaced sampling on the bandwidth waveform signal of job I / O to obtain a sample point sequence corresponding to the signal; performing N-level discrete wavelet transform based on the sample point sequence to obtain wavelet expansion coefficients at N different scales; performing thresholding processing on the wavelet expansion coefficients by selecting corresponding thresholds and threshold rules to obtain the wavelet expansion coefficients at each level after thresholding; performing N-level reconstruction based on the wavelet expansion coefficients after thresholding and the unprocessed wavelet expansion coefficients to obtain a denoised signal; separating different I / O stages through the denoised signal.

2. The job anomaly detection method for large-scale clusters according to claim 1, characterized in that, After the step of "obtaining the total throughput, duration, peak bandwidth, and modal bandwidth of the I / O stage as the characteristic data of the I / O stage", it further includes adding the characteristic data of the I / O stage as new characteristic data to the historical characteristic data of the I / O stage.

3. The job anomaly detection method for a large-scale cluster according to claim 1, characterized in that, The step of "clustering the characteristic data of the I / O stage with the historical characteristic data of the I / O stage to determine whether the characteristic data of the I / O stage is abnormal" is specifically: using the DBSCAN algorithm to cluster the characteristic data of the I / O stage with the historical characteristic data of the I / O stage; if the characteristic data of a certain I / O stage cannot be classified into any category, it is determined that the characteristic data of the I / O stage is abnormal, and the I / O stage is marked as abnormal.

4. The job anomaly detection method for a large-scale cluster according to claim 3, wherein, After the step of "clustering the characteristic data of the I / O stage with the historical characteristic data of the I / O stage to determine whether the characteristic data of the I / O stage is abnormal", it further includes: if an I / O stage is marked as abnormal, summarizing the data related to the abnormality and determining the type of abnormality for the summarized data.

5. The job anomaly detection method for a large-scale cluster according to claim 4, characterized in that, The data related to the abnormality includes the I / O mode; the I / O mode includes four modes: 1-1, N-1, N-M, and N-N; when the number of processes is 1, the I / O mode is 1-1; When r ≥ 0.5, the I / O mode is N-1; When 0.2 ≤ r < 0.5, the I / O mode is N-M; When r < 0.2, the I / O mode is N-N; Wherein, r is the value obtained by dividing the number of processes by the number of files.

Images