Abnormality detection method and system for high-performance computing system

Abstract

The invention discloses an anomaly detection method and system oriented to a high-performance computing system. The method comprises the steps that data fragments obtained after segmentation of monitoring indexes corresponding to computing nodes in the computing system are obtained; extracting feature vectors of the data fragments, and clustering the feature vectors to obtain clusters; training a shared model by utilizing a part of data fragments close to the mass center of the cluster in the data fragments to obtain a trained shared model; and performing mode matching on the extracted real-time monitoring index features and the clustering clusters, and detecting abnormal data in the target node by using the trained sharing model according to a mode matching result to obtain an abnormal data detection result. According to the method, the abnormal data of the nodes in the high-performance computing system can be detected in time, and the accuracy of abnormal detection is ensured.

Description

Technical Field

[0001] The present invention relates to the field of anomaly detection technology, and in particular to an anomaly detection method and system for high performance computing systems. Background Art

[0002] Modern high-performance computing systems play a vital role in various data-intensive applications in social and scientific fields. These systems are usually composed of a large number of computing nodes connected to each other through high-bandwidth, low-latency networks to form a cluster or supercomputer. Each node is equipped with multiple processor cores and significant memory capacity. However, highly dynamic job scheduling and tightly coupled system interactions make anomalies of computing nodes more common, sometimes leading to system-wide crashes. Therefore, anomaly detection is crucial in system management to identify abnormally behaving nodes and improve system stability.

[0003] The various anomaly detection methods in high-performance computing systems can be roughly divided into three groups: supervised methods, semi-supervised methods, and unsupervised methods.

[0004] Supervised methods: TPDS18 and ALBADross use feature extraction and feature selection techniques to process monitoring indicators collected from high-performance computing systems. Then, a machine learning classifier is trained to detect different performance anomalies. These methods require the use of labeled data for training containing normal and abnormal samples. However, this requires extensive domain knowledge and expert annotation costs, which is often time-consuming and expensive. Therefore, it is challenging to apply these methods to large-scale high-performance computing systems.

[0005] Semi-supervised methods: Proctor and ExaMon use autoencoders for anomaly detection. Proctor learns hidden layer features and then uses a supervised classifier to detect anomalies. ExaMon combines the results of reconstruction error and classification probability. Compared with supervised methods, these methods only require relatively small amounts of labeled data and large amounts of unlabeled data for training. However, this limitation limits the ability to model abnormal samples.

[0006] Unsupervised methods: ISC'20 uses Bayesian Gaussian mixture model and Mahalanobis distance to fit Gaussian distribution. RUAD uses long short-term memory network to capture time dependency. Prodigy is an anomaly detection framework based on variational autoencoder. However, these methods require long-term data to train the model and are not suitable for timely anomaly detection. This is not effective in high-performance computing systems because computing nodes frequently and dynamically switch jobs, resulting in a lack of sufficient corresponding data. In addition, they need to train specific deep models for each computing node instead of adopting a general setting, resulting in additional storage and scheduling requirements.

[0007] Considering the complexity and importance of anomaly detection in high-performance computing systems, computing nodes exhibit significant pattern differences between job execution and idle waiting. During job execution, computing nodes have segments with different patterns, as well as segments with similar patterns. This phenomenon is often observed in other nodes or even across nodes. Automatic anomaly detection in high-performance computing systems has been an important topic of interest for many years. Existing methods cannot distinguish the different patterns exhibited by different nodes in different states. In addition, some methods cannot accurately adapt to the specific patterns of each node by generalizing the model trained on all computing nodes. They usually extract the same features from different segments. This ignores the temporal information of the monitoring indicators and cannot achieve real-time detection. In addition, it becomes impractical to train a model for each node. This is because modern high-performance computing systems are composed of a large number of computing nodes. These observations and limitations suggest that it is possible to reduce overhead by clustering similar and repeatable patterns and training shared models. Summary of the invention

[0008] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0009] To this end, the present invention proposes an anomaly detection method for a high-performance computing system, which can quickly and accurately detect whether a computing node of the high-performance computing system has an anomaly.

[0010] Another object of the present invention is to provide an anomaly detection system for high performance computing systems.

[0011] A third object of the present invention is to provide a computer device.

[0012] A fourth object of the present invention is to provide a non-transitory computer-readable storage medium.

[0013] To achieve the above object, the present invention provides, on one hand, an anomaly detection method for a high performance computing system, comprising:

[0014] Obtain the data segments after the monitoring indicators corresponding to the computing nodes in the computing system are segmented;

[0015] Extracting feature vectors of the data segments, and clustering the feature vectors to obtain clusters;

[0016] Using some data segments close to the centroid of the cluster in the data segments to train the shared model to obtain a trained shared model;

[0017] The extracted features of real-time monitoring indicators are used to perform pattern matching with clustering clusters, and the trained shared model is used to detect abnormal data in the target node according to the pattern matching results to obtain abnormal data detection results.

[0018] The anomaly detection method for a high performance computing system according to an embodiment of the present invention may also have the following additional technical features:

[0019] In one embodiment of the present invention, obtaining data segments after the monitoring indicators corresponding to the computing nodes in the computing system are segmented includes:

[0020] Clean the original monitoring indicators of the computing nodes in the computing system to obtain cleaned data;

[0021] Aggregate the cleaned data to obtain aggregate monitoring indicators;

[0022] Performing data standardization on the aggregate monitoring indicators to obtain standardized data;

[0023] Determine the state transition points of the node and treat the standardized data between each state transition point as an independent data segment.

[0024] In one embodiment of the present invention, the extracted features of the real-time monitoring indicators are used to perform pattern matching with the clustering clusters, including:

[0025] Obtain real-time monitoring indicators corresponding to computing nodes in the computing system;

[0026] Extracting feature data of real-time monitoring indicators within a preset time period after the node state conversion; wherein the feature data includes features in statistical domain, time domain and spectral domain;

[0027] The distance between the feature data and the current cluster centroid is calculated to match the most similar pattern of the cluster to obtain a pattern matching result.

[0028] In one embodiment of the present invention, the monitoring indicators include system-level indicators, application-level indicators, and network-level indicators; wherein the system-level indicators include CPU usage, memory usage, disk I / O, disk space, network traffic, process status, and system load;

[0029] The application-level metrics include response time, error rate, transaction throughput, and resource usage;

[0030] The network level indicators include packet loss rate, latency, number of connections, and bandwidth utilization.

[0031] In one embodiment of the present invention, using some data segments close to the centroid of the cluster in the data segments to train the shared model includes:

[0032] A deep learning shared model is constructed using a shared hybrid expert network layer; wherein the hybrid expert network layer includes a plurality of feedforward network layers and gating units represented as expert networks;

[0033] Part of the data fragments close to the centroid of the cluster are input into the deep learning shared model as training data, and the weight assigned to each expert network to process the incoming token is determined by the gating unit using the activation function, so as to assign the data to the corresponding expert network for processing according to the processing weight of the token to train the deep learning shared model.

[0034] In one embodiment of the present invention, the method further includes:

[0035] The real-time monitoring indicators of the computing nodes are used as input data into the trained shared model to generate reconstructed data;

[0036] The reconstruction error between the input data and the reconstructed data represents the probability that the input real-time monitoring indicator is close to normal behavior, that is, the anomaly score; the anomaly score is calculated using the weighted mean square error:

[0037]

[0038] Among them, M is the dimension of the monitoring indicator, W i is the weight assigned to the monitoring indicator of the i-th dimension, and the stability of the data is measured by the average absolute change:

[0039]

[0040] Where T is the number of data points, x t is the tth observation value; the average absolute change of each dimension monitoring indicator is calculated according to the training data of each cluster cluster to obtain the assigned weight.

[0041] In one embodiment of the present invention, the method further includes defining a sliding window along the time axis, and when the anomaly score exceeds the upper limit of k-sigma, the data point is judged to be abnormal; wherein k is a predefined parameter that controls the sensitivity of the anomaly score threshold.

[0042] To achieve the above object, the present invention further provides an anomaly detection system for a high performance computing system, comprising:

[0043] The fragment segmentation module is used to obtain the data fragments of the monitoring indicators corresponding to the computing nodes in the computing system.

[0044] A feature clustering module, used to extract feature vectors of the data segments and cluster the feature vectors to obtain clusters;

[0045] A model training module, used for training a shared model using a portion of the data segments close to the centroid of the cluster in the data segments to obtain a trained shared model;

[0046] The anomaly detection module is used to use the extracted features of real-time monitoring indicators to perform pattern matching with cluster clusters, and use the trained shared model to detect abnormal data in the target node according to the pattern matching results to obtain abnormal data detection results.

[0047] The anomaly detection method and system for high-performance computing systems of the embodiments of the present invention can quickly and accurately detect whether there are anomalies in the computing nodes of the high-performance computing system. The method integrates fragment clustering, Transformer and hybrid expert model architecture in an innovative way. This unique combination improves the scalability and efficiency of the framework, while significantly improving its anomaly detection accuracy and generalization ability.

[0048] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes a computer device, comprising: a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the anomaly detection method for high-performance computing systems as described in the first aspect embodiment.

[0049] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a non-temporary computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the anomaly detection method for high-performance computing systems as described in the first aspect embodiment is implemented.

[0050] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0052] Figure 1 is a flow chart of an abnormality detection method for a high performance computing system according to an embodiment of the present invention;

[0053] Figure 2 is a framework diagram of an abnormality detection method for a high performance computing system according to an embodiment of the present invention;

[0054] Figure 3 is a diagram of the overall architecture of model sharing according to an embodiment of the present invention;

[0055] Figure 4is a structural diagram of an abnormality detection system for a high performance computing system according to an embodiment of the present invention;

[0056] Figure 5 is a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0058] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0059] The following describes an abnormality detection method, system, device and storage medium for a high-performance computing system according to an embodiment of the present invention with reference to the accompanying drawings.

[0060] Figure 1 is a flow chart of an abnormality detection method for a high performance computing system according to an embodiment of the present invention. Figure 1 As shown, including:

[0061] S1, obtain the data segments after the monitoring indicators corresponding to the computing nodes in the computing system are segmented;

[0062] S2, extracting feature vectors of the data segments, and clustering the feature vectors to obtain clusters;

[0063] S3, using part of the data segments close to the centroid of the cluster in the data segments to train the shared model to obtain a trained shared model;

[0064] S4, using the extracted features of the real-time monitoring indicators to perform pattern matching with the clustering clusters, and using the trained shared model to detect abnormal data in the target node according to the pattern matching results to obtain abnormal data detection results.

[0065] Figure 2 FIG. 1 is a framework diagram of an abnormality detection method for a high performance computing system according to an embodiment of the present invention. Figure 2 As shown, the framework includes offline model training and online anomaly detection.

[0066] It can be understood that offline model training first aggregates the original monitoring indicators and standardizes them. According to the idle waiting state of the node and the status of job execution, the data segments are cut. Then, the features of these segments in the statistical domain, time domain and spectral domain are extracted to effectively represent different patterns and process data of different lengths, and hierarchical clustering is used for grouping. Finally, the model sharing method is adopted to solve the problem of insufficient training data in a short time. The combination of Transformer and hybrid expert model further improves the detection performance and generalization ability. The present invention uses several segments near the centroid of the cluster to train a shared model for the cluster.

[0067] In the online anomaly detection phase, features are extracted from the preprocessed monitoring indicators to match the patterns of the existing clusters. Then, the corresponding models are dynamically assigned to identify anomalies in the target nodes.

[0068] In the embodiment of the present invention, the model training stage needs to obtain training data, that is, original monitoring indicators.

[0069] It is understandable that monitoring indicators may include system-level indicators, application-level indicators, and network-level indicators; wherein the system-level indicators include CPU utilization, memory utilization, disk I / O, disk space, network traffic, process status, and system load; application-level indicators include response time, error rate, transaction throughput, and resource usage; and network-level indicators include packet loss rate, latency, number of connections, and bandwidth utilization.

[0070] Exemplarily, system-level indicators: CPU usage: CPU load, reflecting the utilization of system processing power. Memory usage: usage of physical memory and virtual memory, including used, free and cached memory. Disk I / O: the speed and frequency of reading and writing disks, reflecting the busyness of the storage device. Disk space: usage of disk partitions, including used space and free space. Network traffic: the amount of network data entering and leaving the system, including upload and download speeds. Process status: the number, type and status of running processes, helping to identify processes with high resource consumption. System load: the average load of the system in the recent period of time, reflecting the system pressure.

[0071] For example, application-level indicators: Response time: the time required for an application to process a request, used to evaluate performance bottlenecks. Error rate: the proportion of errors that occur when an application processes a request, reflecting the robustness of the system. Transaction throughput: the number of transactions or requests processed per unit time, reflecting the processing capacity of the system. Resource usage: the application's use of the CPU, memory, and disk.

[0072] For example, network-level indicators include: Packet loss rate: the proportion of data packets lost during network transmission, reflecting the stability of network connections. Latency: the round-trip time from sending to receiving a data packet, reflecting the response speed of the network. Number of connections: the number of currently open network connections, used to monitor network load. Bandwidth utilization: the ratio of the actual bandwidth used by a network link to the total bandwidth.

[0073] In the embodiment of the present invention, the acquired data needs to be preprocessed in the model training stage.

[0074] Since each computing node in the infrastructure is monitored regularly (once per minute), monitoring indicators are also collected and stored. However, these raw monitoring indicators cannot be directly used for subsequent pattern clustering, model training, and anomaly detection. Therefore, the present invention proposes a standardized data preprocessing flow designed specifically for computing nodes in high-performance computing systems.

[0075] Exemplarily, the original monitoring indicators of the computing nodes in the computing system are cleaned to obtain cleaned data; the cleaned data are aggregated to obtain aggregated monitoring indicators; the aggregated monitoring indicators are standardized to obtain standardized data; the state transition points of the nodes are determined, and the standardized data between each state transition point is used as an independent data segment. Specifically, it includes:

[0076] 1) Data cleaning: The negative impact of outliers and noise must be minimized. The present invention deletes the top 5% of data points (extreme values) that deviate from the mean. Data loss may occur during data collection and transmission. The present invention linearly fills these missing values ​​with nearby observations. In addition, some indicators often exhibit linear characteristics that are difficult to capture through direct modeling and may not provide meaningful insights into their absolute values. In this case, more attention is paid to the magnitude of the change in indicators rather than their absolute values. Therefore, the present invention effectively eliminates the linear trend and non-stationarity of the data by taking the first-order difference.

[0077] 2) Data Aggregation: Each computing node is equipped with a large number of processor cores, and fine-grained monitoring indicators from these cores provide real-time information about their operation. The operating system dynamically allocates these cores. In contrast, node-level aggregated indicators show greater stability and consistency. Due to the large amount of data, the present invention aggregates the monitoring indicators of individual cores according to their semantics to reflect the overall status of the node. Through this method, the present invention reduces the complexity of the data while retaining the key information required for accurate analysis of the computing nodes.

[0078] 3) Data normalization: Different indicators have different units and ranges, which may lead to different contributions in the anomaly detection process. To address this dimensional difference, the present invention applies data normalization by scaling each data point to an approximate range. Specifically, the j-th dimension indicator of the i-th node at timestamp t is composed of the mean μ of the same node and indicator. i,j and standard deviation σ i,j Standardization:

[0079]

[0080] 4) Data segmentation: Since computing nodes exhibit different patterns during job execution and idle waiting, directly analyzing continuous monitoring indicators will introduce noise and obscure key information. Dividing monitoring indicators into segments of equal length may introduce distortions, especially when the segmentation points are not chosen properly, which may destroy the inherent structure. The present invention adopts a state-based segmentation method to segment monitoring indicators. Specifically, the present invention first determines the state transition points (for example, timestamps indicating the start and end of a job). Then, the present invention treats the monitoring indicators between each state transition point as an independent segment, representing the continuous behavior of the node in a specific state.

[0081] In the embodiment of the present invention, in order to solve the challenge of scalability during the model training stage, the present invention clusters the segments with similar and repeatable patterns.

[0082] It is understandable that feature extraction is a key step in obtaining key information during data analysis. The feature extraction process usually involves converting monitoring indicators into a set of descriptive parameters to describe the nature of the signal.

[0083] Specifically, the present invention uses TSFEL to extract 134 meaningful and interpretable features for each indicator. These features cover the statistical domain, time domain and spectrum domain, including but not limited to median, absolute energy and maximum power spectrum.

[0084] In the clustering process, the present invention adopts hierarchical clustering and Euclidean distance for the following reasons: hierarchical clustering is a clustering technique based on a bottom-up dendrogram, which provides flexibility in selecting strategies and provides a clear interpretation of the clustering structure. Hierarchical clustering allows for multi-level clustering results, which is conducive to discovering complex patterns. Hierarchical clustering can handle clusters of different sizes and shapes, unlike centroid-based clustering algorithms (e.g., k-means), which assume circular or spherical clusters. It is worth noting that operators do not need to iterate and try to determine the optimal number of clusters. The present invention uses the silhouette coefficient to measure the clustering performance. The higher the silhouette coefficient, the larger the distance between clusters and the smaller the distance within the cluster, achieving the expected results of high cohesion within the cluster and low coupling between clusters.

[0085] In practice, the present invention relies on two key observations, namely that data segments usually lack periodicity and are of unequal length. Therefore, clustering without feature extraction or using deep learning models to extract clusters of hidden layer distributions is impractical. Existing techniques may result in redundant information, increased computational cost and complexity. In addition, noise and irrelevant features may mislead the clustering results and reduce their quality. Existing techniques usually require a large amount of data to train the model, which conflicts with the need for fast training and matching of data ranges in a short time. In addition, these models have complex structures, high training costs, and lack of transparency, resulting in poor interpretability of clustering results.

[0086] In contrast, the present invention uses TSFEL to extract clustering features, which is lightweight and effective. These features have clear physical meanings, which not only improve the interpretability of clustering results, but also enhance the generalization ability of clustering models to new data. The time complexity of the prior art and the present invention for each node is recorded as O(T c1 )、O(T f1 +T c2 ) and O(T f2 +T c3 ), where T fi represents the feature extraction time, T ci represents the clustering time. Since T f <<T c , T f2 <<T f1 and T c2 ≈T 23 <<T c1 , the present invention significantly reduces the calculation time.

[0087] In the embodiment of the present invention, the model training stage is as follows Figure 3 shown.

[0088] It can be understood that in order to capture macro trends and fine-grained complex data patterns, the present invention uses Transformer to enhance temporal context information without the need for fixed-size window input, thereby improving detection performance. In addition, the present invention is combined with a hybrid expert model to achieve model sharing by assigning expert networks to different data within clusters. Each expert network can focus on processing specific patterns, thereby improving performance and generalization capabilities.

[0089] like Figure 3As shown, the present invention replaces the dense feedforward network layer in the Transformer with a sparse hybrid expert network layer. This layer consists of n feedforward network layers represented as experts and operates independently on the tokens. The function of the gating unit uses a softmax activation function to determine the weight assigned to each expert to process the incoming token. Each token is assigned to the corresponding expert.

[0090] Exemplarily, the mechanism of the gating unit: First, the gating network performs feature mapping on each input token, which is usually achieved through one or more linear layers to convert the embedding vector of the token into a lower-dimensional representation, which is used for subsequent expert selection. Secondly, by applying the activation function, the gating network calculates the weight of each expert network for processing each token based on the result of the feature mapping. The role of the activation function is to convert the result of the feature mapping into a probability distribution, which indicates the "preference" or "ability" of each expert network to process a specific token. Finally, the gating mechanism selects a small number of expert networks to process each token based on the calculated weights. Expert networks with higher weights will get more "voting rights", while expert networks with lower weights may be ignored. This approach ensures efficient use of computing resources, because most expert networks remain "idle" when processing specific tokens, thereby achieving sparsity of computing.

[0091] It is understandable that using a few segments closest to the centroid of a cluster to train a shared model constitutes a form of data augmentation. Therefore, the present invention improves the position encoding in the Transformer to incorporate position information within and between different segments. Based on this, the present invention can naturally train a shared model for each cluster without requiring a large amount of training data and incurring high training costs.

[0092] After completing the model training, the embodiment of the present invention performs data anomaly detection.

[0093] Specifically, after completing the offline model training phase, a shared model is saved for each clustering cluster. During real-time detection, the present invention performs the same preprocessing steps as the offline phase to obtain the real-time monitoring indicators corresponding to the computing nodes in the computing system. Then, the features of the monitoring indicators are extracted within a short time (e.g., 1 hour) after the node state transition, and the distance between these features and the existing clustering cluster centroid is calculated to match the most similar pattern. Finally, the corresponding shared model is used for detection. However, this does not mean that the present invention must wait for a period of time after each state transition to perform detection. The present invention can continue to use the model that appeared in the same state last time for detection.

[0094] Furthermore, the monitoring indicators from each computing node are input into the shared model, and the shared model generates reconstructed data. The reconstruction error between the input data and the reconstructed data represents the probability that the input monitoring indicator is close to normal behavior, also known as the anomaly score. The present invention uses weighted mean square error to calculate the anomaly score:

[0095]

[0096] Among them, M is the dimension of the monitoring indicator, W i is the weight assigned to the monitoring index of the i-th dimension. The present invention uses the mean absolute change (MAC) to measure the stability of the data:

[0097]

[0098] Where T is the number of data points, x t The present invention calculates the average absolute change of each dimension monitoring index according to the training data of each cluster to obtain the weight.

[0099] In the threshold selection, the present invention uses a sliding window method to dynamically set the threshold of the anomaly score. Specifically, the present invention defines a sliding window along the time axis. When the anomaly score exceeds the upper bound of k-sigma, the data point is considered to be abnormal. Among them, k is a predefined parameter that controls the sensitivity of the threshold. This method effectively detects noteworthy anomalies or fragments that exhibit abnormal behavior through a careful examination of statistical characteristics. Therefore, the present invention can adapt to the dynamic conversion of states and capture abnormal behavior in a timely manner. This ensures the accuracy of model anomaly detection and improves adaptability and robustness.

[0100] According to the anomaly detection method for high-performance computing systems in accordance with the embodiments of the present invention, high-performance computing systems have problems such as slow and difficult anomaly detection, and the intelligent operation and maintenance achievements of commercial data centers are directly applied to high-performance computing systems, which face many challenges such as different composition and behavior, large data association, and inapplicable algorithm models. Research on intelligent operation and maintenance technologies for high-performance computing systems is carried out, focusing on breakthroughs in key technologies for fast and efficient anomaly detection, and realizing related intelligent operation and maintenance functions. This can effectively improve the anomaly detection capabilities of large-scale high-performance computing systems, improve the maturity of key technologies, and provide strong support for the high-reliability operation of large-scale high-performance computing systems.

[0101] In order to implement the above embodiment, Figure 4 As shown, this embodiment also provides an abnormality detection system 10 for a high performance computing system, including:

[0102] The segmentation module 100 is used to obtain the segmented data segments of the monitoring indicators corresponding to the computing nodes in the computing system;

[0103] A feature clustering module 200 is used to extract feature vectors of the data segments and cluster the feature vectors to obtain clusters;

[0104] A model training module 300 is used to train a shared model using a portion of the data segments close to the centroid of the cluster in the data segments to obtain a trained shared model;

[0105] The anomaly detection module 400 is used to perform pattern matching with the clustering clusters using the extracted features of the real-time monitoring indicators, and to detect abnormal data in the target node using the trained shared model according to the pattern matching results to obtain abnormal data detection results.

[0106] According to the anomaly detection system for high-performance computing systems according to the embodiment of the present invention, it is possible to quickly and accurately detect whether a computing node has an anomaly, and also improve the scalability and efficiency of the framework, while significantly improving the anomaly detection accuracy and generalization ability of the model.

[0107] In order to implement the method of the above embodiment, the present invention also provides a computer device, such as Figure 5 As shown, the computer device 600 includes a memory 601 and a processor 602; wherein the processor 602 runs a program corresponding to the executable program code by reading the executable program code stored in the memory 601, so as to implement each step of the method described above.

[0108] In order to implement the above embodiments, the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in the above embodiments is implemented.

[0109] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0110] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. A method for detecting anomalies in a high performance computing system, characterized in that: include: Obtain the data segments after the monitoring indicators corresponding to the computing nodes in the computing system are segmented; Extracting feature vectors of the data segments, and clustering the feature vectors to obtain clusters; Using some data segments close to the centroid of the cluster in the data segments to train the shared model to obtain a trained shared model; The extracted features of real-time monitoring indicators are used to perform pattern matching with clustering clusters, and the trained shared model is used to detect abnormal data in the target node according to the pattern matching results to obtain abnormal data detection results.

2. The method according to claim 1, characterized in that Get the data segments of the monitoring indicators corresponding to the computing nodes in the computing system, including: Clean the original monitoring indicators of the computing nodes in the computing system to obtain cleaned data; Aggregate the cleaned data to obtain aggregate monitoring indicators; Performing data standardization on the aggregate monitoring indicators to obtain standardized data; Determine the state transition points of the node and treat the standardized data between each state transition point as an independent data segment.

3. The method according to claim 2, characterized in that Use the extracted real-time monitoring indicator features to perform pattern matching with clustering clusters, including: Obtain real-time monitoring indicators corresponding to computing nodes in the computing system; Extracting feature data of real-time monitoring indicators within a preset time period after the node state conversion; wherein the feature data includes features in statistical domain, time domain and spectral domain; The distance between the feature data and the current cluster centroid is calculated to match the most similar pattern of the cluster to obtain a pattern matching result.

4. The method according to claim 2, characterized in that: The monitoring indicators include system-level indicators, application-level indicators, and network-level indicators; wherein the system-level indicators include CPU usage, memory usage, disk I / O, disk space, network traffic, process status, and system load; The application-level metrics include response time, error rate, transaction throughput, and resource usage; The network level indicators include packet loss rate, latency, number of connections, and bandwidth utilization.

5. The method according to claim 1, characterized in that Using some of the data segments close to the centroid of the cluster in the data segments to train the shared model includes: A deep learning shared model is constructed using a shared hybrid expert network layer; wherein the hybrid expert network layer includes a plurality of feedforward network layers and gating units represented as expert networks; Part of the data fragments close to the centroid of the cluster are input into the deep learning shared model as training data, and the weight assigned to each expert network to process the incoming token is determined by the gating unit using the activation function, so as to assign the data to the corresponding expert network for processing according to the processing weight of the token to train the deep learning shared model.

6. The method according to claim 5, characterized in that The method further comprises: The real-time monitoring indicators of the computing nodes are used as input data into the trained shared model to generate reconstructed data; The reconstruction error between the input data and the reconstructed data represents the probability that the input real-time monitoring indicator is close to normal behavior, that is, the anomaly score; the weighted mean square error is used to calculate the anomaly score: Among them, M is the dimension of the monitoring indicator, W i is the weight assigned to the monitoring indicator of the i-th dimension, and the stability of the data is measured by the average absolute change: Where T is the number of data points, x t is the tth observation value; the average absolute change of each dimension monitoring indicator is calculated according to the training data of each cluster cluster to obtain the assigned weight.

7. The method according to claim 6, characterized in that The method further includes defining a sliding window along a time axis, and when the anomaly score exceeds an upper bound of k-sigma, the data point is judged to be anomaly; wherein k is a predefined parameter for controlling the threshold sensitivity of the anomaly score.

8. An anomaly detection system for a high performance computing system, characterized in that: include: The fragment segmentation module is used to obtain the data fragments of the monitoring indicators corresponding to the computing nodes in the computing system after segmentation; A feature clustering module, used to extract feature vectors of the data segments and cluster the feature vectors to obtain clusters; A model training module, used for training a shared model using a portion of the data segments close to the centroid of the cluster in the data segments to obtain a trained shared model; The anomaly detection module is used to use the extracted real-time monitoring indicator features and cluster clusters for pattern matching, and use the trained shared model to detect abnormal data in the target node according to the pattern matching results to obtain abnormal data detection results.

9. A computer device, characterized in that: including a processor and a memory; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the anomaly detection method for a high-performance computing system as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it is used to implement the anomaly detection method for high-performance computing systems as described in any one of claims 1 to 7.

Images