Supercomputing system node anomaly detection method

Abstract

The present invention discloses a method for detecting anomalies in supercomputing system nodes, comprising: first determining the indicator sequence and log data in each time window, and then using a pre-trained lightweight text encoding model all‑MiniLM‑L6‑v2 to perform semantic vector encoding on the log data in each time window to obtain the log encoding vector corresponding to each time window; by aggregating and encoding multiple logs, the problem of log data accumulation in the same time window is effectively solved. At the same time, by generating a unified semantic placeholder vector for the window with missing logs, the problem of semantic missing caused by sparse log data is solved. In addition, the all‑MiniLM‑L6‑v2 model itself has a relatively small number of parameters, which can improve the real-time performance and processing efficiency of supercomputing system anomaly detection. On this basis, precise semantic alignment of the indicator sequence and log data is achieved, thereby achieving efficient and accurate anomaly detection effect.

Description

Technical Field

[0001] The present invention relates to the field of supercomputer system node anomaly detection, and in particular to a supercomputer system node anomaly detection method. Background Art

[0002] With the rapid development of high-performance computing technology, the types of data generated during system operation are becoming increasingly diverse, including metrics data (such as CPU usage and memory usage) and raw log data (such as system logs and application logs). These different types of data differ in structure and semantics. To more effectively detect abnormal behavior in systems, researchers have begun exploring alignment processes (including time alignment, data structure alignment, and semantic alignment) to combine these different types of data, thereby more comprehensively capturing the system's operating status and improving the accuracy of system anomaly detection.

[0003] However, in practical applications, how to effectively align different types of data is a difficult problem, because in the alignment process, multiple factors usually need to be considered, such as the data's timestamp, format, semantic relationship, etc.

[0004] In view of this, the present invention is proposed. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a method for detecting anomalies in supercomputing system nodes. Based on the effective alignment of indicator data and log data, it achieves the purpose of efficient and high-accuracy anomaly detection of supercomputing systems by combining indicator data and log data.

[0006] In a first aspect, an embodiment of the present invention provides a method for detecting node anomalies in a supercomputing system, the method comprising:

[0007] Taking the sampling period of the indicator data as the time window, determine the indicator sequence in each time window from the new indicator time series matrix of the target node, and determine the indicator sequence vector in each time window after L2 normalization;

[0008] Traverse the new log data of the target node, filter out the log data whose timestamp falls within the time window, and obtain the log data within each time window, where:

[0009] If there is no log data in a time window, a semantic placeholder is generated based on the indicator sequence in the time window, and the semantic placeholder is used as the log data in the time window. If there are multiple log data in a time window, the multiple log data are spliced ​​into one log data;

[0010] The trained all-MiniLM-L6-v2 model is used to perform text vector encoding on the log data in each time window to obtain the log encoding vector in each time window.

[0011] The log encoding vector in each time window is linearly projected using the trained linear transformation matrix to obtain the projection vector of the log data in each time window. The projection vector of the log data in each time window is then L2-normalized to obtain the semantic vector of the log data in each time window.

[0012] For the same time window, calculate the cosine similarity between the indicator sequence vector and the semantic vector of the log data in the same time window;

[0013] Determine whether the target node is abnormal based on the cosine similarity and the dynamically generated threshold.

[0014] In a second aspect, an embodiment of the present invention provides an electronic device, the electronic device including:

[0015] processor and memory;

[0016] The processor is used to execute the steps of the supercomputing system node anomaly detection method of any embodiment by calling the program or instruction stored in the memory.

[0017] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores a program or instruction, and the program or instruction enables a computer to execute the steps of the supercomputing system node anomaly detection method of any embodiment.

[0018] The embodiments of the present invention have the following technical effects:

[0019] By using the trained all-MiniLM-L6-v2 model to perform text vector encoding on the log data in each time window, the log encoding vector in each time window is obtained. After this processing, the dimension of the log data in each time window is the same. Further, based on the linear projection of the linear transformation moment, the log data is mapped to the same shared semantic space as the indicator data, making the log data and the indicator data comparable, while minimizing the relevant processing parameters, thereby improving processing efficiency, reducing the consumption of computing resources, and improving the accuracy of anomaly detection based on log data and indicator data.

[0020] To address the inconsistent temporal distribution of log data and indicator data (where the amount of indicator data is relatively stable over time, while log data exhibits no data at some timestamps and multiple data at others), this paper adopts a fixed time windowing mechanism based on the sampling period of the indicator data. By using regularly sampled indicator data as an anchor point, it uniformly divides time windows (e.g., every 15 seconds) and precisely maps the log data into these windows by timestamp, constructing structurally aligned and time-synchronized multimodal data pairs. This strategy not only avoids the uneven distribution and empty windows caused by windowing primarily based on log data, but also provides a highly consistent and time-aligned data foundation for subsequent detection and processing.

[0021] This paper balances practical system resource constraints with model representation capabilities in its model design, employing a lightweight dual-channel semantic encoding approach. For log data, the all-MiniLM-L6-v2 pre-trained model is used for fast text encoding, with dimensionality reduction achieved through average pooling and linear projection. For indicator data, linear representation is used after direct normalization. This structure significantly reduces the number of model parameters and computational overhead while maintaining accurate representation of semantic features, making it suitable for complex environments where high-frequency indicator data and sparse log data coexist. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 This is a flow chart of a method for detecting abnormality in a supercomputing system node provided by an embodiment of the present invention;

[0024] Figure 2 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention are described clearly and completely below. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are also within the scope of protection of the present invention.

[0026] Figure 1 This is a flow chart of a method for detecting abnormality in a supercomputing system node provided by an embodiment of the present invention. Figure 1, the supercomputing system node anomaly detection method specifically includes the following steps:

[0027] S1. Taking the sampling period of the indicator data as the time window, determine the indicator sequence in each time window from the new indicator time series matrix of the target node, and determine the indicator sequence vector in each time window after L2 normalization.

[0028] The target node's new indicator time series matrix includes the preprocessed indicator series for the target node during the detection period. Specifically, the matrix rows can contain data for different indicators (e.g., memory usage, CPU usage) at the same timestamp, while the columns contain data for the same indicator at different timestamps. Alternatively, the matrix columns can contain data for different indicators (e.g., memory usage, CPU usage) at the same timestamp, while the rows contain data for the same indicator at different timestamps.

[0029] Assuming that the time period to be tested is from 0:00 on May 15, 2025 to 0:00 on May 15, 2025, and the sampling period of the indicator data is 15 seconds, the time period to be tested is divided into 240 time windows, of which the first time window corresponds to 0:00 on May 15, 2025 to 0:00:15 on May 15, 2025, the second time window corresponds to 0:00:16 on May 15, 2025 to 0:00:30 on May 15, 2025, and so on. In the new indicator time series matrix, the indicator data with timestamps falling between 0:00 on May 15, 2025 and 0:00:15 on May 15, 2025 are divided into the first time window. The time window is denoted as X k ={x1, x2, ... x d}, where X k represents the indicator sequence matrix of the kth time window, x1 represents the data of the first indicator in the kth time window, that is, the indicator sequence, and d represents the number of indicators.

[0030] On this basis, the indicator sequence vectors in each time window after L2 normalization are determined. The indicator sequence vectors in each time window after L2 normalization have the same modulus, providing a basis for subsequent comparison with log data in the shared semantic space.

[0031] Exemplarily, determining the indicator sequence vector within each time window after L2 normalization includes:

[0032] Perform Z-score normalization on the indicator sequence in each time window to obtain the processed indicator sequence in each time window; and perform L2 normalization on the processed indicator sequence in each time window to obtain the indicator sequence vector in each time window.

[0033] The purpose of performing Z-score normalization on the indicator sequences within each time window is to unify the scale of the indicator values, eliminate the influence of dimension, and improve the comparability between different indicators.

[0034] In some embodiments, before determining the indicator sequence within each time window from the new indicator time series matrix of the target node with the sampling period of the indicator data as the time window, it also includes: obtaining the original log data and the indicator time series matrix of the target node within the time period to be detected; preprocessing the indicator time series matrix to reduce the dimension of the indicator time series matrix to obtain a new indicator time series matrix after dimensionality reduction; and processing semantically repeated symbols and semantically possessed symbols in the original log data to obtain processed new log data.

[0035] Among them, the indicator time series matrix is ​​preprocessed to reduce the dimension of the indicator time series matrix, and a new indicator time series matrix after dimensionality reduction is obtained, including:

[0036] For each two indicator columns in the indicator time series matrix, the Pearson correlation coefficient is calculated between each two indicators. If the absolute value of the Pearson correlation coefficient between the two indicators is greater than a first set threshold (for example, the first set threshold is 0.95), it indicates that the two indicators are highly correlated. To achieve dimensionality reduction, only one of the two indicators is retained. That is, one of the two indicator columns corresponding to the two indicators in the indicator time series matrix is ​​deleted to obtain a new indicator time series matrix after dimensionality reduction. In particular, if there is no data for a certain indicator at a timestamp, it is padded with 0 to ensure the integrity of the indicator data.

[0037] Furthermore, one of the two indicators with a higher correlation can be retained by the following strategy: assuming that the two indicators include a first indicator and a second indicator, one of the two indicator columns corresponding to the two indicators in the indicator time series matrix is ​​deleted, including: deleting the indicator column of the indicator with a lower priority among the first indicator and the second indicator;

[0038] Alternatively, delete the indicator columns for indicators with low variance between the first and second indicators. This allows you to filter out information-rich, complementary indicator data, providing a data foundation for anomaly detection based on indicator data. Furthermore, based on expert experience, you can select approximately 50 indicators from the reduced dimensionality for anomaly detection.

[0039] The original log data contains repeated semantic symbols and semantically valid symbols, and the processed new log data is obtained, including:

[0040] For each log entry in the original log data, the log entry is divided into blocks according to spaces, and a preset number of block contents at the front are removed (mainly data such as node names and server names that are not useful for this embodiment of the solution). For the remaining content, the symbol string representing the network address and network port (for example, "xxx.xxx.xxx.xxx@Port") is replaced with a first semantic character (for example, IP PORT), the symbol string representing the memory address (for example, 0xabc123) is replaced with a second semantic character (for example, address), the identifier fragment connected by multiple hyphens (for example, abc-123-def-456) is replaced with a third semantic symbol (for example, ID), and independent preset symbols (for example, common symbols such as ":", ".", "@", and "-") are replaced with spaces, and multiple consecutive spaces are merged into one space to obtain the processed new log data. By replacing characters in the log data that have clear semantic meanings and are difficult to interpret with characters with clear semantics, the quality of the log data can be improved, making the log data easier to interpret in subsequent processing. Log data can be denoised by replacing single preset symbols (such as common symbols such as ":", ".", "@", and "-") with spaces and merging multiple consecutive spaces into a single space.

[0041] S2. Traverse the new log data of the target node, filter out the log data whose timestamps fall within the time window, and obtain the log data within each time window.

[0042] Among them, if there is no log data in a time window, a semantic placeholder is generated according to the indicator sequence in the time window, and the semantic placeholder is used as the log data in the time window. If there are multiple log data in a time window, the multiple log data are spliced ​​into one log data.

[0043] Generating semantic placeholders based on metric sequences within a time window includes dividing all metric sequences into three categories: a first category related to the CPU, a second category related to input / output latency, and a third category related to memory. The specific division method can be based on expert experience or matching set fields. Then, the average value v1 of the first category metric sequence, the average value v2 of the second category metric sequence, and the average value v3 of the third category metric sequence within the time window are calculated. If v1 is greater than the first threshold, the first sub-placeholder is determined to be CPUHIGH; if v1 is less than the negative first threshold, the first sub-placeholder is determined to be CPULOW; otherwise, it is determined to be CPUNORMAL. If v2 is greater than the first threshold, the second sub-placeholder is determined to be IOHIGH; if v2 is less than the negative first threshold, the second sub-placeholder is determined to be IOLOW; otherwise, it is determined to be IONORMAL. If v3 is greater than the first threshold, the third sub-placeholder is determined to be MEMHIGH; if v3 is less than the negative first threshold, the third sub-placeholder is determined to be MEMLOW; otherwise, it is determined to be MEMNORMAL. The time step information corresponding to a time window, the first sub-placeholder, the second sub-placeholder, and the third sub-placeholder are concatenated to form a semantic placeholder. In subsequent processing, this semantic placeholder is input into the MiniLM encoder to generate sentence vectors with different semantic distributions, thus avoiding the problem of difficult convergence of InfoNCE loss caused by the identical sentence vectors in a large number of empty windows. Through this mechanism, even in scenarios with sparse logs, each time window can participate in training and form comparative samples with differences in time distinction and performance status, providing the model with a more stable and learnable semantic structure, significantly improving the training efficiency of contrastive learning under unsupervised conditions and its sensitivity to abnormal behavior.

[0044] In supercomputing systems, indicator data is a regularly sampled continuous time series, typically containing thousands of dimensions of temporal features, and exhibiting nonlinear fluctuations and long-term trends. Log data, on the other hand, is extremely sparse (not every timestamp has a corresponding log entry), and multiple log entries may exist at the same timestamp, containing a large amount of unstructured text. To address these characteristics, the two modalities are independently encoded within each time window. Subsequently, through a comparative learning mechanism, the complementary features between the log and indicator modalities can be effectively mined within the embedding space. Even in complex scenarios, this method can effectively model the semantic differences between the modalities, improving anomaly detection performance.

[0045] Therefore, this embodiment proposes a bimodal data alignment mechanism based on indicators to solve the inconsistency problem between log data and indicator data in sampling frequency and time distribution. Based on the sampling period of regularly sampled indicator data, the time windows with fixed length and continuous non-overlapping (for example, every 15 seconds is a time window) are divided. Then, according to the timestamp of the log data, the log data is projected into the window. For the time window where there is no log data, a semantic placeholder is automatically generated to make the structure complete and form a strictly aligned "log-indicator" modal pair. This mechanism realizes the time alignment and structural alignment between modalities, significantly improving the accuracy and consistency of subsequent comparative learning training. It is particularly suitable for supercomputing cluster operation and maintenance data scenarios with sparse logs and dense indicators.

[0046] S3. Use the trained all-MiniLM-L6-v2 model to perform text vector encoding on the log data in each time window to obtain the log encoding vector in each time window.

[0047] Specifically, for each time window k, the log data set is expressed as: k = { log1, log2, ... log n Where n is the number of log data in the kth window (n can be 0 or more), and log1 represents a log data. The trained all-MiniLM-L6-v2 model is used to encode each log data into a text vector. Each log data is encoded into a 384-dimensional log encoding vector, and the result is: H k ={h1,h2,……h n}.

[0048] By using the trained all-MiniLM-L6-v2 model to encode the text vectors for the log data in each time window, we obtain the log encoding vectors in each time window. After this processing, the dimensions of the log data at each timestamp are the same.

[0049] S4. Linearly project the log encoding vector in each time window using the trained linear transformation matrix to obtain the projection vector of the log data in each time window, and perform L2 normalization on the projection vector of the log data in each time window to obtain the semantic vector of the log data in each time window.

[0050] In order to achieve dimensional alignment between log data and indicator data, it is necessary to further perform linear projection on the log-encoded vector, with the projection dimension being the same as the dimension of the indicator data.

[0051] Specifically, the log coding vector H in each time window is transformed by the trained linear transformation matrix W1 k ={h1,h2,……h n} Perform linear projection to obtain the projection vector of the log data in each time window: , W1 is a learnable linear transformation matrix of size 384×d, and d represents the number of indicators.

[0052] Finally, yes The projection vector inside is L2 normalized to obtain the final semantic vector of the log data.

[0053] At this point, both the indicator data and log data have undergone L2 normalization, have the same modulus length, and have consistent scale standards and comparability, providing a data basis for subsequently determining whether the target node is abnormal by comparing the indicator data and log data.

[0054] By using the lightweight all-MiniLM-L6-v2 model and linear projection based on linear transformation moments, log data is mapped to the same shared semantic space as indicator data, making log data and indicator data comparable while minimizing related processing parameters. This improves processing efficiency, reduces the consumption of computing resources, and enhances the accuracy of anomaly detection based on log data and indicator data.

[0055] This embodiment balances practical system resource constraints with model representation capabilities in its model design, employing a lightweight dual-channel semantic encoding approach. For log data, the all-MiniLM-L6-v2 pre-trained model is used for fast text encoding, with dimensionality reduction achieved through average pooling and linear projection. For indicator data, linear representation is used after direct normalization. This architecture significantly reduces model parameters and computational overhead while maintaining accurate semantic representation, making it suitable for complex environments where high-frequency indicator data and sparse log data coexist.

[0056] S5. For the same time window, calculate the cosine similarity between the indicator sequence vector and the semantic vector of the log data within the same time window.

[0057] S6. Determine whether the target node is abnormal based on the cosine similarity and the dynamically generated threshold.

[0058] Exemplarily, the difference between 1 and the cosine similarity is determined as the anomaly score of the corresponding time window;

[0059] If the anomaly score is greater than the threshold, it is determined that the target node has an anomaly in the corresponding time window;

[0060] Among them, the threshold Dynamically determined as follows:

[0061]

[0062] in, The 95% quantile of the anomaly score for the first N time windows in the detection stream; shape parameter With scale parameter is based on greater than The anomaly score is obtained by fitting the generalized Pareto distribution GPD. represents the confidence level, is the preset value, n means greater than The number of anomaly scores.

[0063] Compared with the use of fixed thresholds, the dynamic threshold mechanism based on the generalized Pareto distribution provided in this embodiment can adapt to changes in data distribution, effectively improve the stability and recall rate of detection, and significantly reduce the false detection rate. It is especially suitable for uncertain environments with concept drift, unlabeled and abnormal outbreaks in supercomputing scenarios.

[0064] Furthermore, the definition of system anomalies is specifically as follows: log data shows that the system has errors, restarts, failures, etc., but the indicator data does not fluctuate significantly; or, the indicator values ​​indicate behaviors such as rapid resource consumption and memory surges, but the log data does not record any abnormal information, or, both the log data and the indicator data show obvious abnormal patterns.

[0065] To illustrate two examples, the following shows the metric and log data for two typical anomaly time windows (the time window is 15 seconds): Time window: 2025-01-01 08:45:30 to 08:45:45. The metric vector (after Z-score normalization): [0.92, 0.85, 0.95, ..., 0.98] indicates a significant increase in CPU utilization, I / O latency, etc. Log data: NOLOG 1734365250 CPUHIGH IONORMAL MEMNORMAL. The model concludes: The metric data is significantly anomaly, but the log data contains no record, indicating that the system resource changes are not covered by the logs, indicating an anomaly.

[0066] Another example: Time window: 2025-01-01 09:00:00 to 09:00:15

[0067] Indicator vector (after Z-score normalization): [-0.12, 0.01, 0.03, ..., -0.05]. The indicator value reflects the stable state.

[0068] Log data: kernel: systemd: Failed to start Lustre monitoring service

[0069] Model judgment: Error keywords (Failed, Error) appear in the log, but the indicators do not change → an exception is also generated.

[0070] This embodiment of the present invention employs a symmetric cross-modal contrastive learning mechanism. During the training phase, pairing relationships are established using the comparison objective of "normal window modalities should be similar." Anomalous modal representations are projected to distant locations in the shared semantic space, reducing the similarity between the two modalities. Without manual labeling, the degree of modal deviation can be detected through cosine similarity scoring. Even if both modalities are anomalous, they are automatically distanced due to "directional drift," resulting in a significant improvement in the anomaly score.

[0071] By constructing a unified semantic space and training a modality pairing similarity objective, this approach can identify cross-modality anomalies in an unsupervised manner while accurately capturing serious system anomalies caused by synchronous drift between log and indicator modalities. This provides a new paradigm for multimodal anomaly detection with high stability and sensitivity. Highly interpretable: Scoring is based on semantic deviations between modality vectors, allowing for tracing back to the root cause of anomalies. Highly real-time: Each window is independently judged, making it suitable for online deployment.

[0072] In some embodiments, a training process of a trained all-MiniLM-L6-v2 model and a trained linear transformation matrix is ​​given.

[0073] Exemplarily, each indicator sequence vector in the first matrix is ​​traversed, and the cosine similarity between an indicator sequence vector in the first matrix and the semantic vector of each log data in the second matrix is ​​calculated by the following formula:

[0074]

[0075] The first matrix is ​​composed of indicator sequence vectors in multiple time windows in a training batch, and the second matrix is ​​composed of semantic vectors of log data in multiple time windows in a training batch.

[0076] The semantic vector of log data is determined as follows:

[0077] The all-MiniLM-L6-v2 model to be trained is used to perform text vector encoding on the log data in each time window to obtain the log encoding vector in each time window. The linear transformation matrix to be learned is used to linearly project the log encoding vector in each time window to obtain the projection vector of the log data in each time window. The projection vector of the log data in each time window is then L2-normalized to obtain the semantic vector of the log data in each time window.

[0078] Represents the indicator sequence vector in the i-th time window and the semantic vector of the log data in the jth time window The cosine similarity between them is given by , cos() represents the function for calculating the cosine similarity, and τ represents the temperature parameter, which is used to adjust the distribution concentration of the cosine similarity.

[0079] so that Maximize the value of The training objective is to minimize the value of , and the parameters of the all-MiniLM-L6-v2 model to be trained and the element values ​​in the linear transformation matrix to be learned are trained and adjusted to obtain the trained all-MiniLM-L6-v2 model and the trained linear transformation matrix. It represents the cosine similarity between the indicator sequence vector and the semantic vector of the log data within the same time window, and is the cosine similarity between positive sample pairs (i.e., the indicator sequence vector and the semantic vector of the log data come from the same time window). It represents the cosine similarity between the indicator sequence vector and the semantic vector of the log data in different time windows, and is the cosine similarity between negative sample pairs (i.e., the indicator sequence vector and the semantic vector of the log data come from different time windows).

[0080] Specifically, for each training batch, the indicator sequence vectors and semantic vectors of the log data within N time windows are extracted. A first matrix of size N×d is constructed from the indicator sequence vectors within the N time windows, where d represents the number of indicators. A second matrix of size N×d is constructed from the semantic vectors of the log data within the N time windows. The cosine similarity between each indicator sequence vector and each semantic vector in the first and second matrices is then calculated to construct a cosine similarity matrix. This cosine similarity matrix is ​​an N×N matrix, where the diagonal elements represent the cosine similarity between pairs of positive samples, and the off-diagonal elements represent the cosine similarity between pairs of negative samples. This construction method enables large-scale automated negative sample mining and fully utilizes the temporal alignment relationship to establish weak supervisory signals.

[0081] By fully interconnecting the samples within a single training batch, eliminating the need for sampling external negative samples, and constructing an N×N similarity matrix, we achieve precise alignment of the two modal data, placing them in the same feature space. Because the training objective is to maximize the values ​​of elements on the diagonal and minimize the values ​​of elements on the off-diagonal, we achieve precise alignment of the spatial geometric structure with the temporal structure. This significantly improves alignment discrimination and convergence speed in high-dimensional, noisy, and label-deficient scenarios like supercomputing.

[0082] This method of fully interconnecting samples within a single training batch has three major advantages: first, the number of negative samples increases exponentially, covering all mismatch situations; second, the gradient information is richer and the direction is more precise, making the model converge faster and more stable; third, in the high-noise, asynchronous logging environment of supercomputers, it can fully learn the discrimination boundaries and significantly enhance the anomaly detection capability.

[0083] This approach is easy to implement, as a single matrix multiplication can generate all the comparison scores for an entire batch. Subsequent bidirectional cross entropy operations only perform softmax / log-sum-exp on the similarity matrix and its transpose, eliminating the need for new dot products. GPU utilization is high, and video memory can be easily utilized.

[0084] To prevent the model from being biased towards a single modality during training (for example, towards indicator data or log data), the system uses a symmetric InfoNCE loss to optimize both the "indicator → log" and "log → indicator" directions simultaneously. This is a symmetric cross-modal contrastive learning mechanism. Specifically, the loss function is defined as:

[0085]

[0086]

[0087]

[0088] Among them, L1 represents the contrast loss between the indicator sequence vector and the semantic vector of the log data based on the indicator sequence vector, and L2 represents the contrast loss between the indicator sequence vector and the semantic vector of the log data based on the semantic vector of the log data; N represents the number of time windows, i and j represent the ordinal numbers of the time windows, exp() represents the exponential function, and log() represents the logarithmic function.

[0089] This loss function encourages positive pairs (main diagonal element values) to achieve high similarity while suppressing all negative pairs, resulting in low similarity. Through symmetric optimization (using bidirectional cross-entropy), both the "metric → log" and "log → metric" directions are optimized simultaneously, each forming an independent gradient path while sharing the same set of model parameters. This achieves a balanced representation and avoids the bias of "only searching in one direction."

[0090] To improve the model's sensitivity to subtle semantic differences between "hard negatives" (pairs of semantically similar negative examples), the system introduces a trainable temperature parameter, τ, to scale all similarity values. This parameter is automatically adjusted during training, enhancing the dynamic range of gradients and improving training stability and discriminative ability.

[0091] In summary, by constructing sample pairs based on time windows and leveraging batch-level similarity modeling, symmetric loss optimization, and temperature scaling strategies, we achieve multimodal semantic spatial alignment without manual labeling, providing an accurate and robust representation foundation for subsequent anomaly detection. Designed for supercomputing scenarios: Using "NO LOG" placeholder vectors and fully connected negative samples, the model learns the semantics of "no log = normal," addressing the issue of highly sparse logs, reducing false positives in empty windows, and catching anomalies. For applications involving tens of thousands of supercomputing nodes, a lightweight model (MiniLM + single linear projection) is selected to minimize model parameters.

[0092] In other words, the contrastive learning module, based on cross-modal contrastive learning, employs batch-level positive and negative sample construction and a symmetric optimization strategy to strengthen the alignment between metrics and log modalities in the semantic space. During training, a similarity matrix is ​​constructed between modalities, minimizing the similarity loss between modal pairs within the same time window and maximizing the difference between negative samples across different time windows. A bidirectional InfoNCE loss is employed, and a temperature parameter is introduced to regulate the centralization of the output distribution, effectively improving the model's discriminative capabilities and training stability. This module achieves semantic alignment learning between modalities without the need for manual labeling.

[0093] During the inference phase of anomaly detection, the trained model parameters (i.e., the trained all-MiniLM-L6-v2 model and the trained linear transformation matrix) are reused. The same encoding process is performed on newly input data. The cosine distance between the log vector and the indicator vector within each time window is then calculated in real time as an anomaly score. The SPOT algorithm is introduced to dynamically update the threshold based on the GPD. A higher score indicates poorer cross-modal consistency and a greater likelihood of an anomaly. If the anomaly score exceeds the threshold, the anomaly is identified. This online anomaly detection and adaptive mechanism can operate stably and long-term in supercomputing environments without manual annotation.

[0094] Furthermore, the embodiment of the present invention also provides an indicator data collection strategy that is adapted to the massive indicator data in supercomputing scenarios, so that massive data can be collected smoothly. Specifically, the original log data and indicator time series matrix of the target node in the time period to be detected are obtained, including:

[0095] A query request in a preset format is sent to the interface of the time series query service. The query request in the preset format includes the node name, the structure to which it belongs, the service address, the target indicator name, and the sampling time interval. When the time series query service receives the query request, it starts a first number of data collection threads. If the first number of data collection threads are all successfully connected, the number of concurrent data collection threads is gradually increased; if there is a connection failure in the first number of data collection threads, the first number is reduced by half; if an error occurs in the data collection thread, the step reduction function is called to reduce the collection step size; if the request times out or the interface is abnormal, an exponential backoff strategy with random jitter is used to perform the query. The line is automatically retried; the time series data returned by the receiving interface, each time series data includes a metadata dictionary, which is used to describe the indicator name and the node to which the indicator belongs. Each time series data also includes a numerical array, which records the indicator data of the corresponding indicator at each timestamp within the sampling time interval; according to the time series data, the complete time series data of each indicator of each node is saved as a file in a preset format; according to the target node, the target file is located from the file in the preset format, and the indicator data is read from the target file to form an indicator time series matrix. The row elements of the indicator time series matrix represent the data of different indicators at the same timestamp, and the column elements represent the data of the same indicator at different timestamps.

[0096] Lightweight dual-channel encoding architecture, balancing efficiency and performance: The embodiment of the present invention takes into account the actual system resource limitations and model representation capabilities in the design of the model structure, and adopts a lightweight dual-channel semantic encoding method: for log data, the all-MiniLM-L6-v2 pre-trained model is used to achieve fast text encoding, and the dimension is reduced through average pooling and linear projection. For indicator data, linear representation is used after direct normalization. This structure greatly reduces the number of model parameters and computational overhead, while maintaining accurate characterization of semantic features. It is suitable for complex operating environments where high-frequency indicator data and sparse log data coexist.

[0097] Unsupervised contrastive learning mechanism: The embodiment of the present invention breaks away from the traditional supervised framework based on reconstruction or classification, designs a two-way contrastive learning mechanism based on the InfoNCE loss function, and constructs a purely contrast-driven unsupervised training path. During the training phase, positive and negative samples are constructed in time windows, and the consistent representation between modalities is learned through bimodal semantic space alignment. Without any manual labels, a discriminant model with the ability to perceive abnormal offsets can be trained. This mechanism has a small number of parameters, fast convergence, and can be updated online. It is particularly suitable for high-performance computing systems that run for a long time and lack annotations.

[0098] Supports high-concurrency indicator collection and distributed deployment, adapting to the actual supercomputing platform operation and maintenance requirements. The embodiment of the present invention also develops a high-concurrency indicator collection module that covers multiple clusters of high-performance computing systems. It can stably collect 3000+ indicators in a single time, supports breakpoint retry, step size adaptation, and thread pool concurrency control, ensuring data integrity and reliability over long periods of time, meeting the engineering and practical requirements of high-performance cluster anomaly monitoring systems.

[0099] Figure 2 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 2 As shown, the electronic device 200 includes one or more processors 201 and a memory 202 .

[0100] The processor 201 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 200 to perform desired functions.

[0101] Memory 202 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 201 may execute the program instructions to implement the supercomputing system node anomaly detection method of any embodiment of the present invention described above and / or other desired functions. The computer-readable storage medium may also store various contents such as initial external parameters and thresholds.

[0102] In one example, the electronic device 200 may further include an input device 203 and an output device 204, which are interconnected via a bus system and / or other connection mechanisms (not shown). The input device 203 may include, for example, a keyboard, a mouse, etc. The output device 204 may output various information to the outside, including warning information, braking force, etc. The output device 204 may include, for example, a display, a speaker, a printer, a communication network, and remote output devices connected thereto.

[0103] Of course, to simplify, Figure 2 Only some of the components related to the present invention in the electronic device 200 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device 200 may further include any other appropriate components according to specific application scenarios.

[0104] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the steps of the supercomputing system node anomaly detection method provided by any embodiment of the present invention.

[0105] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of embodiments of the present invention, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0106] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps of the supercomputing system node anomaly detection method provided by any embodiment of the present invention.

[0107] Computer-readable storage media can be any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0108] It should be noted that the terms used in the present invention are only for describing specific embodiments and are not intended to limit the scope of this application. As shown in the present specification, unless the context clearly indicates an exception, the words "one", "an", "a kind of" and / or "the" do not specifically refer to the singular and may also include the plural. The terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the presence of other identical elements in the process, method or device comprising the elements.

[0109] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the internal parts of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting abnormality of nodes in a supercomputing system, characterized in that: include: Taking the sampling period of the indicator data as the time window, determine the indicator sequence in each time window from the new indicator time series matrix of the target node, and determine the indicator sequence vector in each time window after L2 normalization; Traversing the new log data of the target node, filtering out the log data whose timestamp falls within the time window, and obtaining the log data within each time window, wherein, if no log data exists within a time window, generating a semantic placeholder based on the indicator sequence within the time window, and using the semantic placeholder as the log data within the time window; if multiple log data exist within a time window, splicing the multiple log data into one log data; The trained all-MiniLM-L6-v2 model is used to perform text vector encoding on the log data in each time window to obtain the log encoding vector in each time window. The log encoding vector in each time window is linearly projected using the trained linear transformation matrix to obtain the projection vector of the log data in each time window. The projection vector of the log data in each time window is then L2-normalized to obtain the semantic vector of the log data in each time window. Calculating the cosine similarity between the indicator sequence vector and the semantic vector of the log data within the same time window; Whether an abnormality occurs in the target node is determined according to the cosine similarity and a dynamically generated threshold.

2. The method according to claim 1, characterized in that The determining whether the target node is abnormal according to the cosine similarity and a dynamically generated threshold value includes: Determine the difference between 1 and the cosine similarity as the anomaly score of the corresponding time window; If the anomaly score is greater than the threshold, it is determined that an anomaly occurs in the target node in the corresponding time window; Among them, the threshold Dynamically determined as follows: in, The 95% quantile of the anomaly score for the first N time windows in the detection stream; shape parameter With scale parameter is based on greater than The anomaly score is obtained by fitting the generalized Pareto distribution GPD. represents the confidence level, is the preset value, n means greater than The number of anomaly scores.

3. The method according to claim 1, characterized in that Before determining the indicator sequence within each time window from the new indicator time series matrix of the target node using the sampling period of the indicator data as the time window, the method further includes: Obtaining the original log data and indicator time series matrix of the target node within the time period to be detected; The indicator time series matrix is ​​preprocessed to reduce the dimension of the indicator time series matrix to obtain a new indicator time series matrix after dimensionality reduction; and semantically repeated symbols and semantically possessed symbols in the original log data are processed to obtain processed new log data.

4. The method according to claim 3, characterized in that The preprocessing of the indicator time series matrix to reduce the dimension of the indicator time series matrix to obtain a new indicator time series matrix after dimensionality reduction includes: The Pearson correlation coefficient between each two indicators in the indicator time series matrix is ​​calculated. If the absolute value of the Pearson correlation coefficient between the two indicators is greater than a first set threshold, one of the two indicator columns corresponding to the two indicators in the indicator time series matrix is ​​deleted to obtain a new indicator time series matrix after dimensionality reduction.

5. The method according to claim 4, characterized in that The two indicators include a first indicator and a second indicator, and deleting one of the two indicator columns corresponding to the two indicators in the indicator time series matrix includes: Deleting the indicator column of the indicator whose priority meets the first set condition among the first indicator and the second indicator; Alternatively, the indicator column of the indicator whose variance between the first indicator and the second indicator meets the second set condition is deleted.

6. The method according to claim 3, characterized in that The processing of semantically repeated symbols and semantically correct symbols in the original log data to obtain processed new log data includes: For each log entry in the original log data, the log entry is divided into blocks according to spaces, and a preset number of block contents at the front are removed. For the remaining content, the symbol string representing the network address and the network port is replaced by a first semantic character, the symbol string representing the memory address is replaced by a second semantic character, the identifier fragment connected by multiple short dashes is replaced by a third semantic symbol, the single preset symbol is replaced by a space, and multiple consecutive spaces are merged into one space to obtain processed new log data.

7. The method according to claim 1, characterized in that The determining of the indicator sequence vectors in each time window after L2 normalization includes: Perform Z-score normalization on the indicator sequence in each time window to obtain the processed indicator sequence in each time window; and perform L2 normalization on the processed indicator sequence in each time window to obtain the indicator sequence vector in each time window.

8. The method according to claim 1, characterized in that The trained all-MiniLM-L6-v2 model and the trained linear transformation matrix are obtained as follows: Traverse each indicator sequence vector in the first matrix and calculate the cosine similarity between an indicator sequence vector in the first matrix and the semantic vector of each log data in the second matrix using the following formula: The first matrix is ​​composed of indicator sequence vectors in multiple time windows in a training batch, and the second matrix is ​​composed of semantic vectors of log data in multiple time windows in a training batch; The semantic vector of the log data is determined in the following manner: The all-MiniLM-L6-v2 model to be trained is used to perform text vector encoding on the log data in each time window to obtain the log encoding vector in each time window. The linear transformation matrix to be learned is used to linearly project the log encoding vector in each time window to obtain the projection vector of the log data in each time window. The projection vector of the log data in each time window is then L2-normalized to obtain the semantic vector of the log data in each time window. Represents the indicator sequence vector in the i-th time window and the semantic vector of the log data in the jth time window The cosine similarity between them, cos() represents the function for calculating the cosine similarity, and τ represents the temperature parameter; so that Maximize the value of The value of is minimized as the training objective, and the parameters of the all-MiniLM-L6-v2 model to be trained and the element values ​​in the linear transformation matrix to be learned are trained and adjusted to obtain the trained all-MiniLM-L6-v2 model and the trained linear transformation matrix.

9. The method according to claim 8, characterized in that When training the all-MiniLM-L6-v2 model and the linear transformation matrix, the following loss function is used: Among them, L1 represents the contrast loss between the indicator sequence vector and the semantic vector of the log data based on the indicator sequence vector, L2 represents the contrast loss between the indicator sequence vector and the semantic vector of the log data based on the semantic vector of the log data; N represents the number of time windows, i, j represent the sequence number of the time window, exp() represents the exponential function, log() represents the logarithmic function, Represents the indicator sequence vector in the i-th time window and the semantic vector of the log data in the i-th time window The cosine similarity between .

10. The method according to claim 3, characterized in that The obtaining of the original log data and the indicator time series matrix of the target node within the time period to be detected includes: Sending a query request in a preset format to an interface of a time series query service, wherein the query request in the preset format includes a node name, a structure to which it belongs, a service address, a target indicator name, and a sampling time interval. Upon receiving the query request, the time series query service starts a first number of data collection threads. If all of the first number of data collection threads are successfully connected, the number of concurrent data collection threads is gradually increased. If any of the first number of data collection threads fails to connect, the first number is reduced by half. If a data collection thread reports an error, a step reduction function is called to reduce the collection step size. If the request times out or the interface is abnormal, an exponential backoff strategy with random jitter is used for automatic retry. Receive the time series data returned by the interface, each time series data includes a metadata dictionary, the metadata dictionary is used to describe the indicator name and the node to which the indicator belongs, and each time series data also includes a value array, the value array records the indicator data of the corresponding indicator at each timestamp within the sampling time interval; Saving the complete time series data of each indicator of each node into a file in a preset format according to the time series data; According to the target node, the target file is located from the file in the preset format, and the indicator data is read from the target file to form the indicator time series matrix, wherein the row elements of the indicator time series matrix represent the data of different indicators at the same timestamp, and the column elements represent the data of the same indicator at different timestamps.

Images