Deep learning-based calculation network system log anomaly detection method and system

Abstract

The invention discloses a deep learning-based calculation network system log anomaly detection method and system, and the method comprises the steps: obtaining an original log stream, carrying out the traversal segmentation of the log stream based on a preset time step parameter t and a window length parameter T through employing a sliding window division method, and generating a plurality of groups of log sequences; respectively extracting semantic features, sequence features and quantity features of each group of log sequences; respectively carrying out feature processing on the semantic features, the sequence features and the quantity features; the processed semantic features, sequence features and quantity features are fused into a unified feature vector through a dynamic gating mechanism, and log anomaly classification is completed through processing of a full connection layer, a GELU activation function, Dropout regularization and a Softmax function in sequence; according to the method, the calculation efficiency and the real-time performance of log anomaly detection of the computing network system are remarkably improved; the method effectively improves the detection precision and model robustness, and adapts to a complex and dynamic log environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of log anomaly detection, and in particular relates to a method and system for detecting anomaly in computing network system logs based on deep learning. Background Art

[0002] With the rapid development of information technology and the widespread adoption of the internet, digital transformation is accelerating across all industries. Information systems have become the core support for enterprise operations and services. Log data, as a key information carrier generated during system operation, records a rich set of information, including system status, user behavior, and performance metrics. It plays an irreplaceable role in operations management, troubleshooting, and security analysis. In sectors such as finance, e-commerce, and cloud computing, large-scale distributed systems generate terabytes of log data daily, making log analysis a crucial component in ensuring system stability and reliability.

[0003] Traditional log anomaly detection methods rely primarily on manual rules and pattern matching techniques. Operations and maintenance personnel write regular expressions or keyword matching rules to filter out abnormal entries from logs and then perform fault diagnosis based on their experience.

[0004] However, with the gradual maturity of technologies related to large-scale distributed systems and the rise of various large-scale service platforms, the composition of software systems has become increasingly complex. Relying solely on manual intervention to complete log anomaly detection is difficult to adapt to the complex and changing log formats and system environments, and the maintenance cost is high. As a result, existing methods often face problems such as low computing efficiency and insufficient real-time performance when processing massive log data, making it difficult to meet the operation and maintenance needs of large-scale distributed systems. Summary of the Invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a method for detecting anomalies in computing network system logs based on deep learning, which can improve computing efficiency, real-time performance, and enhance detection accuracy and robustness; on the other hand, to provide a system for detecting anomalies in computing network system logs based on deep learning.

[0006] Technical solution: The method for detecting abnormality in a computing network system log according to the present invention comprises the following steps:

[0007] (1) After obtaining the original log stream, the sliding window partitioning method is used to perform traversal segmentation on the log stream based on the preset time step parameter △t and window length parameter △T, generating multiple groups of log sequences. This realizes adaptive segmentation processing of the continuous log stream, which not only preserves the temporal correlation of log events, but also flexibly adapts to the requirements of computing network systems of different scales through configurable time parameters (△t and △T), providing a structured input data foundation for subsequent feature extraction.

[0008] (2) Semantic features, sequence features, and quantity features of each log sequence are extracted separately, and a multi-dimensional log representation system is constructed, which overcomes the limitations of a single feature perspective and provides multi-source information support for comprehensive detection of complex abnormal scenarios;

[0009] (3) Feature processing is performed on semantic features, sequence features, and quantity features to enhance the model’s robustness to noise and sensitivity to hidden anomalies;

[0010] (4) The processed semantic features, sequence features and quantity features are fused into a unified feature vector through a dynamic gating mechanism, and are processed in sequence by a fully connected layer, GELU activation function, Dropout regularization and Softmax function to complete the log anomaly classification, realizing the adaptive fusion of heterogeneous features and solving the problem of insufficient scenario adaptability of traditional static weighting methods; combined with GELU activation function and Dropout regularization, it effectively prevents overfitting while ensuring classification accuracy, and finally outputs an interpretable anomaly probability distribution.

[0011] Preferably, step 2 includes:

[0012] (21) Log parsing: Define log parsing hyperparameters, including log parsing expression, similarity threshold, maximum depth of search tree, and stop word list; After preprocessing the log, use the SentenceBert method to embed log words and convert the log text into a fixed-dimensional numerical vector; Use the Semantic-Drain method to cluster log templates, encode the log text through SentenceBert, and calculate the cosine similarity to determine the category to which the log entry belongs;

[0013] (22) Semantic feature extraction: The original log is parsed into a log template using the Semantic-Drain method, and the pre-trained language model SentenceBert is used to generate the overall semantic representation of the log event;

[0014] (23) Sequence feature extraction: convert the log sequence into a log template sequence arranged by template ID, and generate a sequence vector according to the template number;

[0015] (24) Quantitative feature extraction: Count the number of occurrences of each log template in the log sequence and generate a quantitative feature vector.

[0016] Comprehensive representation of computing network system logs is achieved through multi-level log feature extraction: First, through intelligent log parsing combining SentenceBert word embedding and Semantic-Drain clustering, unstructured raw logs are converted into standardized log templates, which not only preserves semantic integrity but also improves processing efficiency; secondly, semantic feature extraction is used to capture the deep semantic information of the logs, sequence feature extraction is used to model the temporal pattern of log events, and quantitative feature extraction is used to count the frequency of anomalies. This constructs a "semantic-temporal-statistical" trinity feature system, providing multi-dimensional and complementary feature representations for subsequent anomaly detection, significantly improving the model's ability to recognize complex anomaly patterns. At the same time, standardized template processing greatly reduces the dimensionality of the feature space, taking into account both computational efficiency and detection accuracy.

[0017] Preferably, in the log analysis, the similarity calculation formula is:

[0018]

[0019] Among them, A and B represent two vectors of length n, represents the dot product of vectors A and B, and Represent the lengths of vector A and vector B respectively.

[0020] The semantic vectors and cosine similarity calculations generated by SentenceBert can accurately quantify the degree of semantic association between different log entries. This not only overcomes the limitations of traditional keyword or regular expression matching, but also achieves fine-grained similarity assessment of log content through deep semantic understanding. In addition, the stability and comparability of similarity calculations are ensured by vector dot product and normalization processing, providing a reliable quantitative basis for subsequent log template clustering, significantly improving the accuracy and automation level of log parsing, and laying a high-quality semantic feature foundation for anomaly detection.

[0021] Preferably, the processing of the semantic features in step 3 uses a hierarchical convolutional distillation network (HCDN) to achieve fine refinement of semantic information through multi-stage feature compression and pattern extraction, including:

[0022] Local pattern extraction: extract local n-gram semantic patterns through 3×3 convolution kernel;

[0023] Long-distance association modeling: Capturing long-distance semantic dependencies through k×k depthwise separable convolution;

[0024] Dynamic pooling layer: Adaptive pooling strategy P2 = AdaptiveAvgPool1D (C2) is used, where AdaptiveAvgPool1D is the adaptive pooling function in the Pytorch programming library, C2 is the result of the first layer of convolution, and the features are mapped to a 256-dimensional semantic space through a fully connected layer.

[0025] A multi-level feature processing architecture enables deep mining and efficient representation of log semantic information: first, a 3×3 convolution kernel is used to accurately capture local n-gram semantic patterns (such as key error phrases), then a depthwise separable convolution is used to establish long-distance semantic associations across logs, and finally a dynamic pooling layer is used to adaptively compress feature dimensions. This "local-global-dynamic" three-stage processing mechanism not only retains fine-grained semantic features but also significantly reduces computational complexity. The resulting 256-dimensional semantic space representation contains rich contextual information and has good feature discriminability, providing high-information-density semantic feature input for subsequent anomaly detection.

[0026] Preferably, the processing of the sequence features in step 3 includes:

[0027] Log sequence embedding: The template ID is mapped into a 64-dimensional continuous vector through the embedding layer, and the mapping method is E base =Embedding(S)∈R N×n×128 , where Embedding is a function in the Pytorch programming library that converts discrete vectors into continuous vectors, S is the input sequence vector, the embedding matrix is ​​randomly initialized and optimized through training, N represents the batch size, and n represents the number of log entries in the sliding window;

[0028] Position coding: Introduce sinusoidal position coding, the formula is:

[0029]

[0030] Where i is the sequence position index, k[0, M] is the dimension index, and the final sequence vector embedding expression is obtained by adding the continuous vector obtained by log sequence embedding to the position encoding: E ​​= E base +E pos ∈R N×n×64 ;

[0031] Transformer encoder: The embedded representation E of the sequence vector is linearly transformed to generate query Q, key K and value V: Q = EW q , K=EW k , V=EW k , and calculate its attention score:

[0032]

[0033] Multiply the attention score by the value V to get the attention-weighted vector:

[0034]

[0035] Efficient modeling of log sequence features is achieved through a three-stage architecture of "embedding layer + position encoding + Transformer". First, the discrete template ID is mapped into a 64-dimensional continuous vector through a learnable embedding layer, preserving the semantic correlation between templates. Then, sinusoidal position encoding is combined to inject temporal position information, solving the problem that traditional RNN models are difficult to process in parallel. Finally, the Transformer's self-attention mechanism is used to dynamically capture long-distance dependencies and automatically focus on key log events by calculating query-key attention weights. This not only breaks through the efficiency bottleneck of traditional sequence models when processing extremely long log sequences, but also can adaptively identify potential correlation patterns between abnormal events, significantly improving the model's ability to detect complex abnormal sequences while maintaining computational efficiency and scalability.

[0036] Preferably, the processing of the quantitative features in step 3 adopts a bidirectional LSTM-attention network, including:

[0037] Bidirectional LSTM encoding: Forward LSTM captures normal evolution trends: where x t is the initial quantity feature vector, LSTM forward It is the forward propagation part of the bidirectional LSTM network; the backward LSTM finds the reverse mutation point: Among them LSTM backward It is the backward propagation part of the bidirectional LSTM network; the splicing result is:

[0038] Attention mechanism: Calculate attention score:

[0039]

[0040] Where W h ∈R 64×128 , W x ∈R 64×M , v∈R 64 is a learnable parameter, v T and b are initialization hyperparameters;

[0041] Introducing a learnable gating parameter g to generate dynamically adjusted attention weights Where σ is the sigmoid function, α t Parameters for adjusting the time step weight;

[0042] The final output is formed by weighting and aggregating the features of each time step through the attention weight: Where n represents the total number of time steps.

[0043] Intelligent processing of quantitative features is achieved through bidirectional temporal modeling and dynamic feature focusing: the bidirectional LSTM structure simultaneously captures the positive evolution law and reverse mutation characteristics of the number of log events, forming a complete temporal context representation; combined with the attention mechanism with learnable gating parameters, it can adaptively identify and enhance the key quantitative features of abnormal time points, effectively suppressing the interference information of normal logs; it not only solves the problem of traditional statistical methods being insensitive to temporal patterns, but also significantly improves the model's detection ability for sudden and gradual anomalies through dynamic weight allocation, so that the representation of quantitative features not only retains the complete temporal evolution information, but also highlights the key feature points related to the anomaly.

[0044] Preferably, the dynamic gating mechanism described in step 4 includes:

[0045] Feature splicing: Splice semantic features, sequence features, and quantity features along the channel dimension to form a vector H = [h s ||h q ||h r ]∈R N×(256+128+64) , where h s , h q , h r These are the results after modeling the above three types of features;

[0046] Feature compression: The feature space is compressed through the GELU activation function and the 64-dimensional fully connected layer:

[0047] where W1∈R 448×64 , b1∈R 64 ;

[0048] Weighted fusion: Generate weight vector α=Softmax(W2·z1+b2)∈R through the fully connected layer N×3 , where W2∈R 64×3 , b1∈R 3 , perform weighted fusion of semantic features, sequence features and quantity features according to the weight vector: h fused =α s ·h s +α q ·h q +α r ·h r ∈R N×448 , where α s , α q , α r are the weights corresponding to the above three types of features respectively.

[0049] Through the three-stage fusion strategy of "feature splicing-compression-adaptive weighting", intelligent integration of multimodal log features is achieved: first, the integrity of the original features is preserved through channel splicing, then the GELU activation function and fully connected layer are used to perform nonlinear compression and dimensionality reduction of the feature space, and finally, a learnable dynamic weight allocation module is used to automatically adjust the contribution ratio of semantic, sequence and quantitative features according to the current log scenario. This mechanism not only solves the problem that traditional fixed weight fusion methods are difficult to adapt to complex and changeable log patterns, but also can dynamically highlight the most relevant feature dimensions for different types of anomalies. It not only ensures the flexibility of feature fusion, but also ensures the optimality of feature combination through end-to-end training optimization, significantly improving the adaptability and discrimination accuracy of the model in diverse anomaly detection scenarios.

[0050] The computing network system log anomaly detection system of the present invention includes:

[0051] The log sequence partitioning module is used to obtain the original log stream and then use the sliding window partitioning method to traverse the log stream based on the preset time step parameter △t and window length parameter △T to generate multiple groups of log sequences;

[0052] Feature extraction module, used to extract semantic features, sequence features and quantity features of each group of log sequences;

[0053] Feature processing module, used to process semantic features, sequence features and quantity features respectively;

[0054] The feature fusion and classification module is used to fuse the processed semantic features, sequence features, and quantitative features into a unified feature vector through a dynamic gating mechanism. The vector is then processed sequentially through a fully connected layer, a GELU activation function, Dropout regularization, and a Softmax function to complete log anomaly classification.

[0055] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the method for detecting anomalies in a computing network system log based on deep learning is implemented.

[0056] An electronic device includes a memory, a processor, and a program stored in the memory and runnable on the processor, characterized in that when the processor executes the program, the method for detecting anomalies in a computing network system log based on deep learning is implemented.

[0057] Beneficial effects: Compared with the existing technology, the present invention has the following significant advantages: 1. The sliding window partitioning method is combined with the multi-feature parallel extraction mechanism to significantly improve the computational efficiency and real-time performance of the network system log anomaly detection; the collaborative processing of the layered convolutional distillation network (HCDN), Transformer encoder and bidirectional LSTM-attention network is adopted to effectively improve the detection accuracy and model robustness, and adapt to the complex and dynamic log environment; 2. The dynamic gating mechanism adaptively integrates semantic features, sequence features and quantitative features, solves the problem that the importance of features changes dynamically with the scene in traditional methods, and realizes the detection of log anomaly patterns. Multi-dimensional precise modeling enhances the system's ability to identify diverse anomaly types; 3. Combining deep separable convolution, adaptive pooling, and dynamic attention weight allocation technology, it reduces computational overhead while ensuring model performance, thereby improving the lightweight level of the method; through the combined application of pre-trained language models and log template clustering, the model's generalization ability on unseen log data is further enhanced; 4. A complete closed loop is formed from log parsing to anomaly classification, without the need for manual intervention in feature engineering; through a unified feature processing and fusion framework, end-to-end automated analysis of computing network system logs from raw data to anomaly judgment is achieved, greatly reducing the complexity of operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0059] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0060] like Figure 1 As shown, the specific steps of the present invention are:

[0061] After obtaining the complete original log stream, we use the sliding window partitioning method, introduce the time step parameter △t and the window length parameter △T, and perform traversal segmentation on the log stream to obtain a group of log sequences;

[0062] Next, multi-feature extraction is performed. Based on the log grouping results, the semantic features, sequence features, and quantitative features of each group of log sequences are extracted. First, the Semantic-Drain method is used to parse the original log data into log templates. Before log parsing, the relevant hyperparameters in the log parsing process are defined, including the log parsing expression, similarity threshold, maximum depth of the search tree, and stop word list; then the log is preprocessed to remove repeated redundant information; the SentenceBert method is used to quickly embed log words and convert the log text into a fixed-dimensional numerical vector; finally, the Semantic-Drain method is used to cluster log templates. The deep tree structure is searched based on the length of the log content, the first word of the log, and the similarity of the log text to determine the log category to which the log entry belongs. When calculating the similarity, the two log texts are encoded into vectors using the SentenceBert method, and the segment similarity between the two texts is calculated using the cosine similarity calculation formula between vectors. The similarity calculation formula is as follows:

[0063]

[0064] Among them, A and B represent two vectors of length n, represents the dot product of vectors A and B, and Represent the lengths of vector A and vector B, respectively. First, the log vector is compared with the log group mean vector for similarity to quickly identify potential matching groups. Then, the textual similarity between the log vector and each log vector within the group is compared with the similarity threshold set in the hyperparameters. If the textual similarity exceeds the threshold, the entry with the greatest similarity is returned. Otherwise, if no log group meets the criteria, a new log group containing only the current log identifier is created, with the log content as the log event, and the newly created log group is integrated into the parse tree.

[0065] After the log parsing is completed, semantic, sequence and quantity features are extracted respectively.

[0066] Semantic feature extraction: The log template obtained by the log parsing algorithm is input into the SentenceBert encoder. The encoder first encodes the vocabulary in the log template into numeric vectors and then aggregates these vectors to generate the overall semantic representation of the log event. This results in a 384-dimensional vector as the semantic embedding expression of the log segment.

[0067] Sequence feature extraction: After the raw log data is processed by the log parsing algorithm, it is converted into a specific log template. Each log template is assigned a unique identifier, typically numbered in numerical sequence, such as T1, T2, T3, and so on. The log sequence is converted into its corresponding template ID according to the order of the original log entries. The resulting sequence is the log template sequence. The vector formed by arranging the log template sequence by template number is the log sequence vector. For example, the raw log sequence S1 = {s1, s2, s3} can be converted into the corresponding log template sequence T1 = {t1, t2, t3}. The log template numbers in T1 are extracted, and t1 corresponds to 1, t2 to 2, and t3 to 3. The vector composed of these numbers (1, 2, 3) is the sequence feature vector of sequence S1.

[0068] Quantitative feature extraction: The quantitative feature vector is used to record the number of occurrences of each log event in a specific time window. The length of this vector is determined by the number of all different event types in the log (i.e., the total number of log templates). For example, for log sequences S1, S2, and S3, assuming that their corresponding log template sequences are T1 = {t1, t1, t1}, T2 = {t1, t1, t2}, and T3 = {t1, t2, t3}, there are three log templates in total, so the dimension of the quantitative feature is 3; in S1, the number of occurrences of the t1, t2, and t3 templates are 3, 0, and 0, respectively, so the quantitative feature vector corresponding to S1 is (3, 0, 0), the quantitative feature vector corresponding to S2 is (2, 1, 0), and so on for S3.

[0069] After extracting the feature vectors from the log sequence, the present invention further designs adaptive neural networks for different feature types based on the characteristics of each feature, thereby improving the fitting ability of the model.

[0070] Based on the following three observations, we designed a hierarchical convolutional distillation network to finely refine semantic information: 1) The 384-dimensional vector space contains a large amount of redundant information. Directly inputting this information into a traditional convolutional neural network (CNN) can lead to a sharp increase in the number of model parameters and computational complexity. 2) The semantic patterns in log text exhibit local correlations. For example, abnormal keyword combinations such as "OutOfMemory" often appear in the form of n-grams, which requires the algorithm to consider mechanisms that can effectively extract local features. 3) The semantic patterns corresponding to different types of anomalies vary significantly, making it difficult for a single convolutional kernel size to capture all important semantic patterns. Therefore, we use a hierarchical convolutional distillation network (HCDN) to achieve fine-grained refinement of semantic information through multi-stage feature compression and pattern extraction. HCDN is divided into three layers: local pattern extraction, long-distance association modeling, and dynamic pooling. Local pattern extraction: By introducing a 3×3 convolution kernel to extract local n-gram semantic patterns, it captures local abnormal patterns of three-word combinations such as "OutOfMemory", while reducing the vector dimension, removing redundant information, and reducing the complexity of subsequent processing. Long-distance association modeling: A k×k depthwise separable convolution is designed to capture long-distance semantic dependencies in log text, ensuring that abnormal patterns are considered in a wider context, while reducing the number of parameters in large-scale convolution operations. It is divided into two steps: depthwise convolution and pointwise convolution. Dynamic pooling layer: To adapt to the feature expression of different abnormal types, the present invention applies an adaptive pooling strategy, and its calculation process is: P2=AdaptiveAvgPool1D(C2), where AdaptiveAvgPool1D is the adaptive pooling function in the Pytorch programming library, and C2 is the result of the first convolution layer. Finally, the features are mapped to a 256-dimensional semantic space through a fully connected layer.

[0071] Through log sequence embedding, position encoding and Transformer encoder methods, we can obtain sequence feature vectors with long-range dependency modeling capabilities; Log sequence embedding: The discrete ID is mapped into a 64-dimensional continuous vector through the embedding layer. The mapping method is: E base =Embedding(S)∈R N×n×128 , where Embedding is a function in the Pytorch programming library that converts discrete vectors into continuous vectors, S is the input sequence vector, the embedding matrix is ​​randomly initialized and optimized through training, N represents the batch size, and n represents the number of log entries in the sliding window; Position encoding: To enhance the model's ability to perceive sequence positions, this paper introduces sinusoidal position encoding:

[0072]

[0073] Where i is the sequence position index, k[0, M] is the dimension index, and the final sequence vector embedding expression is obtained by adding the embedding vector obtained in the first step to the position code: E = E base +E pos ∈R N×n×64 ; Transformer encoder: The embedded expression E of the sequence vector is linearly transformed to generate query Q, key K and value V: Q = EW q , K=EW k , V=EW k , and calculate its attention score:

[0074]

[0075] Multiply the attention score by the value V to get the attention-weighted vector:

[0076]

[0077] The resulting sequence vector can effectively capture sequence position information, ensuring the ability to model long-range dependencies while effectively reducing computational overhead.

[0078] A bidirectional LSTM-attention network is used to process quantitative features. The forward LSTM is used to capture the normal evolution trend, and the backward LSTM is used to find the reverse mutation point. The bidirectional LSTM encoding is calculated as follows: Forward: where x t is the initial quantity feature vector, LSTM forward It is the forward propagation part of the bidirectional LSTM network; backward: Among them LSTM backward It is the backward propagation part of the bidirectional LSTM network; the splicing result is: The attention mechanism is used to focus on key information and improve feature representation capabilities. The attention score calculation formula is:

[0079]

[0080] Among them, W h ∈R 64×128 , W x ∈R 64×M , v∈R 64 is a learnable parameter, v T and b are initialization hyperparameters.

[0081] Introducing a learnable gating parameter g to generate dynamically adjusted attention weights:

[0082]

[0083] Among them, σ is the sigmoid function, α t is the parameter for adjusting the time step weight.

[0084] The final output is formed by weighting and aggregating the features of each time step through the attention weight: , where n represents the total number of time steps.

[0085] The processed semantic features, quantitative features, and sequence features are integrated into a unified feature vector through a dynamic gating mechanism. The fused feature representation sequence is then classified through a fully connected network layer, GELU activation function, Dropout regularization, and Softmax function. The dynamic gating mechanism can integrate heterogeneous features and aims to address the problem of dynamic changes in feature importance with the scene based on the principle of dynamic weight distribution. Feature concatenation: The three types of features are concatenated along the channel dimension to form a vector: H = [h s ||h q ||h r ]∈R N×(256+128+64) , where h s , h q , h r These are the results after modeling the above three types of features; Feature compression: Using the activation function GELU, the feature space is compressed through a 64-dimensional fully connected layer:

[0086] z1=GELU(W1·H+b1)∈R N×64 , where W1∈R 448×64 , b1∈R 64

[0087] Weighted fusion: Generate weight vectors through fully connected layers:

[0088] α=Softmax(W2·z1+b2)∈R N×3 , where W1∈R 64×3 , b1∈R 3

[0089] The three types of features are weightedly fused according to the weight vector:

[0090] h fused =α s ·h s +α q ·h q +α r ·h r ∈R N×448

[0091] where α s , α q , α r are the weights corresponding to the above three types of features;

[0092] Based on the above functions, an anomaly detection method that integrates log semantic information and multiple log features is implemented, and the detection accuracy and robustness are improved through deep learning technology.

[0093] To validate the performance of the proposed algorithm in a real-world production environment, we used log data provided by the open-source project LogPai. These logs are sourced from real-world production environments and are widely used in academia and industry, demonstrating authoritative reliability. We used the most representative HDFS and BGL datasets. HDFS is a distributed log dataset generated using a load-balanced deployment on a 203-node HDFS cluster. The BGL dataset is log data collected from the BlueGene / L supercomputer system at Lawrence Livermore National Laboratory.

[0094] Evaluation benchmarks and evaluation indicators: This algorithm is compared with three log anomaly detection algorithms that dominate the academic community: (1) Deeplog uses the sequence modeling characteristics of the LSTM network to construct a prediction model for log template sequences and log parameters. (2) LogCNN fully utilizes the characteristics of the CNN network structure to extract features from logs. (3) LogBert is based on the BERT model and introduces two training tasks: predicting masked log keys and making normal logs as close to each other as possible in the vector space. Anomaly detection and recognition of logs is a typical binary classification problem. Therefore, the present invention uses four quantitative evaluation indicators commonly used in academia, namely precision, accuracy, recall rate and F1 score, to evaluate the effectiveness of the algorithm.

[0095] As shown in Table 1, Pytorch was used as the main framework for model building, training, and prediction. Python was used as the main tool for data processing and service construction. Java was used to build and implement the visualization system. The CPU configuration was i5-12600kf and the GPU configuration was RTX 4070S.

[0096] Table 1 Introduction to experimental environment configuration

[0097]

[0098] As can be seen from Table 2, on the HDFS dataset, the algorithm of the present invention performs significantly better than the other three baseline algorithms in terms of precision, recall and F1 value.

[0099] Table 2 compares the accuracy, precision, recall and F1 value of the proposed algorithm with DeepLog, LogCNN and LogBert on the HDFS dataset

[0100]

[0101] As can be seen in Table 3, on the BGL dataset, the algorithm of the present invention performs significantly better than the other three baseline algorithms in terms of accuracy, precision, recall and F1 value.

[0102] Table 3 compares the accuracy, precision, recall and F1 value of the proposed algorithm with DeepLog, LogCNN and LogBert on the BGL dataset

[0103]

[0104] After analysis, the reasons for the experimental results are as follows: 1) DeepLog only relies on the LSTM model to model the log sequence, and does not fully consider the various features of the log in the modeling process, and is more dependent on the order of the log sequence; 2) LogCNN fully utilizes the characteristics of the CNN network structure in model design to extract features from the log, which improves the effect of anomaly detection, but still does not fully extract the large amount of semantic information contained in the log itself; 3) LogBert uses the Bert model in model design to fully extract the semantic information contained in the log, and has a good effect on the analysis of complex templates, but only considers a single semantic feature and ignores the sequence feature of the log. The algorithm proposed in the present invention fully extracts the semantic anomaly, sudden anomaly and sequence anomaly features that may appear in the log in the modeling stage through multi-feature extraction and fusion. Therefore, it has a greater effect than the baseline algorithm on complex and changeable real data sets.

[0105] The embodiments of the present invention can also provide a visual log anomaly detection system, which includes: a log collection module, which completes the automated log collection function and persists it in a cloud server for storage; a log parsing module, which realizes the parsing from the original log to the log module and persists the parsed results; a log feature extraction module, which completes the extraction of the semantic features, quantitative features and sequence features of the log, converts them into vector data that can be recognized by the log anomaly model, and persists the data; a log anomaly model training module, which pre-trains the log anomaly detection algorithm and persists the model parameters obtained from the training results; a log anomaly detection module, which provides log anomaly detection functions for incremental log text and full log text; a log anomaly visualization interface module, which provides a visual system interface, clearly and intuitively presents the results of log anomaly model detection, and provides functions such as log anomaly source location and anomaly statistics.

[0106] After users train the log anomaly detection model and set information such as the log data source and log format, they can automatically complete log anomaly detection and improve detection accuracy and robustness.

[0107] The invention also discloses an electronic device.

[0108] Specifically, the electronic device can be a computer device such as a desktop computer, a laptop computer, a PDA, and a cloud server. The computer device may include, but is not limited to, a processor and a memory. The processor and the memory may be connected via a bus or other means. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, graphics processing units (GPU), embedded neural network processors (NPU) or other dedicated deep learning coprocessors, discrete gate or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned chips.

[0109] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory. The memory may include a program storage area and a data storage area, wherein the program storage area may store a control unit, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0110] The invention also discloses a computer-readable storage medium.

[0111] Specifically, a computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method in the above-mentioned method implementation is implemented. Those skilled in the art will understand that the implementation of all or part of the process in the above-mentioned embodiment method of the present application can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it may include the process of the implementation of each of the above-mentioned methods. Among them, the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated: HDD) or a solid-state drive (SSD), etc.; the storage medium may also include a combination of the above-mentioned types of memories.

Claims

1. A method for detecting anomaly in computing network system logs based on deep learning, characterized in that: The following steps are involved: (1) After obtaining the original log stream, the sliding window partitioning method is used to perform traversal segmentation on the log stream based on the preset time step parameter △t and window length parameter △T to generate multiple groups of log sequences; (2) Extract the semantic features, sequence features, and quantity features of each group of log sequences respectively; (3) Perform feature processing on semantic features, sequence features and quantity features respectively; (4) The processed semantic features, sequence features and quantity features are fused into a unified feature vector through a dynamic gating mechanism, and are processed in sequence by a fully connected layer, GELU activation function, Dropout regularization and Softmax function to complete the log anomaly classification.

2. The method for detecting abnormality in a computing network system log according to claim 1, characterized in that: Step 2 includes: (21) Log parsing: Define log parsing hyperparameters, including log parsing expression, similarity threshold, maximum depth of search tree, and stop word list; After preprocessing the log, use the SentenceBert method to embed log words and convert the log text into a fixed-dimensional numerical vector; Use the Semantic-Drain method to cluster log templates, encode the log text through SentenceBert, and calculate the cosine similarity to determine the category to which the log entry belongs; (22) Semantic feature extraction: The original log is parsed into a log template using the Semantic-Drain method, and the pre-trained language model SentenceBert is used to generate the overall semantic representation of the log event; (23) Sequence feature extraction: convert the log sequence into a log template sequence arranged by template ID, and generate a sequence vector according to the template number; (24) Quantitative feature extraction: Count the number of occurrences of each log template in the log sequence and generate a quantitative feature vector.

3. The method for detecting abnormality in a computing network system log according to claim 2, characterized in that: In the log analysis, the similarity calculation formula is: Among them, A and B represent two vectors of length n, represents the dot product of vectors A and B, and Represent the lengths of vector A and vector B respectively.

4. The method for detecting abnormality in a computing network system log according to claim 1, wherein: The semantic features described in step 3 are processed using a layered convolutional distillation network (HCDN). This network uses multi-stage feature compression and pattern extraction to achieve fine-grained semantic information extraction, including: Local pattern extraction: extract local n-gram semantic patterns through 3×3 convolution kernel; Long-distance association modeling: Capturing long-distance semantic dependencies through k×k depthwise separable convolution; Dynamic pooling layer: Adaptive pooling strategy P2 = AdaptiveAvgPool1D (C2) is used, where AdaptiveAvgPool1D is the adaptive pooling function in the Pytorch programming library, C2 is the result of the first layer of convolution, and the features are mapped to a 256-dimensional semantic space through a fully connected layer.

5. The method for detecting abnormality in a computing network system log according to claim 1, wherein: The processing of sequence features in step 3 includes: Log sequence embedding: The template ID is mapped into a 64-dimensional continuous vector through the embedding layer, and the mapping method is E base =Embedding(S)∈R N×n×128 , where Embedding is a function in the Pytorch programming library that converts discrete vectors into continuous vectors, S is the input sequence vector, the embedding matrix is ​​randomly initialized and optimized through training, N represents the batch size, and n represents the number of log entries in the sliding window; Position coding: Introduce sinusoidal position coding, the formula is: Where i is the sequence position index, k[0, M] is the dimension index, and the final sequence vector embedding expression is obtained by adding the continuous vector obtained by log sequence embedding to the position encoding: E ​​= E base +E pos ∈R N×n×64 ; Transformer encoder: The embedded representation E of the sequence vector is linearly transformed to generate query Q, key K and value V: Q = EW q , K=EW k , V=EW k , and calculate its attention score: Multiply the attention score by the value V to get the attention-weighted vector:

6. The method for detecting abnormality in a computing network system log according to claim 1, characterized in that: The processing of the quantitative features described in step 3 uses a bidirectional LSTM-attention network, including: Bidirectional LSTM encoding: Forward LSTM captures normal evolution trends: where x t is the initial quantity feature vector, LSTM forward It is the forward propagation part of the bidirectional LSTM network; the backward LSTM finds the reverse mutation point: Among them LSTM backward It is the backward propagation part of the bidirectional LSTM network; the splicing result is: Attention mechanism: Calculate attention score: Where W h ∈R 64×128 , W x ∈R 64×M , v∈R 64 is a learnable parameter, v T and b are initialization hyperparameters; Introducing a learnable gating parameter g to generate dynamically adjusted attention weights Where σ is the sigmoid function, α t Parameters for adjusting the time step weight; The final output is formed by weighting and aggregating the features of each time step through the attention weight: Where n represents the total number of time steps.

7. The method for detecting abnormality in a computing network system log according to claim 1, characterized in that: The dynamic gating mechanism described in step 4 includes: Feature splicing: Splice semantic features, sequence features, and quantity features along the channel dimension to form a vector H = h s ||h q ||h r ]∈R N×(256+128+64) , where h s , h q , h r These are the results after modeling the above three types of features; Feature compression: The feature space is compressed through the GELU activation function and the 64-dimensional fully connected layer: in b1∈R 64 ; Weighted fusion: Generate weight vector α=Softmax(W2·z1+b2)∈R through the fully connected layer N×3 , where W2∈R 64×3 , b1∈R 3 , perform weighted fusion of semantic features, sequence features and quantity features according to the weight vector: h fused =α s ·h s +α q ·h q +α r ·h r ∈R N×448 , where α s , α q , α r are the weights corresponding to the above three types of features respectively.

8. A deep learning-based computing network system log anomaly detection system, characterized by: include: The log sequence partitioning module is used to obtain the original log stream and then use the sliding window partitioning method to traverse the log stream based on the preset time step parameter △t and window length parameter △T to generate multiple groups of log sequences; Feature extraction module, used to extract semantic features, sequence features and quantity features of each group of log sequences; Feature processing module, used to process semantic features, sequence features and quantity features respectively; The feature fusion and classification module is used to fuse the processed semantic features, sequence features, and quantitative features into a unified feature vector through a dynamic gating mechanism. The vector is then processed sequentially through a fully connected layer, a GELU activation function, Dropout regularization, and a Softmax function to complete log anomaly classification.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the deep learning-based computing network system log anomaly detection method according to any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for detecting anomalies in computing network system logs based on deep learning according to any one of claims 1 to 7 is implemented.

Images