Anomaly detection and processing method and system based on deep learning

Abstract

The present invention relates to the field of electronic digital data processing technology, specifically to an anomaly detection and processing method and system based on deep learning. The present invention first extracts and fuses features of a log text sequence, a log timestamp sequence, and a set of operating status indicator sequences to obtain an initial feature vector sequence; then a hierarchical graph neural network is constructed to update, propagate, and fuse the initial feature vector sequence to obtain a multi-level fusion feature vector sequence, and a standardized pattern vector sequence is calculated in combination with the log text sequence and the set of operating status indicator sequences, and an adaptive anomaly threshold is obtained according to the standardized pattern vector sequence and candidate anomaly logs are detected; finally, the candidate anomaly logs are verified, the verified anomaly logs are determined, the verified anomaly logs are classified, and a processing suggestion text is generated according to the classification results and an early warning prompt is given. The present invention can reduce the false detection rate and improve the accuracy of log anomaly detection.

Description

Technical Field

[0001] The present invention relates to the technical field of electrical digital data processing, and in particular to an anomaly detection and processing method and system based on deep learning. Background Art

[0002] In modern information systems and network environments, log data is an important record of system operation status and is widely used for monitoring, fault diagnosis, and security analysis. With the increase in system complexity and the expansion of business scale, the scale and diversity of log data are also growing exponentially. Traditional anomaly detection methods, such as rule-based or statistical model-based methods, are difficult to adapt to complex and changing log patterns, and have limited detection capabilities for unknown or new anomalies.

[0003] Deep learning technology has powerful feature extraction and pattern recognition capabilities and is widely used in the field of anomaly detection. Anomaly detection methods based on deep learning can automatically learn the potential features and correlations of log data, improving the accuracy and generalization of detection. However, log anomaly detection methods based on deep learning still face some challenges in practical applications. For example, due to the diversity and complexity of log data, there are uncommon but normal log patterns, which leads to an increase in the false positive rate. At the same time, the existing methods lack further verification of anomalies after anomaly detection, which limits the efficiency of anomaly processing and the accuracy of anomaly detection.

[0004] To this end, an anomaly detection and processing method and system based on deep learning are proposed. Summary of the invention

[0005] The purpose of the present invention is to provide an anomaly detection and processing method and system based on deep learning. First, feature extraction and fusion are performed on a log text sequence, a log timestamp sequence, and a set of operating status indicator sequences to obtain an initial feature vector sequence; then a hierarchical graph neural network is constructed to perform feature update, propagation, and fusion on the initial feature vector sequence to obtain a multi-level fused feature vector sequence; then, a standardized pattern vector sequence is calculated by combining the log text sequence, the set of operating status indicator sequences, and the multi-level fused feature vector sequence, and an adaptive anomaly threshold is obtained according to the standardized pattern vector sequence and candidate anomaly logs are detected; finally, the candidate anomaly logs are verified, the verified anomaly logs are determined, the verified anomaly logs are classified, and a processing suggestion text is generated according to the classification result and an early warning prompt is given.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] Deep learning-based anomaly detection and processing methods, including:

[0008] Obtain a log text sequence, a log timestamp sequence, and a set of operating status indicator sequences at the corresponding time; the operating status indicator sequence set includes A sequence of operating status indicators;

[0009] Constructing a feature extraction network, extracting a semantic feature vector sequence according to the log text sequence, extracting a time series feature vector sequence according to the log timestamp sequence, and extracting a state feature vector sequence according to the operating state indicator sequence set; fusing the semantic feature vector sequence, the time series feature vector sequence, and the state feature vector sequence to obtain an initial feature vector sequence;

[0010] Constructing a hierarchical graph neural network to perform feature update, propagation and fusion on the initial feature vector sequence to obtain a multi-level fusion feature vector sequence; the hierarchical graph neural network includes a message layer, a session layer and a system layer;

[0011] Construct a candidate anomaly detection network, calculate a standardized pattern vector sequence according to the log text sequence, the operating status indicator sequence set and the multi-level fusion feature vector sequence; obtain an adaptive anomaly threshold according to the standardized pattern vector sequence and detect candidate anomaly logs;

[0012] An exception verification processing network is constructed to verify the candidate exception logs and determine the verified exception logs; the verified exception logs are classified to obtain exception categories and exception cause categories; a processing suggestion text is generated according to the exception category and the exception cause category and an early warning prompt is given.

[0013] Preferably, extracting a semantic feature vector sequence according to the log text sequence includes:

[0014] Preprocessing the log text sequence according to the log template matching rule to obtain a standardized log template sequence;

[0015] Convert each standardized log template in the standardized log template sequence into a word vector according to a pre-trained word vector model to obtain a word vector sequence;

[0016] The word vector sequence is processed through a multi-head self-attention mechanism to obtain a semantic feature vector sequence.

[0017] Preferably, extracting a time series feature vector sequence according to the log timestamp sequence includes:

[0018] Set a time window for each log timestamp ; Indicates the number of timestamps in the log sequence. timestamp of the log, represents the step size of the time window;

[0019] Calculate the time interval sequence within each time window according to the log timestamp sequence;

[0020] Calculating the time series statistical features of each log timestamp according to the time interval sequence to obtain a time series statistical feature vector; the time series statistical features include time interval mean, time interval standard deviation, time interval skewness and time interval kurtosis;

[0021] Analyzing the periodic characteristics of the log timestamp sequence by fast Fourier transform to obtain a periodic characteristic vector of each log timestamp; the periodic characteristic vector includes a plurality of main frequency components;

[0022] Analyzing the trend characteristics of the time interval sequence by exponential smoothing to obtain a trend characteristic vector of each log timestamp; the trend characteristic vector includes a smoothing value, a smoothing value change amount and a smoothing value change rate;

[0023] The time series statistical feature vector, the period feature vector and the trend feature vector are concatenated to obtain a time series feature vector;

[0024] The time series feature vectors of all the log timestamps are calculated to obtain the time series feature vector sequence.

[0025] Preferably, extracting the state feature vector sequence according to the operating state indicator sequence set includes: calculating The state statistical characteristics and state change characteristics of the operating status indicators in each time window are obtained. The initial state feature vector; The initial state feature vectors are spliced ​​in dimension to obtain the state feature vector; the state feature vectors in all the time windows are calculated to obtain the state feature vector sequence; the state statistical features include the state indicator mean and the state indicator standard deviation; the state change features include the state indicator change rate and the state indicator change acceleration.

[0026] Preferably, fusing the semantic feature vector sequence, the temporal feature vector sequence and the state feature vector sequence comprises:

[0027] The semantic feature vector sequence, the temporal feature vector sequence and the state feature vector sequence are standardized and concatenated in the vector dimension to obtain a standardized feature vector sequence;

[0028] An adaptive weight vector is generated according to the standardized feature vector sequence, and the standardized feature vector sequence and the adaptive weight vector are weightedly fused to obtain the initial feature vector sequence.

[0029] Preferably, the hierarchical graph neural network comprises:

[0030] A message node set and a message edge set are constructed according to the log text sequence and the initial feature vector sequence to obtain a message layer graph; the initial feature of each message node is the initial feature vector corresponding to each log text; the message edge set includes a temporal association edge set and a semantic association edge set;

[0031] The message node set is divided to obtain a session node set, and a session edge set is constructed according to the log text sequence to obtain a session layer graph; the initial feature of each session node is a weighted average of the initial features of the message node corresponding to the session node; the session edge set includes a time-dependent edge set and a function-dependent edge set;

[0032] Obtaining system component information to construct a system node set and a system edge set, and obtaining a system layer graph; each system node represents a system component, and the system node initial feature is an aggregate feature of all the session node initial features related to the system component; the system edge set includes a call relationship edge set and a data dependency edge set;

[0033] A graph attention mechanism is applied to the message layer, the session layer and the system layer to perform feature propagation and update respectively, so as to obtain a message node feature sequence, a session node feature sequence and a system node feature sequence; a message session mapping matrix and a session system mapping matrix are constructed, and feature fusion between different graphs is performed on the message node features, the session node features and the system node features to obtain the multi-level fused feature sequence.

[0034] Preferably, calculating the standardized pattern vector sequence according to the log text sequence, the operating status indicator sequence set and the multi-level fusion feature vector sequence includes:

[0035] Set the first scale time window for each log timestamp and the second scale time window ; Indicates the number of timestamps in the log sequence. Log timestamp, and Respectively represent the step size of the first scale time window and the second scale time window;

[0036] In each of the first scale time window and the second scale time window, log statistical features, state statistical features and pattern features are extracted and spliced ​​to obtain a first window feature vector and a second window feature vector of each log timestamp; the log statistical features include log quantity distribution and keyword frequency; the pattern features are aggregated features of multi-level fusion features;

[0037] Conditional entropy is calculated according to the first window feature vector and the second window feature vector respectively to obtain a first conditional entropy and a second conditional entropy, and an entropy correlation coefficient between the first conditional entropy and the second conditional entropy is calculated; a standardized pattern vector is established according to the first conditional entropy, the second conditional entropy and the entropy correlation coefficient; the standardized pattern vector of all the log timestamps is calculated to obtain a standardized pattern vector sequence.

[0038] Preferably, obtaining an adaptive anomaly threshold according to the standardized pattern vector sequence and detecting candidate anomaly logs comprises:

[0039] Detecting a mutation time point according to the standardized pattern vector sequence, and segmenting the log timestamp sequence according to the mutation time point to obtain a plurality of segmented timestamp sequences;

[0040] Calculate the log anomaly score at each log timestamp according to the standardized pattern vector sequence to obtain an anomaly score sequence;

[0041] Fitting a linear model in each segmented timestamp sequence according to the standardized pattern vector sequence and the anomaly score sequence to obtain a piecewise linear model;

[0042] Calculating an adaptive anomaly threshold for each of the log timestamps according to the piecewise linear model;

[0043] The candidate anomaly log is detected according to the anomaly score sequence and the adaptive anomaly threshold.

[0044] Preferably, the abnormal verification processing network includes:

[0045] Calculate the anomaly score deviation of the candidate anomaly log according to the anomaly score sequence and the piecewise linear model; calculate the pattern deviation of the candidate anomaly log according to the standardized pattern vector sequence; extract the score change sequence and the pattern vector change sequence according to the anomaly score sequence and the standardized pattern vector sequence within the time window corresponding to the candidate anomaly log, and calculate the time series correlation of the score change sequence and the pattern vector change sequence; calculate the comprehensive verification score according to the anomaly score deviation, the pattern deviation and the time series correlation; when the comprehensive verification score is greater than the verification threshold, determine the candidate anomaly log as the verified anomaly log;

[0046] The score deviation, the pattern deviation, the time series correlation, the initial feature vector and the multi-level fusion feature vector of the verified exception log are extracted to construct an exception feature vector; the exception feature vector is processed by a first multi-layer perceptron to obtain an exception category; the exception feature vector is processed by a second multi-layer perceptron to obtain an exception cause category; a historical processing experience library is matched according to the exception category and the exception cause category, a processing suggestion text is output, and an early warning prompt is issued.

[0047] Deep learning-based anomaly detection and processing system, including:

[0048] The data acquisition unit acquires a log text sequence, a log timestamp sequence, and a set of operation status indicator sequences at corresponding times; the set of operation status indicator sequences includes A sequence of operating status indicators;

[0049] A feature extraction unit extracts a semantic feature vector sequence according to the log text sequence, extracts a time series feature vector sequence according to the log timestamp sequence, and extracts a state feature vector sequence according to the operating state indicator sequence set; and fuses the semantic feature vector sequence, the time series feature vector sequence, and the state feature vector sequence to obtain an initial feature vector sequence;

[0050] A hierarchical graph unit constructs a hierarchical graph neural network to perform feature update, propagation and fusion on the initial feature vector sequence to obtain a multi-level fusion feature vector sequence; the hierarchical graph neural network includes a message layer, a session layer and a system layer;

[0051] A candidate anomaly detection unit calculates a standardized pattern vector sequence according to the log text sequence, the operating status indicator sequence set and the multi-level fusion feature vector sequence; obtains an adaptive anomaly threshold according to the standardized pattern vector sequence and detects candidate anomaly logs;

[0052] The exception verification processing unit verifies the candidate exception logs to determine the verified exception logs; classifies the verified exception logs to obtain exception categories and exception cause categories; generates processing suggestion texts according to the exception categories and the exception cause categories and issues early warning prompts.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. The present invention extracts multi-source heterogeneous features from log text sequences, log timestamp sequences, and operating status indicator sequences. Semantic feature vector sequences are extracted from log text sequences to capture deep semantic information of log content; temporal features are extracted from log timestamp sequences to reflect the temporal regularity and abnormal patterns of log generation; and state features are extracted from the set of operating status indicator sequences to reflect changes in the operating status of the system. A hierarchical graph neural network is then introduced to further process the fused multi-source heterogeneous features, and a graph attention mechanism is applied at the message layer, session layer, and system layer to realize feature propagation and updating, and to realize information interaction between layers, which can more comprehensively capture the complex associations between data, improve the ability to recognize uncommon normal log data, and thus improve the accuracy of anomaly detection.

[0055] 2. The present invention provides an adaptive anomaly detection method. First, a standardized pattern vector is established using conditional entropy and entropy correlation coefficient to capture the variability and uncertainty of the data; then, the log timestamp sequence is segmented by detecting the mutation time points in the standardized pattern vector sequence, and in each segment, a linear model is fitted to dynamically model normal behavior, and an adaptive anomaly threshold is calculated. According to the anomaly score sequence and the adaptive threshold, candidate anomaly logs are detected, which fully considers the pattern changes in the local time window, can dynamically adjust the threshold, adapt to the normal changes of the data, reduce the false detection rate and improve the accuracy of anomaly detection.

[0056] 3. The present invention introduces an anomaly verification processing network after anomaly detection to further verify and classify candidate anomaly logs. First, the anomaly score deviation, pattern deviation, and temporal correlation within the time window of the candidate anomaly log are calculated to form a comprehensive verification score. When the comprehensive verification score exceeds the threshold, the candidate anomaly log is confirmed as a verified anomaly log, and possible false positives are filtered out; then, multiple features of the verified anomaly log are extracted, an anomaly feature vector is constructed, and the anomaly feature vector is processed by a multi-layer perceptron to classify the category and cause of the anomaly, and a processing suggestion text is generated based on the classification result and an early warning prompt is given. Through a sophisticated verification and classification process, false positives are further reduced, and the efficiency of anomaly processing and the accuracy of anomaly detection are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of a deep learning-based anomaly detection and processing method flow chart provided in an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of the structure of an anomaly detection and processing system based on deep learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] Embodiment 1

[0061] The microservice architecture system of an Internet company includes core business modules such as payment services, order services, and user services, and processes more than 100 million requests per day. During the operation of the system, a large amount of log data and operation status indicator data will be generated, which reflects the operation status of the system. In order to ensure the stable operation of the system, it is necessary to detect and handle system anomalies in a timely manner. Figure 1 As shown, the anomaly detection and processing method based on deep learning of the present invention realizes anomaly detection and processing of system logs, including:

[0062] Obtain a log text sequence, a log timestamp sequence, and a set of operating status indicator sequences at the corresponding time; the operating status indicator sequence set includes A sequence of operating status indicators;

[0063] Constructing a feature extraction network, extracting a semantic feature vector sequence according to the log text sequence, extracting a time series feature vector sequence according to the log timestamp sequence, and extracting a state feature vector sequence according to the operating state indicator sequence set; fusing the semantic feature vector sequence, the time series feature vector sequence, and the state feature vector sequence to obtain an initial feature vector sequence;

[0064] Constructing a hierarchical graph neural network to perform feature update, propagation and fusion on the initial feature vector sequence to obtain a multi-level fusion feature vector sequence; the hierarchical graph neural network includes a message layer, a session layer and a system layer;

[0065] Construct a candidate anomaly detection network, calculate a standardized pattern vector sequence according to the log text sequence, the operating status indicator sequence set and the multi-level fusion feature vector sequence; obtain an adaptive anomaly threshold according to the standardized pattern vector sequence and detect candidate anomaly logs;

[0066] An exception verification processing network is constructed to verify the candidate exception logs and determine the verified exception logs; the verified exception logs are classified to obtain exception categories and exception cause categories; a processing suggestion text is generated according to the exception category and the exception cause category and an early warning prompt is given.

[0067] Further, extracting a semantic feature vector sequence according to the log text sequence includes:

[0068] Preprocessing the log text sequence according to the log template matching rule to obtain a standardized log template sequence;

[0069] Convert each standardized log template in the standardized log template sequence into a word vector according to a pre-trained word vector model to obtain a word vector sequence;

[0070] The word vector sequence is processed through a multi-head self-attention mechanism to obtain a semantic feature vector sequence.

[0071] Through the preprocessing of log template matching rules and the conversion of pre-trained word vector models, combined with the multi-head self-attention mechanism, the semantic information in the log text can be effectively captured, the noise impact of the log text can be reduced, and the expressiveness and accuracy of the semantic features can be improved.

[0072] Further, extracting a time series feature vector sequence according to the log timestamp sequence includes:

[0073] Set a time window for each log timestamp ; Indicates the number of timestamps in the log sequence. timestamp of the log, represents the step size of the time window;

[0074] Calculate the time interval sequence within each time window according to the log timestamp sequence;

[0075] Calculating the time series statistical features of each log timestamp according to the time interval sequence to obtain a time series statistical feature vector; the time series statistical features include time interval mean, time interval standard deviation, time interval skewness and time interval kurtosis;

[0076] Analyzing the periodic characteristics of the log timestamp sequence by fast Fourier transform to obtain a periodic characteristic vector of each log timestamp; the periodic characteristic vector includes a plurality of main frequency components;

[0077] Analyzing the trend characteristics of the time interval sequence by exponential smoothing to obtain a trend characteristic vector of each log timestamp; the trend characteristic vector includes a smoothing value, a smoothing value change amount and a smoothing value change rate;

[0078] The time series statistical feature vector, the period feature vector and the trend feature vector are concatenated to obtain a time series feature vector;

[0079] The time series feature vectors of all the log timestamps are calculated to obtain the time series feature vector sequence.

[0080] Using time windows to extract time series statistical features, periodic features and trend features, and extracting time series features from log timestamp sequences helps capture the temporal correlation and changing trends of log events and improve the ability to detect periodic and sudden anomalies; integrating multiple time series features enables the model to more comprehensively reflect log time information.

[0081] Further, extracting a state feature vector sequence according to the operating state indicator sequence set includes: calculating The state statistical characteristics and state change characteristics of the operating status indicators in each time window are obtained. The initial state feature vector; The initial state feature vectors are spliced ​​in dimension to obtain the state feature vector; the state feature vectors in all the time windows are calculated to obtain the state feature vector sequence; the state statistical features include the state indicator mean and the state indicator standard deviation; the state change features include the state indicator change rate and the state indicator change acceleration.

[0082] By calculating the state statistical characteristics and state change characteristics and splicing them in dimensions, a comprehensive characterization of the operating state indicators is achieved, which can effectively reflect the static characteristics and dynamic change characteristics of the system operating state and enhance the integrity of the state characteristics.

[0083] Furthermore, fusing the semantic feature vector sequence, the temporal feature vector sequence and the state feature vector sequence comprises:

[0084] The semantic feature vector sequence, the temporal feature vector sequence and the state feature vector sequence are standardized and concatenated in the vector dimension to obtain a standardized feature vector sequence;

[0085] An adaptive weight vector is generated according to the standardized feature vector sequence, and the standardized feature vector sequence and the adaptive weight vector are weightedly fused to obtain the initial feature vector sequence.

[0086] Through standardization processing and adaptive weight fusion mechanism, effective fusion of different types of features is achieved, avoiding the influence of feature scale differences. At the same time, the importance of different features is adjusted by adaptive weights, which improves the representativeness of the fused features.

[0087] Furthermore, the hierarchical graph neural network includes:

[0088] A message node set and a message edge set are constructed according to the log text sequence and the initial feature vector sequence to obtain a message layer graph; the initial feature of each message node is the initial feature vector corresponding to each log text; the message edge set includes a temporal association edge set and a semantic association edge set;

[0089] The message node set is divided to obtain a session node set, and a session edge set is constructed according to the log text sequence to obtain a session layer graph; the initial feature of each session node is a weighted average of the initial features of the message node corresponding to the session node; the session edge set includes a time-dependent edge set and a function-dependent edge set;

[0090] Obtaining system component information to construct a system node set and a system edge set, and obtaining a system layer graph; each system node represents a system component, and the system node initial feature is an aggregate feature of all the session node initial features related to the system component; the system edge set includes a call relationship edge set and a data dependency edge set;

[0091] A graph attention mechanism is applied to the message layer, the session layer and the system layer to perform feature propagation and update respectively, so as to obtain a message node feature sequence, a session node feature sequence and a system node feature sequence; a message session mapping matrix and a session system mapping matrix are constructed, and feature fusion between different graphs is performed on the message node features, the session node features and the system node features to obtain the multi-level fused feature sequence.

[0092] A three-layer graph structure design is adopted to model the relationship between nodes at different levels. Through the graph attention mechanism and inter-layer feature fusion, the full utilization of information at all levels of the system is achieved, and the expressiveness and interpretability of features are enhanced.

[0093] Further, calculating a standardized pattern vector sequence according to the log text sequence, the operating status indicator sequence set and the multi-level fusion feature vector sequence includes:

[0094] Set the first scale time window for each log timestamp and the second scale time window ; Indicates the number of timestamps in the log sequence. Log timestamp, and Respectively represent the step size of the first scale time window and the second scale time window;

[0095] In each of the first scale time window and the second scale time window, log statistical features, state statistical features and pattern features are extracted and spliced ​​to obtain a first window feature vector and a second window feature vector of each log timestamp; the log statistical features include log quantity distribution and keyword frequency; the pattern features are aggregated features of multi-level fusion features;

[0096] Conditional entropy is calculated according to the first window feature vector and the second window feature vector respectively to obtain a first conditional entropy and a second conditional entropy, and an entropy correlation coefficient between the first conditional entropy and the second conditional entropy is calculated; a standardized pattern vector is established according to the first conditional entropy, the second conditional entropy and the entropy correlation coefficient; the standardized pattern vector of all the log timestamps is calculated to obtain a standardized pattern vector sequence.

[0097] Further, calculating the first conditional entropy, the second conditional entropy and the entropy correlation coefficient includes:

[0098] Select from the eigenvector space of the first window eigenvector The features of the first dimension are used as the first conditional features, and the features of the remaining dimensions are used as the first behavioral features. The first conditional features of all log timestamps are clustered to obtain The first condition category is clustered for the first behavior features of all log timestamps, and we get First behavior category;

[0099] Select the same feature vector from the feature vector space of the second window feature vector The features of the first dimension are used as the second conditional features, and the features of the remaining dimensions are used as the second behavioral features. The second conditional features of all log timestamps are clustered to obtain The second condition category is clustered for the second behavior features of all log timestamps, and we get a second behavior category;

[0100] The calculation formulas for the first conditional entropy and the second conditional entropy are as follows: in, and Respectively represent Log timestamp The first conditional entropy and the second conditional entropy of; and Respectively represent The first behavioral category and the a second behavior category; and Respectively represent The first condition category and the Second condition category; express of In the first window feature vector of the nearest neighbor log timestamp, and the number of express The first window feature vectors belong to the number of Indicates the log timestamp of Among the second window feature vectors of the nearest neighbor log timestamps, and the number of express The second window feature vector belongs to the number of

[0101] Further, calculating the entropy correlation coefficient between the first conditional entropy and the second conditional entropy includes: extracting the log timestamp The first conditional entropy sequence and the second conditional entropy sequence in the third scale time window are used to calculate the Pearson correlation coefficient of the first conditional entropy sequence and the second conditional entropy sequence to obtain the entropy correlation coefficient; the step size of the third scale time window is and The minimum value in .

[0102] Feature extraction and conditional entropy analysis based on dual-scale time windows, combined with the aggregation of multi-level fusion features, effectively characterizes the changing characteristics of the system behavior pattern and improves the expressiveness and discrimination of the standardized pattern vector; the calculation of the entropy correlation coefficient enhances the ability to capture pattern consistency and changing laws.

[0103] Further, obtaining an adaptive anomaly threshold according to the standardized pattern vector sequence and detecting candidate anomaly logs includes:

[0104] Detecting a mutation time point according to the standardized pattern vector sequence, and segmenting the log timestamp sequence according to the mutation time point to obtain a plurality of segmented timestamp sequences;

[0105] Calculate the log anomaly score at each log timestamp according to the standardized pattern vector sequence to obtain an anomaly score sequence;

[0106] Fitting a linear model in each segmented timestamp sequence according to the standardized pattern vector sequence and the anomaly score sequence to obtain a piecewise linear model;

[0107] Calculating an adaptive anomaly threshold for each of the log timestamps according to the piecewise linear model;

[0108] The candidate anomaly log is detected according to the anomaly score sequence and the adaptive anomaly threshold.

[0109] Further, calculating the log anomaly score at each log timestamp according to the standardized pattern vector sequence includes: in, Indicates Log timestamp The log anomaly score of and Respectively expressed in Log timestamp The conditional entropy deviation and contextual abnormality of and represents the weight coefficient; Indicates the log timestamp The first conditional entropy mean within the first scale time window; Indicates the log timestamp The second conditional entropy mean within the second scale time window; and They represent the means of the first conditional entropy and the second conditional entropy of history respectively; and They represent the standard deviations of the historical first conditional entropy and the second conditional entropy respectively; Indicates the log timestamp The normalized pattern vector of ; and denote the mean and standard deviation of the historical pattern vector respectively; represents the Euclidean norm.

[0110] Further, calculating the adaptive anomaly threshold of each log timestamp according to the piecewise linear model includes: in, Indicates Log timestamp Adaptive anomaly threshold; Represents a piecewise linear model in The value of and represents the adaptive parameters; Indicates the standard deviation of the anomaly score of the current segment timestamp sequence; Indicates the anomaly score sequence at the log timestamp The rate of change, , Indicates the log timestamp forward The mean anomaly score for each log timestamp.

[0111] Table 1 Examples of anomaly scores and adaptive anomaly thresholds

[0112] Log timestamp Anomaly score Adaptive Threshold Whether to select exception log 10:00:12.342 0.92 0.90 yes 10:00:15.761 0.70 0.75 no 10:01:03.204 1.10 0.95 yes 10:03:27.119 0.62 0.70 no

[0113] Table 1 shows examples of anomaly scores and adaptive anomaly thresholds provided in an embodiment of the present application. Logs with anomaly scores higher than the adaptive threshold are marked as candidate anomaly logs.

[0114] Based on the standardized pattern vector sequence, the adaptive anomaly threshold and piecewise linear model are calculated to detect candidate anomaly logs. The adaptive threshold reduces the error caused by the fixed threshold and improves the detection flexibility. The piecewise linear model enhances the explanatory power of the anomaly score and supports more accurate anomaly detection.

[0115] Furthermore, the abnormal verification processing network includes:

[0116] Calculate the anomaly score deviation of the candidate anomaly log according to the anomaly score sequence and the piecewise linear model; calculate the pattern deviation of the candidate anomaly log according to the standardized pattern vector sequence; extract the score change sequence and the pattern vector change sequence according to the anomaly score sequence and the standardized pattern vector sequence within the time window corresponding to the candidate anomaly log, and calculate the time series correlation of the score change sequence and the pattern vector change sequence; calculate the comprehensive verification score according to the anomaly score deviation, the pattern deviation and the time series correlation; when the comprehensive verification score is greater than the verification threshold, determine the candidate anomaly log as the verified anomaly log;

[0117] The score deviation, the pattern deviation, the time series correlation, the initial feature vector and the multi-level fusion feature vector of the verified exception log are extracted to construct an exception feature vector; the exception feature vector is processed by a first multi-layer perceptron to obtain an exception category; the exception feature vector is processed by a second multi-layer perceptron to obtain an exception cause category; a historical processing experience library is matched according to the exception category and the exception cause category, a processing suggestion text is output, and an early warning prompt is issued.

[0118] An anomaly verification and processing network is built to verify candidate anomaly logs based on comprehensive verification scores, reducing the false alarm rate and ensuring the reliability of anomaly detection results. Anomaly categories and anomaly cause categories are classified through a multi-layer perceptron, and processing suggestion texts are generated, improving system maintenance efficiency.

[0119] The anomaly detection performance of the method of the present invention is shown in Table 2. The precision and recall rates can reach 89.4% and 90.8%, and the false positive rate is only 2.3%, which proves that the anomaly detection method of the present invention can achieve high-precision anomaly detection.

[0120] Table 2 Anomaly detection performance

[0121] index value Accuracy 89.4% Recall 90.8% False positive rate 2.3%

[0122] The anomaly detection and processing method provided by the present invention obtains a set of log text sequences, timestamp sequences and operating status indicator sequences, comprehensively considers log semantic features, time series features and operating status features, and multi-source information fusion improves the detection effect; a hierarchical graph neural network is used to realize the fusion of cross-level features, which significantly enhances the feature expression capability and the ability to capture anomalies in complex systems; an adaptive anomaly threshold and verification mechanism improve the adaptability and robustness of anomaly detection, effectively reduces the false alarm rate, and thus improves the accuracy of anomaly detection; finally, through anomaly classification and cause analysis, explainability and processing suggestions are provided, which improves practicality and provides strong support for system maintenance and optimization.

[0123] Embodiment 2

[0124] like Figure 2 As shown, the present invention also provides an anomaly detection and processing system based on deep learning, and applies an anomaly detection and processing method based on deep learning, including:

[0125] The data acquisition unit acquires a log text sequence, a log timestamp sequence, and a set of operation status indicator sequences at corresponding times; the set of operation status indicator sequences includes A sequence of operating status indicators;

[0126] A feature extraction unit extracts a semantic feature vector sequence according to the log text sequence, extracts a time series feature vector sequence according to the log timestamp sequence, and extracts a state feature vector sequence according to the operating state indicator sequence set; and fuses the semantic feature vector sequence, the time series feature vector sequence, and the state feature vector sequence to obtain an initial feature vector sequence;

[0127] A hierarchical graph unit constructs a hierarchical graph neural network to perform feature update, propagation and fusion on the initial feature vector sequence to obtain a multi-level fusion feature vector sequence; the hierarchical graph neural network includes a message layer, a session layer and a system layer;

[0128] A candidate anomaly detection unit calculates a standardized pattern vector sequence according to the log text sequence, the operating status indicator sequence set and the multi-level fusion feature vector sequence; obtains an adaptive anomaly threshold according to the standardized pattern vector sequence and detects candidate anomaly logs;

[0129] The exception verification processing unit verifies the candidate exception logs to determine the verified exception logs; classifies the verified exception logs to obtain exception categories and exception cause categories; generates processing suggestion texts according to the exception categories and the exception cause categories and issues early warning prompts.

[0130] Table 3 shows the ablation experiment results of the present invention, which verifies the necessity of multi-source features. And from Table 2, it can be seen that the hierarchical graph neural network can bring significant performance improvement to the anomaly detection and processing system.

[0131] Table 3 Ablation experiment results

[0132] Model configuration Accuracy Recall Complete Model 89.4% 90.8% No semantic features 86.2% 85.5% No timing characteristics 87.5% 84.2% Stateless Features 88.1% 85.8% Non-hierarchical graph neural network 83.9% 80.4%

[0133] Further, extracting a semantic feature vector sequence according to the log text sequence includes:

[0134] Preprocessing the log text sequence according to the log template matching rule to obtain a standardized log template sequence;

[0135] Convert each standardized log template in the standardized log template sequence into a word vector according to a pre-trained word vector model to obtain a word vector sequence;

[0136] The word vector sequence is processed through a multi-head self-attention mechanism to obtain a semantic feature vector sequence.

[0137] Further, extracting a time series feature vector sequence according to the log timestamp sequence includes:

[0138] Set a time window for each log timestamp Indicates the number of timestamps in the log sequence. timestamp of the log, represents the step size of the time window;

[0139] Calculate the time interval sequence within each time window according to the log timestamp sequence;

[0140] Calculating the time series statistical features of each log timestamp according to the time interval sequence to obtain a time series statistical feature vector; the time series statistical features include time interval mean, time interval standard deviation, time interval skewness and time interval kurtosis;

[0141] Analyzing the periodic characteristics of the log timestamp sequence by fast Fourier transform to obtain a periodic characteristic vector of each log timestamp; the periodic characteristic vector includes a plurality of main frequency components;

[0142] Analyzing the trend characteristics of the time interval sequence by exponential smoothing to obtain a trend characteristic vector of each log timestamp; the trend characteristic vector includes a smoothing value, a smoothing value change amount and a smoothing value change rate;

[0143] The time series statistical feature vector, the period feature vector and the trend feature vector are concatenated to obtain a time series feature vector;

[0144] The time series feature vectors of all the log timestamps are calculated to obtain the time series feature vector sequence.

[0145] Further, extracting a state feature vector sequence according to the operating state indicator sequence set includes: calculating The state statistical characteristics and state change characteristics of the operating status indicators in each time window are obtained. The initial state feature vector; The initial state feature vectors are spliced ​​in dimension to obtain the state feature vector; the state feature vectors in all the time windows are calculated to obtain the state feature vector sequence; the state statistical features include the state indicator mean and the state indicator standard deviation; the state change features include the state indicator change rate and the state indicator change acceleration.

[0146] Furthermore, fusing the semantic feature vector sequence, the temporal feature vector sequence and the state feature vector sequence comprises:

[0147] The semantic feature vector sequence, the temporal feature vector sequence and the state feature vector sequence are standardized and concatenated in the vector dimension to obtain a standardized feature vector sequence;

[0148] An adaptive weight vector is generated according to the standardized feature vector sequence, and the standardized feature vector sequence and the adaptive weight vector are weightedly fused to obtain the initial feature vector sequence.

[0149] Furthermore, the hierarchical graph neural network includes:

[0150] A message node set and a message edge set are constructed according to the log text sequence and the initial feature vector sequence to obtain a message layer graph; the initial feature of each message node is the initial feature vector corresponding to each log text; the message edge set includes a temporal association edge set and a semantic association edge set;

[0151] The message node set is divided to obtain a session node set, and a session edge set is constructed according to the log text sequence to obtain a session layer graph; the initial feature of each session node is a weighted average of the initial features of the message node corresponding to the session node; the session edge set includes a time-dependent edge set and a function-dependent edge set;

[0152] Obtaining system component information to construct a system node set and a system edge set, and obtaining a system layer graph; each system node represents a system component, and the system node initial feature is an aggregate feature of all the session node initial features related to the system component; the system edge set includes a call relationship edge set and a data dependency edge set;

[0153] A graph attention mechanism is applied to the message layer, the session layer and the system layer to perform feature propagation and update respectively, so as to obtain a message node feature sequence, a session node feature sequence and a system node feature sequence; a message session mapping matrix and a session system mapping matrix are constructed, and feature fusion between different graphs is performed on the message node features, the session node features and the system node features to obtain the multi-level fused feature sequence.

[0154] Further, calculating a standardized pattern vector sequence according to the log text sequence, the operating status indicator sequence set and the multi-level fusion feature vector sequence includes:

[0155] Set the first scale time window for each log timestamp and the second scale time window Indicates the number of timestamps in the log sequence. Log timestamp, and Respectively represent the step size of the first scale time window and the second scale time window; in each of the first scale time window and the second scale time window, extract log statistical features, state statistical features and pattern features and splice them to obtain the first window feature vector and the second window feature vector of each log timestamp; the log statistical features include log quantity distribution and keyword frequency; the pattern features are aggregated features of multi-level fusion features;

[0156] Conditional entropy is calculated according to the first window feature vector and the second window feature vector respectively to obtain a first conditional entropy and a second conditional entropy, and an entropy correlation coefficient between the first conditional entropy and the second conditional entropy is calculated; a standardized pattern vector is established according to the first conditional entropy, the second conditional entropy and the entropy correlation coefficient; the standardized pattern vector of all the log timestamps is calculated to obtain a standardized pattern vector sequence.

[0157] Further, obtaining an adaptive anomaly threshold according to the standardized pattern vector sequence and detecting candidate anomaly logs includes:

[0158] Detecting a mutation time point according to the standardized pattern vector sequence, and segmenting the log timestamp sequence according to the mutation time point to obtain a plurality of segmented timestamp sequences;

[0159] Calculate the log anomaly score at each log timestamp according to the standardized pattern vector sequence to obtain an anomaly score sequence;

[0160] Fitting a linear model in each segmented timestamp sequence according to the standardized pattern vector sequence and the anomaly score sequence to obtain a piecewise linear model;

[0161] Calculating an adaptive anomaly threshold for each of the log timestamps according to the piecewise linear model;

[0162] The candidate anomaly log is detected according to the anomaly score sequence and the adaptive anomaly threshold.

[0163] Furthermore, the abnormal verification processing network includes:

[0164] Calculate the anomaly score deviation of the candidate anomaly log according to the anomaly score sequence and the piecewise linear model; calculate the pattern deviation of the candidate anomaly log according to the standardized pattern vector sequence; extract the score change sequence and the pattern vector change sequence according to the anomaly score sequence and the standardized pattern vector sequence within the time window corresponding to the candidate anomaly log, and calculate the time series correlation of the score change sequence and the pattern vector change sequence; calculate the comprehensive verification score according to the anomaly score deviation, the pattern deviation and the time series correlation; when the comprehensive verification score is greater than the verification threshold, determine the candidate anomaly log as the verified anomaly log;

[0165] The score deviation, the pattern deviation, the time series correlation, the initial feature vector and the multi-level fusion feature vector of the verified exception log are extracted to construct an exception feature vector; the exception feature vector is processed by a first multi-layer perceptron to obtain an exception category; the exception feature vector is processed by a second multi-layer perceptron to obtain an exception cause category; a historical processing experience library is matched according to the exception category and the exception cause category, a processing suggestion text is output, and an early warning prompt is issued.

[0166] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. Anomaly detection and processing method based on deep learning, characterized in that: include: Get the log text sequence, log timestamp sequence and the corresponding time operation status indicator sequence set; The operating status indicator sequence set includes A sequence of operating status indicators; Constructing a feature extraction network, extracting a semantic feature vector sequence according to the log text sequence, extracting a time series feature vector sequence according to the log timestamp sequence, and extracting a state feature vector sequence according to the operating state indicator sequence set; fusing the semantic feature vector sequence, the time series feature vector sequence, and the state feature vector sequence to obtain an initial feature vector sequence; Constructing a hierarchical graph neural network to perform feature update, propagation and fusion on the initial feature vector sequence to obtain a multi-level fusion feature vector sequence; the hierarchical graph neural network includes a message layer, a session layer and a system layer; Construct a candidate anomaly detection network, calculate a standardized pattern vector sequence according to the log text sequence, the operating status indicator sequence set and the multi-level fusion feature vector sequence; obtain an adaptive anomaly threshold according to the standardized pattern vector sequence and detect candidate anomaly logs; Calculating the standardized pattern vector sequence includes: setting a first scale time window and a second scale time window for each log timestamp; extracting a first window feature vector and a second window feature vector of each log timestamp; calculating conditional entropy according to the first window feature vector and the second window feature vector respectively to obtain a first conditional entropy and a second conditional entropy, and calculating an entropy correlation coefficient between the first conditional entropy and the second conditional entropy; establishing a standardized pattern vector according to the first conditional entropy, the second conditional entropy and the entropy correlation coefficient; calculating the standardized pattern vectors of all the log timestamps to obtain the standardized pattern vector sequence; An exception verification processing network is constructed to verify the candidate exception logs and determine the verified exception logs; the verified exception logs are classified to obtain exception categories and exception cause categories; a processing suggestion text is generated according to the exception category and the exception cause category and an early warning prompt is given.

2. The deep learning-based anomaly detection and processing method according to claim 1, characterized in that: Extracting a semantic feature vector sequence according to the log text sequence includes: Preprocessing the log text sequence according to the log template matching rule to obtain a standardized log template sequence; Convert each standardized log template in the standardized log template sequence into a word vector according to a pre-trained word vector model to obtain a word vector sequence; The word vector sequence is processed through a multi-head self-attention mechanism to obtain a semantic feature vector sequence.

3. The deep learning-based anomaly detection and processing method according to claim 1, characterized in that: Extracting a time series feature vector sequence according to the log timestamp sequence includes: Set a time window for each log timestamp ; Indicates the number of timestamps in the log sequence. timestamp of the log, represents the step size of the time window; Calculate the time interval sequence within each time window according to the log timestamp sequence; Calculating the time series statistical features of each log timestamp according to the time interval sequence to obtain a time series statistical feature vector; the time series statistical features include time interval mean, time interval standard deviation, time interval skewness and time interval kurtosis; Analyzing the periodic characteristics of the log timestamp sequence by fast Fourier transform to obtain a periodic characteristic vector of each log timestamp; the periodic characteristic vector includes a plurality of main frequency components; Analyzing the trend characteristics of the time interval sequence by exponential smoothing to obtain a trend characteristic vector of each log timestamp; the trend characteristic vector includes a smoothing value, a smoothing value change amount and a smoothing value change rate; The time series statistical feature vector, the period feature vector and the trend feature vector are concatenated to obtain a time series feature vector; The time series feature vectors of all the log timestamps are calculated to obtain the time series feature vector sequence.

4. The deep learning-based anomaly detection and processing method according to claim 1, characterized in that: Extracting a state feature vector sequence according to the operating state indicator sequence set includes: calculating The state statistical characteristics and state change characteristics of the operating status indicators in each time window are obtained. The initial state feature vector; The initial state feature vectors are spliced ​​in dimension to obtain the state feature vector; the state feature vectors in all the time windows are calculated to obtain the state feature vector sequence; the state statistical features include the state indicator mean and the state indicator standard deviation; the state change features include the state indicator change rate and the state indicator change acceleration.

5. The deep learning-based anomaly detection and processing method according to claim 1, characterized in that: The fusing of the semantic feature vector sequence, the temporal feature vector sequence and the state feature vector sequence comprises: The semantic feature vector sequence, the temporal feature vector sequence and the state feature vector sequence are standardized and concatenated in the vector dimension to obtain a standardized feature vector sequence; An adaptive weight vector is generated according to the standardized feature vector sequence, and the standardized feature vector sequence and the adaptive weight vector are weightedly fused to obtain the initial feature vector sequence.

6. The deep learning-based anomaly detection and processing method according to claim 1, characterized in that: The hierarchical graph neural network comprises: A message node set and a message edge set are constructed according to the log text sequence and the initial feature vector sequence to obtain a message layer graph; the initial feature of each message node is the initial feature vector corresponding to each log text; the message edge set includes a temporal association edge set and a semantic association edge set; The message node set is divided to obtain a session node set, and a session edge set is constructed according to the log text sequence to obtain a session layer graph; the initial feature of each session node is a weighted average of the initial features of the message node corresponding to the session node; the session edge set includes a time-dependent edge set and a function-dependent edge set; Obtaining system component information to construct a system node set and a system edge set, and obtaining a system layer graph; each system node represents a system component, and the system node initial feature is an aggregate feature of all the session node initial features related to the system component; the system edge set includes a call relationship edge set and a data dependency edge set; A graph attention mechanism is applied to the message layer, the session layer and the system layer to perform feature propagation and update respectively, so as to obtain a message node feature sequence, a session node feature sequence and a system node feature sequence; a message session mapping matrix and a session system mapping matrix are constructed, and feature fusion between different graphs is performed on the message node features, the session node features and the system node features to obtain the multi-level fused feature vector sequence.

7. The deep learning-based anomaly detection and processing method according to claim 1, characterized in that: Calculating a standardized pattern vector sequence according to the log text sequence, the operating status indicator sequence set and the multi-level fusion feature vector sequence includes: Set the first scale time window for each log timestamp and the second scale time window ; Indicates the number of timestamps in the log sequence. Log timestamp, and Respectively represent the step size of the first scale time window and the second scale time window; In each of the first scale time window and the second scale time window, log statistical features, state statistical features and pattern features are extracted and spliced ​​to obtain a first window feature vector and a second window feature vector of each log timestamp; the log statistical features include log quantity distribution and keyword frequency; the pattern features are aggregated features of multi-level fusion features; Conditional entropy is calculated according to the first window feature vector and the second window feature vector respectively to obtain a first conditional entropy and a second conditional entropy, and an entropy correlation coefficient between the first conditional entropy and the second conditional entropy is calculated; a standardized pattern vector is established according to the first conditional entropy, the second conditional entropy and the entropy correlation coefficient; the standardized pattern vector of all the log timestamps is calculated to obtain a standardized pattern vector sequence.

8. The deep learning-based anomaly detection and processing method according to claim 1, characterized in that: Obtaining an adaptive anomaly threshold according to the standardized pattern vector sequence and detecting candidate anomaly logs includes: Detecting a mutation time point according to the standardized pattern vector sequence, and segmenting the log timestamp sequence according to the mutation time point to obtain a plurality of segmented timestamp sequences; Calculate the log anomaly score at each log timestamp according to the standardized pattern vector sequence to obtain an anomaly score sequence; Fitting a linear model in each segmented timestamp sequence according to the standardized pattern vector sequence and the anomaly score sequence to obtain a piecewise linear model; Calculating an adaptive anomaly threshold for each of the log timestamps according to the piecewise linear model; The candidate anomaly log is detected according to the anomaly score sequence and the adaptive anomaly threshold.

9. The deep learning-based anomaly detection and processing method according to claim 1, characterized in that: The abnormal verification processing network includes: Calculate the anomaly score deviation of the candidate anomaly log according to the anomaly score sequence and the piecewise linear model; calculate the pattern deviation of the candidate anomaly log according to the standardized pattern vector sequence; extract the score change sequence and the pattern vector change sequence according to the anomaly score sequence and the standardized pattern vector sequence within the time window corresponding to the candidate anomaly log, and calculate the time series correlation of the score change sequence and the pattern vector change sequence; calculate the comprehensive verification score according to the anomaly score deviation, the pattern deviation and the time series correlation; when the comprehensive verification score is greater than the verification threshold, determine the candidate anomaly log as the verified anomaly log; The score deviation, the pattern deviation, the time series correlation, the initial feature vector and the multi-level fusion feature vector of the verified exception log are extracted to construct an exception feature vector; the exception feature vector is processed by a first multi-layer perceptron to obtain an exception category; the exception feature vector is processed by a second multi-layer perceptron to obtain an exception cause category; a historical processing experience library is matched according to the exception category and the exception cause category, a processing suggestion text is output, and an early warning prompt is issued.

10. Anomaly detection and processing system based on deep learning, characterized in that: include: A data acquisition unit, which acquires a log text sequence, a log timestamp sequence, and a set of operation status indicator sequences at corresponding times; The operating status indicator sequence set includes A sequence of operating status indicators; A feature extraction unit extracts a semantic feature vector sequence according to the log text sequence, extracts a time series feature vector sequence according to the log timestamp sequence, and extracts a state feature vector sequence according to the operating state indicator sequence set; and fuses the semantic feature vector sequence, the time series feature vector sequence, and the state feature vector sequence to obtain an initial feature vector sequence; A hierarchical graph unit constructs a hierarchical graph neural network to perform feature update, propagation and fusion on the initial feature vector sequence to obtain a multi-level fusion feature vector sequence; the hierarchical graph neural network includes a message layer, a session layer and a system layer; A candidate anomaly detection unit calculates a standardized pattern vector sequence according to the log text sequence, the operating status indicator sequence set and the multi-level fusion feature vector sequence; obtains an adaptive anomaly threshold according to the standardized pattern vector sequence and detects candidate anomaly logs; Calculating the standardized pattern vector sequence includes: setting a first scale time window and a second scale time window for each log timestamp; Extracting the first window feature vector and the second window feature vector of each log timestamp; calculating conditional entropy according to the first window feature vector and the second window feature vector respectively to obtain the first conditional entropy and the second conditional entropy, and calculating the entropy correlation coefficient between the first conditional entropy and the second conditional entropy; establishing a standardized pattern vector according to the first conditional entropy, the second conditional entropy and the entropy correlation coefficient; calculating the standardized pattern vectors of all the log timestamps to obtain the standardized pattern vector sequence; The exception verification processing unit verifies the candidate exception logs to determine the verified exception logs; classifies the verified exception logs to obtain exception categories and exception cause categories; generates processing suggestion texts according to the exception categories and the exception cause categories and issues early warning prompts.

Images