Network management log anomaly detection method and device, equipment and storage medium
By preprocessing and dimensionality reduction of network management log data, Drain algorithm and PCA principal component analysis combined with bidirectional LSTM model, the network management system abnormalities are quickly identified, which improves identification efficiency and accuracy and reduces labor costs.
Patent Information
- Application Number
- CN202510397665.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-01
- Publication Date
- 2025-07-04
AI Technical Summary
In the prior art, network management logs are complex and abnormalities cannot be quickly identified, resulting in low abnormal analysis efficiency and accuracy of network management system.
By preprocessing and dimensionality reduction of the original network management log data, using Drain algorithm analysis and One-Hot encoding, combined with PCA principal component analysis and bidirectional LSTM model for evaluation, we quickly identify abnormal network management logs.
It improves the recognition efficiency of abnormal network management logs, reduces labor costs, and has high reusability. It only requires standardizing the input vector to detect subsequent logs, solving the problem of low recognition efficiency and accuracy in the prior art.
Smart Images
Figure CN120263630A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of communication network management and operation and maintenance, and particularly to a method, device, equipment and storage medium for detecting abnormal network management logs. Background Art
[0002] Currently, during the daily maintenance of network management and operation and maintenance, complex network management systems may be abnormal due to various factors, and this abnormality may cause some functions of the network management system to fail to operate normally.
[0003] In the prior art, operation and maintenance personnel analyze the abnormalities of the network management system by using the running states and important events of the network management system recorded in the network management logs at various critical moments. However, this method cannot quickly identify abnormal network management logs due to the large number and complexity of the network management logs, resulting in low efficiency and accuracy in analyzing the abnormalities of the network management system. Therefore, there is an urgent need for a technology that can automatically detect abnormalities in the network management system. Summary of the Invention
[0004] The present invention provides a method, device, equipment and storage medium for detecting abnormal network management logs, which can solve the technical problems in the prior art that abnormal network management logs cannot be quickly identified, resulting in low efficiency and accuracy in analyzing the abnormalities of the network management system.
[0005] To achieve the above object, the present invention provides the following technical solutions:
[0006] In a first aspect, an embodiment of the present invention provides a method for detecting abnormal network management logs, the method including:
[0007] Preprocess the original network management log data to obtain the target network management log data after preprocessing;
[0008] Reduce the dimension of the target network management log data to obtain the target network management log data after dimension reduction;
[0009] Evaluate the target network management log data after dimension reduction through a log detection model to obtain an evaluation result of the network management log data.
[0010] Optionally, the step of preprocessing the original network management log data to obtain the target network management log data after preprocessing includes:
[0011] Parse the original network management log data through the Drain algorithm to obtain the target network management log data after parsing;
[0012] Perform One-Hot encoding on the target network management log data after parsing to obtain the target network management log data after preprocessing.
[0013] Optionally, the step of performing dimensionality reduction on the target network management log data to obtain the dimensionally reduced target network management log data includes:
[0014] Arrange the target network management log data to obtain a first matrix;
[0015] Calculate the eigenvalues of the covariance matrix of the first matrix and the eigenvectors corresponding to the eigenvalues;
[0016] Select the largest K eigenvalues among the eigenvalues as the target eigenvalues, arrange the target eigenvectors corresponding to the target eigenvalues by rows to obtain a second matrix;
[0017] Calculate the product of the first matrix and the second matrix, and use the obtained product as the dimensionally reduced target network management log data.
[0018] Optionally, the step of evaluating the dimensionally reduced target network management log data through a log detection model to obtain a network management log data evaluation result includes:
[0019] Divide the dimensionally reduced target network management log data evenly to obtain W sample subset data;
[0020] Select any one of the W sample subset data as the validation set;
[0021] Use the remaining sample subset data as the training set to input into the log detection model to obtain the network management log data evaluation result output by the log detection model;
[0022] Among the W sample subset data, select any one of the sample subset data that has not been used as the validation set as the validation set, and return to execute the step of using the remaining sample subset data as the training set to input into the log detection model to obtain the network management log data evaluation result output by the log detection model;
[0023] Until all W sample subset data have been selected, obtain W network management log data evaluation results;
[0024] Calculate the mean of the W network management log data evaluation results, and use the mean as the final network management log data evaluation result.
[0025] Optionally, the log detection model includes a bidirectional long short-term memory network layer, a fully connected layer, and a classification output layer containing a softmax function.
[0026] Optionally, the step of adjusting the parameters of the log detection model based on the validation set and the grid search method to obtain the log detection model after parameter adjustment includes:
[0027] Based on the validation set, the activation function, the number of hidden layers, and the number of memory units in the log detection model are adjusted by the grid search method to obtain multiple groups of parameter combinations;
[0028] Based on the validation set, the best parameter combination among the multiple groups of parameter combinations is determined;
[0029] Based on the best parameter combination, a log detection model after parameter tuning is obtained.
[0030] In a second aspect, an embodiment of the present invention provides a network management log anomaly detection device, and the device includes:
[0031] A preprocessing module, configured to preprocess the original network management log data to obtain preprocessed target network management log data;
[0032] A data processing module, configured to reduce the dimension of the target network management log data to obtain dimension-reduced target network management log data;
[0033] A data evaluation module, configured to evaluate the dimension-reduced target network management log data through a log detection model to obtain a network management log data evaluation result.
[0034] Optionally, the data processing module is configured to:
[0035] Arrange the target network management log data to obtain a first matrix;
[0036] Calculate the eigenvalues of the covariance matrix of the first matrix and the eigenvectors corresponding to the eigenvalues;
[0037] Select the largest K eigenvalues among the eigenvalues as target eigenvalues, and arrange the target eigenvectors corresponding to the target eigenvalues row by row to obtain a second matrix;
[0038] Calculate the product of the first matrix and the second matrix, and use the obtained product as the dimension-reduced target network management log data.
[0039] In a third aspect, an embodiment of the present invention further provides an electronic device, including: a memory, a processor; the processor is used to read and execute a computer program stored in the memory to implement the steps of the foregoing network management log anomaly detection method.
[0040] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed, the steps of the foregoing network management log anomaly detection method are implemented.
[0041] The beneficial effects brought by the technical solutions provided by the embodiments of the present invention include:
[0042] Using principal component analysis to reduce the dimensionality of the target network management log data, for the characteristic of the diversity of network management log features, the main feature information can be effectively extracted. Then, using the characteristic that the bidirectional LSTM in the log detection model has good recognition ability for complex sequence data, the evaluation result of the network management log data can be obtained quickly. This not only improves the efficiency of identifying abnormal network management logs, reduces the labor cost, but also has high reusability. Only by normalizing the input vector of the input log detection model can subsequent logs be detected, solving the technical problem in the prior art that abnormal network management logs cannot be quickly identified, resulting in low efficiency and accuracy of abnormal analysis of the network management system. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.
[0044] Figure 1 It is a schematic flowchart of an embodiment of the network management log anomaly detection method of the present invention;
[0045] Figure 2 It is a schematic flowchart of the network management log data parsing process of an embodiment of the present invention;
[0046] Figure 3 For Figure 1 It is a detailed flowchart of step S10 in
[0047] Figure 4 For Figure 1 It is a detailed flowchart of step S20 in
[0048] Figure 5 For Figure 1 It is a detailed flowchart of step S30 in
[0049] Figure 6 It is a schematic diagram of the architecture of the log detection model of an embodiment of the present invention;
[0050] Figure 7 It is a schematic flowchart of the process for obtaining the evaluation result of the network management log data of an embodiment of the present invention;
[0051] Figure 8 It is a schematic diagram of the functional modules of an embodiment of the network management log anomaly detection device of the present invention;
[0052] Figure 9 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed implementation manners
[0053] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.
[0054] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.
[0055] In a first aspect, an embodiment of the present invention provides a method for detecting anomalies in network management logs.
[0056] In one embodiment, referring to Figure 1 , Figure 1 is a schematic flowchart of an embodiment of the method for detecting anomalies in network management logs of the present invention. As shown in Figure 1 , the method for detecting anomalies in network management logs includes:
[0057] Step S10: Preprocess the original network management log data to obtain the target network management log data after preprocessing;
[0058] In this embodiment, by preprocessing the original network management log data, the target network management log data after preprocessing is obtained, so as to clean and perform One-Hot encoding on the original network management log data, thereby improving the subsequent data processing speed and the efficiency of detecting anomalies in network management logs.
[0059] In some specific embodiments, referring to Figure 3 , Figure 3 is Figure 1 a detailed flowchart of step S10 in Figure 3 . As shown in
[0060] Step S101: Parse the original network management log data through the Drain algorithm to obtain the parsed target network management log data;
[0061] Step S102: Perform One-Hot encoding on the parsed target network management log data to obtain the target network management log data after preprocessing.
[0062] In this embodiment, the Drain algorithm is an online log parsing method based on a fixed-depth tree, mainly used to parse the original log messages into structured data. The core idea of the Drain algorithm is to accurately identify and classify log messages through preprocessing, length matching, per-token search, and similarity comparison.
[0063] Specifically, the first step: preprocess the messages through professional knowledge;
[0064] Drain allows users to provide simple regular expressions based on professional knowledge representing common variables such as IP addresses and block IDs, and then Drain removes the tokens matched by these regular expressions from the original log messages. The number of tokens refers to the total number of all tokens obtained after tokenization in a text dataset. A token is the basic unit in natural language processing and can be a word, a character, or other substrings.
[0065] The second step: search by log message length;
[0066] Drain starts from the root node of the parse tree with the preprocessed log messages. The first-level nodes of the parse tree distinguish log groups by the length of the log messages (the number of tokens), and Drain selects the path to the first-level nodes based on the log message length of the preprocessed log messages. Based on the assumption that log messages with the same log event may have the same log message length.
[0067] The third step: search by previous tokens;
[0068] Drain traverses from the first-level nodes searched in the second step to the leaf nodes. It should be noted that this step is based on the assumption that the tokens at the start position of the log messages are more likely to be constants.
[0069] Drain selects the next internal node through the tokens at the start position of the log messages. For example, for the log message Receive with node Length = 4, Drain traverses from the first-level node Length = 4 to the second-level node Receive because the token at the first position of the log message is Receive. In this instance, the number of internal nodes is depth - 2, so there are depth - 2 layers that use the first depth - 2 tokens of the log message as the search rules.
[0070] In some cases, a log message may start with a parameter, for example, 120bytes received. Such log messages may cause a branch explosion in the parse tree because each parameter will be encoded in an internal node. To avoid branch explosion, only tokens that do not contain numbers are considered in this step.
[0071] If a token contains a number, it will match a special internal node *. For example, for the above log message "120bytes received", Drain will traverse to the internal node * instead of 120. Additionally, the parameter maxChild limits the maximum number of child nodes of a node. If a node already has maxChild child nodes, then any unmatched tokens will match the special internal node * among all its child nodes.
[0072] Step 4: Search by token similarity;
[0073] Before this step, the Drain algorithm has traversed a leaf node, which contains a list of log groups. The log messages in these log groups follow the rules encoded in the internal nodes of the path.
[0074] In this step, the Drain algorithm selects the most suitable log group from the list of log groups. By calculating the similarity simSeq between the log message and the log events of each log group:
[0075]
[0076] where seq1(i) represents the i-th log message, seq2(i) represents the i-th log event, Seq(i) is the i-th token of the sequence; n is the length of the log message sequence, and the equ function is defined as follows:
[0077]
[0078] After finding the log group with the maximum similarity simSeq, it is compared with a predefined similarity threshold st. If the similarity simSeq is greater than or equal to the similarity threshold st, then the Drain algorithm will return this group as the best match; otherwise, it will return a flag indicating that there is no suitable one.
[0079] Step 5: Update the parse tree:
[0080] If a suitable log group is returned in Step 4, the Drain algorithm will add the log ID of the current log message to the log IDs in the returned log group. Additionally, the log events in the returned log group will be updated.
[0081] Drain scans for tokens at the same positions in the log messages and log events. If the two tokens are the same, the token at that position is not modified. Otherwise, the token at that position is updated with a wildcard * in the log event. If no suitable log group can be found, a new log group is created based on the current log message, where the log ID only contains the ID of the log message and the log event is this log message. Then, Drain updates the parse tree with the new log group.
[0082] As can be seen from the above description of the Drain algorithm, by parsing the original network management log data using the Drain algorithm, the parsed target network management log data can be obtained. Refer to Figure 2 , Figure 2 which is a schematic diagram of the parsing process of network management log data according to an embodiment of the present invention. As Figure 2 shown, root is the root node of the network management log. Assuming the length length of the keyword is 3, and the keywords include: send, block, and receive, then the Drain algorithm filters the events (Log ID) of the network management into two categories, 1 and 2, through the keywords. That is, the preliminary parsing of the network management log is completed.
[0083] Then, the parsed target network management log data is One-Hot encoded to obtain the preprocessed target network management log data. Among them, One-Hot encoding is a process of converting categorical variables into a form that is easy for machine learning algorithms to utilize. One-Hot encoding is the representation of categorical variables as binary vectors. Exemplarily, the network management log events corresponding to the network management log IDs, that is, the parsed target network management log data, are One-Hot encoded to obtain the preprocessed target network management log data as shown in Table 1.
[0084] Table 1
[0085] Network management log ID Network management log event One-hot 1 Invalid*index,m_iThisSubType* [1,0,0,0] 2 Fatel error,EXCP* [0,1,0,0] 3 Drop*alarm,no*alarm [0,0,1,0] 4 Bad pre alarm,*size [0,0,0,1]
[0086] Step S20, perform dimensionality reduction on the target network management log data to obtain the target network management log data after dimensionality reduction;
[0087] In this embodiment, the preprocessed target network management log data is dimensionally reduced by PCA (Principal Component Analysis) to obtain the target network management log data after dimensionality reduction. Among them, PCA is one of the most widely used data dimensionality reduction algorithms. The main idea of PCA is to map n-dimensional features to k dimensions, and these k dimensions are new orthogonal features, also known as principal components, which are reconstructed from the original n-dimensional features.
[0088] In some specific embodiments, refer to Figure 4 ,Figure 4 is Figure 1 a detailed process schematic diagram of step S20 in. As Figure 4 shown, step S20 includes:
[0089] Step S201, arrange the target network management log data to obtain a first matrix;
[0090] Step S202, calculate the eigenvalues of the covariance matrix of the first matrix and the eigenvectors corresponding to the eigenvalues;
[0091] Step S203, select the largest K eigenvalues among the eigenvalues as target eigenvalues, and arrange the target eigenvectors corresponding to the target eigenvalues row by row to obtain a second matrix;
[0092] Step S204, calculate the product of the first matrix and the second matrix, and use the obtained product as the target network management log data after dimensionality reduction.
[0093] In this embodiment, the preprocessed target network management log data is formed into a first matrix X of NxM by columns. For a first matrix X containing n samples and M features, where each sample x i =(x i1 , x i2 ,..., x im ) is an m-dimensional vector.
[0094] Calculate the covariance matrix C of the first matrix X. The covariance matrix is an important tool for analyzing multi-dimensional data, and it can describe the relationships between various features in the preprocessed target network management log data. In machine learning, the covariance matrix is often used in techniques such as principal component analysis (PCA) to help understand the internal structure of the preprocessed target network management log data. By calculating the covariance matrix, the correlation between features and the distribution characteristics of the data can be better understood.
[0095] Regarding the calculation of the covariance matrix C of the first matrix X, exemplarily, assume there is a matrix This is a data set containing 4 samples and 2 features, and the features are feature 1 and feature 2 respectively.
[0096] First step, calculate the mean of each feature: For feature 1, For feature 2,
[0097] Second step, centralize the data: Subtract the mean of each feature from each data point to obtain a centralized data set,
[0098] Third step: Calculate the covariance matrix:
[0099] Calculate the variance of Feature 1
[0100] Calculate the covariance of Feature 1 and Feature 2:
[0101]
[0102] Calculate the variance of Feature 2:
[0103] Therefore, the matrix The covariance matrix of
[0104] It is easy to understand that the above parameters are for reference only and are not limited here.
[0105] Calculate the covariance matrix C of the first matrix X. After obtaining the covariance matrix C, calculate the eigenvalues of the covariance matrix C and the eigenvectors corresponding to each eigenvalue. Then select the largest K target eigenvalues from the eigenvalues of the covariance matrix C, arrange the target eigenvalues from largest to smallest, and arrange the target eigenvectors corresponding to the target eigenvalues row by row to obtain the second matrix P.
[0106] Calculate the product Y of the first matrix X multiplied by the second matrix P, and use the obtained product Y = PX as the target network management log data after dimensionality reduction. Among them, the size of the matrix Y is K×M.
[0107] Step S30, evaluate the target network management log data after dimensionality reduction through the log detection model to obtain the network management log data evaluation result.
[0108] In this embodiment, a log detection model is constructed, referring to Figure 6 , Figure 6 is a schematic diagram of the log detection model architecture of an embodiment of the present invention. As Figure 6 shown, the log detection model includes a bidirectional long short-term memory network layer (bidirectional LSTM, Long Short-Term Memory), a fully connected layer, and a classification output layer containing a softmax function. The target network management log data after dimensionality reduction is used as an input vector and input into the log detection model. After being processed by the bidirectional long short-term memory network layer (bidirectional LSTM, Long Short-Term Memory) in the log detection model, the output of the bidirectional LSTM is put into the fully connected layer, and after being processed by the softmax function, the network management log data evaluation result output by the log detection model is obtained, that is, the classification result of the network management log data. The classification result includes abnormal network management log data or not abnormal network management log data.
[0109] Among them, the basic structure of the LSTM includes:
[0110] Forget gate: determines how much information from previous time steps should be discarded;
[0111] Input gate: determines how much information at the current time step should be written into the cell state;
[0112] Output gate: determines how much information should be output to the next time step.
[0113] Bidirectional LSTM runs two independent LSTMs at each time step, one from the start to the end of the sequence (forward LSTM) and the other from the end to the start of the sequence (backward LSTM). These two LSTMs can effectively identify log sequences with strong context relevance.
[0114] Fully connected layers (FC) are a basic type of layer in neural networks, usually located in the last few layers of the network and used as the output layer for classification tasks. The main characteristic of fully connected layers is that each neuron is connected to every neuron in the previous layer, which means that each input affects each output. In fully connected layers, the input vector undergoes a linear transformation through a weight matrix, then a bias term is added, and finally a non-linear transformation is performed through an activation function (such as ReLU, Sigmoid, Tanh, etc.).
[0115] The softmax function is an activation function that can normalize a numerical vector into a probability distribution vector, and the sum of all probabilities is 1. The Softmax function is used as the last layer of a neural network for the output of multi-classification problems.
[0116] In some specific embodiments, refer to Figure 5 , Figure 5 is Figure 1 a detailed process schematic diagram of step S30 in Figure 5 As shown in
[0117] Step S301: evenly divide the dimension-reduced target network management log data to obtain W sample subset data;
[0118] Step S302: select any sample subset data that has not been used as a validation set sample subset data from the W sample subset data as the validation set;
[0119] Step S303: use the remaining sample subset data as the training set to input into the log detection model to obtain the evaluation result of the network management log data output by the log detection model;
[0120] Step S304: adjust the parameters of the log detection model based on the validation set and the grid search method to obtain the log detection model after parameter adjustment;
[0121] Step S305: Use the log detection model with adjusted parameters as the log detection model, and return to execute Step S302 to Step S305 until each sample subset data has been used as the validation set, obtaining W network management log data evaluation results;
[0122] Step S306: Calculate the mean of the W network management log data evaluation results, and use the mean as the final network management log data evaluation result.
[0123] In this embodiment, referring to Figure 7 , split the dimension-reduced target network management log data into W sample subset data of equal size, traverse these W subsets in sequence, each time use the current subset as the validation set, and use all the remaining samples as the training set for model training and evaluation. Finally, use the average of the W network management log data evaluation results as the final network management log data evaluation result.
[0124] Specifically, select any sample subset data that has not been used as the validation set from the W sample subset data as the validation set. For example Figure 7 select the last sample subset data in
[0125] as the validation set, and then use the remaining sample subset data as the training set to input into the log detection model, obtaining the network management log data evaluation result E1 output by the log detection model.
[0126] Verify the network management log data evaluation result E1 output by the log detection model based on the validation set, and adjust the parameters of the log detection model based on the verification result to obtain the log detection model with adjusted parameters.
[0127] Use the log detection model with adjusted parameters as the log detection model, and return to execute Step S302 to Step S305 until each sample subset data has been selected as the validation set, and then W network management log data evaluation results can be obtained. Then calculate the mean of the W network management log data evaluation results, and the obtained mean is the final network management log data evaluation result.
[0127] Further, in one embodiment, Step S304 specifically includes:
[0128] Based on the validation set, adjust the activation function, the number of hidden layers, and the number of memory units in the log detection model through the grid search method to obtain multiple groups of parameter combinations;
[0129] Based on the validation set, determine the best parameter combination among the multiple groups of parameter combinations;
[0130] Based on the best parameter combination, obtain the log detection model with adjusted parameters.
[0131] In this embodiment, the verification set is compared with the evaluation result E1 of the network management log data, and the log detection model is tuned based on the comparison result. Specifically: through the grid search method, the parameters to be adjusted, namely the activation function, the number of hidden layers, and the number of memory units, are used as the input parameters of the parameter matrix and put into the log detection model. By the exhaustive method, all parameter combinations are traversed, and the evaluation results of the network management log data corresponding to each parameter combination are compared with the verification set respectively to find the best parameter combination among all parameter combinations. Using the best parameter combination to replace the corresponding parameters in the log detection model, the tuned log detection model can be obtained.
[0132] Taking the tuned log detection model as the log detection model, and returning to execute steps S302 to S305, the accuracy of the evaluation result of the network management log data of the tuned log detection model can be further verified.
[0133] In this embodiment, the original network management log data is preprocessed to obtain the preprocessed target network management log data; the target network management log data is dimensionally reduced to obtain the dimensionally reduced target network management log data; the dimensionally reduced target network management log data is evaluated through the log detection model to obtain the evaluation result of the network management log data. Through this embodiment, the principal component analysis is used to dimensionally reduce the target network management log data. For the characteristic of the diversity of network management log features, the main feature information can be effectively extracted. Then, using the characteristic that the bidirectional LSTM in the log detection model has good recognition ability for complex sequence data, the evaluation result of the network management log data can be obtained quickly. This not only improves the efficiency of identifying abnormal network management logs, reduces the labor cost, but also has high reusability. Only by standardizing the input vector of the input log detection model can the subsequent logs be detected, solving the technical problems in the prior art that abnormal network management logs cannot be quickly identified, resulting in low efficiency and accuracy of the abnormal analysis of the network management system.
[0134] In the second aspect, an embodiment of the present invention further provides a network management log anomaly detection device.
[0135] In one embodiment, referring to Figure 8 , Figure 8 is a schematic diagram of the functional modules of an embodiment of the network management log anomaly detection device of the present invention. As Figure 8 shown, the network management log anomaly detection device includes:
[0136] A preprocessing module 10, configured to preprocess the original network management log data to obtain the preprocessed target network management log data;
[0137] A data processing module 20, configured to dimensionally reduce the target network management log data to obtain the dimensionally reduced target network management log data;
[0138] The data evaluation module 30 is configured to evaluate the dimension-reduced target network management log data through a log detection model to obtain a network management log data evaluation result.
[0139] Optionally, in one embodiment, the preprocessing module 10 is specifically configured to:
[0140] Parse the original network management log data through the Drain algorithm to obtain the parsed target network management log data;
[0141] Perform One-Hot encoding on the parsed target network management log data to obtain the preprocessed target network management log data.
[0142] Optionally, in one embodiment, the data processing module 20 is specifically configured to:
[0143] Arrange the target network management log data to obtain a first matrix;
[0144] Calculate the eigenvalues of the covariance matrix of the first matrix and the eigenvectors corresponding to the eigenvalues;
[0145] Select the largest K eigenvalues among the eigenvalues as target eigenvalues, and arrange the target eigenvectors corresponding to the target eigenvalues row by row to obtain a second matrix;
[0146] Calculate the product of the first matrix and the second matrix, and use the obtained product as the dimension-reduced target network management log data.
[0147] Optionally, in one embodiment, the data evaluation module 30 is specifically configured to:
[0148] Divide the dimension-reduced target network management log data equally to obtain W sample subset data;
[0149] Select any sample subset data that has not been used as a validation set sample subset data among the W sample subset data as the validation set;
[0150] Use the remaining sample subset data as the training set to input into the log detection model to obtain the network management log data evaluation result output by the log detection model;
[0151] Adjust the parameters of the log detection model based on the validation set and the grid search method to obtain the log detection model after parameter adjustment;
[0152] Use the log detection model after parameter adjustment as the log detection model, and return to the step of selecting any sample subset data that has not been used as a validation set sample subset data among the W sample subset data as the validation set until each sample subset data has been used as a validation set, to obtain W network management log data evaluation results;
[0153] Calculate the mean of the evaluation results of W network management log data, and use the mean as the final evaluation result of the network management log data.
[0154] Optionally, in one embodiment, the log detection model includes a bidirectional long short-term memory network layer, a fully connected layer, and a classification output layer including a softmax function.
[0155] Optionally, in one embodiment, the network management log anomaly detection device further includes a parameter tuning module configured to:
[0156] Based on the validation set, adjust the activation function, the number of hidden layers, and the number of memory units in the log detection model through the grid search method to obtain multiple groups of parameter combinations;
[0157] Based on the validation set, determine the best parameter combination among the multiple groups of parameter combinations;
[0158] Based on the best parameter combination, obtain the log detection model after parameter tuning.
[0159] Wherein, the functions of each module in the above network management log anomaly detection device correspond to the steps in the above network management log anomaly detection method embodiment, and their functions and implementation processes will not be elaborated here one by one.
[0160] In a third aspect, an embodiment of the present invention further provides an electronic device, the structure of which is as Figure 9 shown, including: a memory and a processor, the processor is used to read and execute the computer program stored in the memory to implement the foregoing network management log anomaly detection method.
[0161] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium, in which computer executable instructions are stored, and when the computer executable instructions are executed, the foregoing network management log anomaly detection method is implemented.
[0162] Finally, it should be noted that: in some processes described in the embodiments of the present invention, multiple operations or steps appear in a specific order, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present invention or may be executed in parallel. The serial numbers of the operations are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.
[0163] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.
Claims
1. A method for detecting anomalies in network management logs, characterized in that, The method includes: Preprocessing the original network management log data to obtain the target network management log data after preprocessing; Reducing the dimension of the target network management log data to obtain the target network management log data after dimension reduction; Evaluating the target network management log data after dimension reduction through a log detection model to obtain the evaluation result of the network management log data.
2. The network management log anomaly detection method according to claim 1, wherein, The step of preprocessing the original network management log data to obtain the target network management log data after preprocessing includes: Parsing the original network management log data through the Drain algorithm to obtain the target network management log data after parsing; Performing One-Hot encoding on the target network management log data after parsing to obtain the target network management log data after preprocessing.
3. The network management log anomaly detection method according to claim 1, wherein The step of reducing the dimension of the target network management log data to obtain the target network management log data after dimension reduction includes: Arranging the target network management log data to obtain a first matrix; Calculating the eigenvalues of the covariance matrix of the first matrix and the eigenvectors corresponding to the eigenvalues; Selecting the largest K eigenvalues among the eigenvalues as the target eigenvalues, arranging the target eigenvectors corresponding to the target eigenvalues row by row to obtain a second matrix; Calculating the product of the first matrix and the second matrix, and using the obtained product as the target network management log data after dimension reduction.
4. The network management log anomaly detection method according to claim 1, wherein The step of evaluating the target network management log data after dimension reduction through a log detection model to obtain the evaluation result of the network management log data includes: Step S301, evenly dividing the target network management log data after dimension reduction to obtain W sample subset data; Step S302, selecting any one of the W sample subset data that has not been used as a validation set sample subset data as the validation set; Step S303, using the remaining sample subset data as the training set to input into the log detection model to obtain the evaluation result of the network management log data output by the log detection model; Step S304, adjusting the parameters of the log detection model based on the validation set and the grid search method to obtain the log detection model after parameter adjustment; Step S305, using the log detection model after parameter adjustment as the log detection model, and returning to execute steps S302 to S305 until each sample subset data has been used as the validation set to obtain W evaluation results of the network management log data; Step S306, calculating the mean value of the W evaluation results of the network management log data, and using the mean value as the final evaluation result of the network management log data.
5. The network management log anomaly detection method according to claim 4, wherein The log detection model includes a bidirectional long short-term memory network layer, a fully connected layer, and a classification output layer including a softmax function.
6. The network management log anomaly detection method according to claim 4, wherein, The step of adjusting the parameters of the log detection model based on the validation set and the grid search method to obtain the log detection model after parameter adjustment includes: Based on the validation set, adjusting the activation function, the number of hidden layers, and the number of memory units in the log detection model through the grid search method to obtain multiple groups of parameter combinations; Based on the validation set, determining the best parameter combination among the multiple groups of parameter combinations; Based on the best parameter combination, obtaining the log detection model after parameter adjustment.
7. A network management log anomaly detection device, characterized in that, The device includes: A preprocessing module, configured to preprocess the original network management log data to obtain the preprocessed target network management log data; A data processing module, configured to perform dimensionality reduction on the target network management log data to obtain the dimensionally reduced target network management log data; A data evaluation module, configured to evaluate the dimensionally reduced target network management log data through a log detection model to obtain a network management log data evaluation result.
8. The network management log anomaly detection device according to claim 7, wherein The data processing module is configured to: Arrange the target network management log data to obtain a first matrix; Calculate the eigenvalues of the covariance matrix of the first matrix and the eigenvectors corresponding to the eigenvalues; Select the largest K eigenvalues among the eigenvalues as the target eigenvalues, and arrange the target eigenvectors corresponding to the target eigenvalues row by row to obtain a second matrix; Calculate the product of the first matrix and the second matrix, and use the obtained product as the dimensionally reduced target network management log data.
9. An electronic device, characterized in that, Including: A memory and a processor; The processor is configured to read and execute the computer program stored in the memory to implement the steps of the network management log anomaly detection method according to any one of claims 1-6.
10. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed, the steps of the network management log anomaly detection method according to any one of claims 1-6 are implemented.