A network log anomaly analysis method, system, terminal and storage medium

Abstract

The application discloses a network log anomaly analysis method and system, a terminal and a storage medium, and relates to the technical field of network log anomaly analysis.The method comprises the following steps: acquiring network traffic data, performing data preprocessing on the network traffic data to obtain target traffic data; acquiring a network traffic public data set, training a data stream anomaly detection algorithm model according to the network traffic public data set, and obtaining a trained target data stream anomaly detection algorithm model; inputting the target traffic data into the target data stream anomaly detection algorithm model, analyzing the target traffic data based on the target data stream anomaly detection algorithm model, and outputting normal traffic data and abnormal traffic data; and performing abnormal point labeling processing on the abnormal traffic data to obtain abnormal labeled traffic data.The application uses an AI algorithm to realize abnormal traffic detection in network data, identify abnormal traffic, has high detection accuracy, is fast in detection efficiency, and provides corresponding network security protection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network monitoring, and particularly relates to a network log anomaly analysis method and system, a terminal and a computer readable storage medium. BACKGROUND

[0002] Network anomalies (the exclamation mark appearing when a network anomaly usually indicates that the device cannot connect to the Internet) and server errors (usually referring to problems encountered when trying to access or use a server) are common problems in the process of using the Internet, which may be caused by factors such as unstable network connection, excessive server load or software failure, and these problems will cause users to be unable to normally access web pages, send and receive emails, use online services or carry out other network activities; at the same time, if you want to know what happened in the network in order to gain insight into potential threats and prevent them from becoming attacks, you need to view logs, perform log analysis to obtain log anomaly analysis results.

[0003] Log analysis refers to the process of processing and interpreting collected log data, identifying abnormal events or discovering potential problems, but the existing log analysis only uses some detected data for simple analysis, and does not perform effective data analysis, resulting in low accuracy of abnormal analysis and low efficiency.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] The main purpose of the present application is to provide a network log anomaly analysis method, system, terminal and computer readable storage medium, which aims to solve the problem of low accuracy and slow efficiency of network log anomaly analysis in the prior art, and cannot quickly detect abnormal traffic.

[0006] To achieve the above purpose, the present application provides a network log anomaly analysis method, which comprises the following steps:

[0007] Obtaining network traffic data, pre-processing the network traffic data to obtain target traffic data;

[0008] Obtaining a network traffic public data set, training a data stream anomaly detection algorithm model according to the network traffic public data set to obtain a trained target data stream anomaly detection algorithm model;

[0009] Inputting the target traffic data into the target data stream anomaly detection algorithm model, analyzing the target traffic data based on the target data stream anomaly detection algorithm model, and outputting normal traffic data and abnormal traffic data;

[0010] The abnormal traffic data is annotated to obtain anomaly-annotated traffic data.

[0011] Optionally, in the network log anomaly analysis method, the data preprocessing includes feature extraction and data normalization.

[0012] The process of acquiring network traffic data and preprocessing the network traffic data to obtain target traffic data specifically includes:

[0013] After acquiring network traffic data, feature extraction is performed on the network traffic data to extract specific features, which are used to reflect the normal and abnormal behavior of network traffic.

[0014] After feature extraction from the network traffic data, data normalization is performed on the network traffic data to eliminate the dimensional differences between different features, thereby obtaining the target traffic data.

[0015] Optionally, the network log anomaly analysis method, wherein obtaining a publicly available network traffic dataset and training a data flow anomaly detection algorithm model based on the publicly available network traffic dataset to obtain a trained target data flow anomaly detection algorithm model specifically includes:

[0016] Obtain a publicly available dataset of network traffic, which includes various types of normal and abnormal data;

[0017] The data flow anomaly detection algorithm model is trained based on various normal and abnormal data in the publicly available network traffic dataset to obtain the trained target data flow anomaly detection algorithm model.

[0018] The data flow anomaly detection algorithm model includes a data flow anomaly detection model based on random cut forest and a long short-term memory network model.

[0019] Optionally, in the network log anomaly analysis method, the data flow anomaly detection model based on random cut forest is used for feature extraction; and the long short-term memory network model is used as a traffic classifier for detecting and classifying abnormal traffic.

[0020] Optionally, the network log anomaly analysis method, wherein training the data flow anomaly detection algorithm model based on various normal and abnormal data in the publicly available network traffic dataset to obtain a trained target data flow anomaly detection algorithm model specifically includes:

[0021] A set of random cut trees is constructed using the random cut forest-based data stream anomaly detection model, and the random cut trees are used to detect abnormal traffic data. If a data point is in an abnormal position in multiple trees, it is considered to be abnormal traffic data.

[0022] The random cut forest-based data stream anomaly detection model is initialized, the number of random cut trees and related parameters are set, network traffic data in the network traffic public data set is input into the random cut forest-based data stream anomaly detection model for preliminary anomaly detection, and an abnormal score of each data point is output. Suspected abnormal points are screened out;

[0023] A long short-term memory network model is constructed, the network structure, hidden layer size, and learning rate of the long short-term memory network model are set, the long short-term memory network model is trained using labeled abnormal traffic data to learn the pattern difference between normal traffic data and abnormal traffic data;

[0024] The suspected abnormal points output by the random cut forest-based data stream anomaly detection model are input into the trained long short-term memory network model to use the trained long short-term memory network model to predict and classify network traffic data and determine whether it is abnormal traffic data;

[0025] The output results of the random cut forest-based data stream anomaly detection model and the long short-term memory network model are fused in a preset manner to obtain a fusion result. According to the fusion result, a preset threshold is set, and the abnormal score of a data point is compared with the preset threshold to determine whether the network traffic data is abnormal. If the abnormal score of a data point exceeds the preset threshold, it is marked as abnormal traffic data.

[0026] Optionally, the network log anomaly analysis method, wherein the target traffic data is input into the target data stream anomaly detection algorithm model, the target traffic data is analyzed based on the target data stream anomaly detection algorithm model, and normal traffic data and abnormal traffic data are output. Specifically, it includes:

[0027] The target traffic data is input into the random cut forest-based data stream anomaly detection model in the target data stream anomaly detection algorithm model. The random cut forest-based data stream anomaly detection model performs preliminary anomaly detection on the target traffic data and outputs an abnormal score of each data point in the target traffic data. Suspected abnormal points are screened out to obtain a first analysis result.

[0028] input the suspected abnormal point into the long short-term memory network model in the target data stream anomaly detection algorithm model, the long short-term memory network model performs prediction and classification on the suspected abnormal point, judges whether the suspected abnormal point is abnormal traffic data, and obtains a second analysis result;

[0029] The first analysis result and the second analysis result are fused to obtain a fusion result, a preset threshold is set according to the fusion result, the abnormal scores of all data points are compared with the preset threshold to judge whether the network traffic data in the target traffic data is abnormal, and if the abnormal score of a certain data point exceeds the preset threshold, the data point is marked as abnormal traffic data, and the remaining network traffic data is marked as normal traffic data.

[0030] Optionally, the network log anomaly analysis method comprises:

[0031] A preset acceptance index is set, and the identification accuracy of the target data stream anomaly detection algorithm model is evaluated using the preset acceptance index.

[0032] If the preset acceptance index is greater than a preset value, it indicates that the identification accuracy of the target data stream anomaly detection algorithm model meets the anomaly detection requirements of the network traffic data.

[0033] In addition, to achieve the above object, the application further provides a network log anomaly analysis system, wherein the network log anomaly analysis system comprises:

[0034] A data acquisition and processing module is configured to acquire network traffic data, pre-process the network traffic data, and obtain target traffic data.

[0035] A model training module is configured to acquire a network traffic public data set, train a data stream anomaly detection algorithm model based on the network traffic public data set, and obtain a trained target data stream anomaly detection algorithm model.

[0036] A data analysis module is configured to input the target traffic data into the target data stream anomaly detection algorithm model, analyze the target traffic data based on the target data stream anomaly detection algorithm model, and output normal traffic data and abnormal traffic data.

[0037] An abnormal data labeling module is configured to perform abnormal point labeling processing on the abnormal traffic data, and obtain abnormal labeled traffic data.

[0038] In addition, to achieve the above object, the application further provides a terminal, wherein the terminal comprises a memory, a processor, and a network log anomaly analysis program stored in the memory and executable on the processor, and the network log anomaly analysis program implements the steps of the network log anomaly analysis method when executed by the processor.

[0039] In addition, to achieve the above object, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a network log anomaly analysis program, and the network log anomaly analysis program implements the steps of the network log anomaly analysis method when executed by a processor.

[0040] In the application, network traffic data is acquired, data preprocessing is performed on the network traffic data to obtain target traffic data, a network traffic public data set is acquired, a data stream anomaly detection algorithm model is trained according to the network traffic public data set to obtain a trained target data stream anomaly detection algorithm model, the target traffic data is input into the target data stream anomaly detection algorithm model, the target traffic data is analyzed based on the target data stream anomaly detection algorithm model, and normal traffic data and abnormal traffic data are output, and the abnormal traffic data is subjected to abnormal point labeling processing to obtain abnormal labeled traffic data. The network traffic data is subjected to abnormal identification by combining the data stream anomaly detection model based on the random forest and the long short-term memory network model, the abnormal traffic appearing is quickly identified, the detection accuracy is high, the detection efficiency is fast, and corresponding network security is provided. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a flowchart of a preferred embodiment of the network log anomaly analysis method of the application;

[0042] Figure 2 is a schematic diagram of the security monitoring of the network traffic data by using the AI algorithm in the preferred embodiment of the network log anomaly analysis method of the application;

[0043] Figure 3 is a structure diagram of a preferred embodiment of the network log anomaly analysis system of the application;

[0044] Figure 4 is a structure diagram of a preferred embodiment of the terminal of the application. DETAILED DESCRIPTION

[0045] To make the object, technical scheme and advantages of the application more clear and definite, the application is further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application.

[0046] The network log anomaly analysis method provided by the preferred embodiment of the present application comprises the following steps as shown in the figure: Figure 1

[0047] In step S10, network traffic data is acquired, and data preprocessing is performed on the network traffic data to obtain target traffic data.

[0048] Specifically, the network traffic data targeted by the present application refers to all network data that can be acquired on the network, such as data of FTP, HTTP, SSH, Email, and HTTPS protocols. When acquiring network traffic data, data preprocessing needs to be performed on the network traffic data, which includes feature extraction and data normalization processing. When the network traffic data is acquired, feature extraction is performed on the network traffic data, and specific features are extracted from the network traffic data (feature extraction refers to extracting meaningful features from network traffic data, such as dictionary feature extraction, text feature extraction, image feature extraction, etc.), wherein the specific features are used to reflect normal behavior and abnormal behavior of network traffic. Data normalization processing is a standardization processing of network traffic data to eliminate the dimensional difference between different features, so that the model can better learn. Therefore, after feature extraction is performed on the network traffic data, data normalization processing is performed on the network traffic data to eliminate the dimensional difference between different features, and target traffic data is obtained.

[0049] Further, network traffic data can also be filtered (filtering the original log data to remove irrelevant or redundant information, which can reduce the complexity of processing data and improve the effectiveness of subsequent analysis), cleaned (such as format content error cleaning, logical error cleaning, removing unnecessary data, correlation verification, etc.), and processed for abnormal values (such as abnormal value deletion, abnormal value filling, etc.) according to actual needs, so that the processed data is more in line with the requirements.

[0050] In step S20, a network traffic public data set is acquired, and a data stream anomaly detection algorithm model is trained according to the network traffic public data set to obtain a trained target data stream anomaly detection algorithm model.

[0051] Specifically, a network traffic public data set is acquired, which includes a plurality of normal data and abnormal data. The reason for acquiring the network traffic public data set is that a large amount of normal data and abnormal data is needed to train the model. In order to ensure the accuracy of anomaly detection after the model is trained, the larger the data quantity is, the higher the accuracy is. Since it is difficult to collect such a quantity of data, the public data set needs to be used. This data set comes from the real world, is reliable and effective, and is easy to be recognized. ​

[0052] Then, after obtaining the network traffic public dataset, a data stream anomaly detection algorithm model can be trained according to normal data and abnormal data in the network traffic public dataset, to obtain a trained target data stream anomaly detection algorithm model; wherein the data stream anomaly detection algorithm model includes a random cut forest-based data stream anomaly detection model (RRCF, Robust Random Cut Forest) and a long short-term memory network model (LSTM, Long Short-Term Memory); the random cut forest-based data stream anomaly detection model is used for feature extraction; and the long short-term memory network model is used as a traffic classifier for detection and classification of abnormal traffic.

[0053] The random forest in the random cut forest-based data stream anomaly detection model RRCF is an algorithm that integrates multiple trees through the idea of ensemble learning, and its basic unit is a decision tree, and its essence belongs to a branch of machine learning, i.e., ensemble learning (Ensemble Learning) method. There are two keywords in the name of the random forest, one is "random", and the other is "forest". "Forest" is easy to understand, a tree is called a tree, and hundreds of trees can be called a forest, which is also the embodiment of the main idea of "integration" of the random forest. Each decision tree is a classifier (assuming that the current problem is a classification problem), so for an input sample, N trees will have N classification results. The random forest integrates all the classification voting results, and specifies the class with the most votes as the final output. Among all current algorithms, RRCF has excellent accuracy and can effectively run on large data sets, can process input samples with high-dimensional features, does not need to reduce dimensions, can evaluate the importance of each feature in the classification problem, can obtain an unbiased estimate of the internal generation error during the generation process, and can also obtain good results for default value problems.

[0054] The long short-term memory network model LSTM is a time recurrent neural network, which is specially designed to solve the long-term dependence problem of general RNN (recurrent neural network). All RNNs have a chain form of repeated neural network modules. LSTM is a powerful recurrent neural network that solves the vanishing gradient problem of RNN by introducing a gating mechanism, so as to learn and retain past information. LSTM performs well in machine translation, speech recognition and natural language processing tasks. LSTM networks can capture long-term dependencies, and they have a memory unit that can store information for a long time. In traditional RNNs, there is a problem of gradient vanishing and explosion when training the model on long sequences. LSTM networks handle this problem by using a gating mechanism that selectively recalls or forgets information. LSTM enables the model to capture and remember important context even if there is a significant time interval between related events in the sequence. Therefore, LSTMS are used where understanding context is important. For example, machine translation. LSTM has been used in anomaly detection tasks such as detecting fraud and network intrusion, which can be trained to identify patterns in data that deviate from the norm and label them as potential anomalies.

[0055] Further, the specific process of training the data stream anomaly detection algorithm model is as follows:

[0056] A set of random cut trees is constructed using the random cut forest-based data stream anomaly detection model RRCF, which is used to detect abnormal traffic data. If a data point is in an abnormal position in multiple trees (far away from most data points), it is considered to be abnormal traffic data.

[0057] The random cut forest-based data stream anomaly detection model RRCF is initialized, the number of random cut trees and other related parameters are set, the network traffic data in the network traffic public data set is input into the random cut forest-based data stream anomaly detection model RRCF for preliminary anomaly detection, and the anomaly score of each data point is output (RRCF will output the anomaly score of each data point, the higher the score, the more likely the anomaly point is an anomaly point), and the suspected anomaly points are screened out.

[0058] A long short-term memory network model LSTM is constructed, the network structure, hidden layer size and learning rate of the long short-term memory network model LSTM are set, and the long short-term memory network model LSTM is trained using labeled abnormal traffic data to learn the pattern difference between normal traffic data and abnormal traffic data.

[0059] The suspected abnormal points (or all data points) output by the random cut forest-based data stream anomaly detection model RRCF are input into the trained long short-term memory network model LSTM to predict and classify the network traffic data by using the trained long short-term memory network model LSTM, so as to determine whether the network traffic data is abnormal.

[0060] The output results of the random cut forest-based data stream anomaly detection model RRCF and the long short-term memory network model LSTM are fused in a preset manner (for example, a weighted average, voting or other fusion strategy can be used) to obtain a fusion result. A preset threshold is set according to the fusion result, and the abnormal score of a data point is compared with the preset threshold to determine whether the network traffic data is abnormal. If the abnormal score of a certain data point exceeds the preset threshold, the data point is marked as abnormal traffic data.

[0061] Then, after the data stream anomaly detection algorithm model is trained to obtain the trained target data stream anomaly detection algorithm model, the trained target data stream anomaly detection algorithm model can be used to analyze and identify new network traffic data.

[0062] In step S30, the target traffic data is input into the target data stream anomaly detection algorithm model, and the target traffic data is analyzed based on the target data stream anomaly detection algorithm model to output normal traffic data and abnormal traffic data.

[0063] Specifically, first, the target traffic data is input into the random cut forest-based data stream anomaly detection model RRCF in the target data stream anomaly detection algorithm model. The random cut forest-based data stream anomaly detection model RRCF performs preliminary anomaly detection on the target traffic data and outputs an abnormal score of each data point in the target traffic data. Suspected abnormal points are filtered out to obtain a first analysis result. Then, the suspected abnormal points are input into the long short-term memory network model LSTM in the target data stream anomaly detection algorithm model. The long short-term memory network model LSTM predicts and classifies the suspected abnormal points to determine whether the suspected abnormal points are abnormal traffic data, and obtains a second analysis result. Finally, the first analysis result and the second analysis result are fused to obtain a fusion result. A preset threshold is set according to the fusion result. The abnormal scores of all data points are compared with the preset threshold to determine whether the network traffic data in the target traffic data is abnormal. If the abnormal score of a certain data point exceeds the preset threshold, the data point is marked as abnormal traffic data, and the remaining network traffic data is marked as normal traffic data.

[0064] In step S40, the abnormal traffic data is subjected to abnormal point labeling processing to obtain abnormal labeled traffic data.

[0065] Specifically, after the target traffic data is detected for abnormalities by the random forest-based data stream anomaly detection model RRCF and the long short-term memory network model LSTM, the obtained abnormal traffic data is subjected to abnormal point labeling processing, so as to obtain abnormal labeled traffic data. Then, the specific reasons for network abnormalities can be analyzed according to the abnormal labeled traffic data, so as to solve network abnormal problems in a timely manner according to the specific reasons, provide corresponding network security protection, for example, when a potential security event is found, corresponding response measures are performed, for example, blocking access of an attacker, repairing a vulnerability, clearing malicious activities, etc. At the same time, an investigation is carried out to understand the source of the event, the scope of influence and measures to prevent similar events from occurring again. In addition, all analysis results, actions and findings can be recorded, and appropriate reports can be generated. These reports can be used to guide future security decisions, improve security measures and compliance requirements.

[0066] Further, a preset acceptance index (for example, F1 score is used as the acceptance index) is set, and the preset acceptance index is used to evaluate the recognition accuracy of the target data stream anomaly detection algorithm model. If the preset acceptance index is greater than a preset value (for example, F1 score is greater than or equal to 0.9), it indicates that the recognition accuracy of the target data stream anomaly detection algorithm model meets the requirements of network traffic data anomaly detection. The performance of the anomaly detection system is evaluated by using cross-validation or other evaluation methods, including accuracy, recall rate, F1 score and other indicators, which are public indicators for judging the accuracy of the model, so as to evaluate the recognition accuracy of the target data stream anomaly detection algorithm model.

[0067] The beneficial effects of the present application are:

[0068] (1) By analyzing log data, abnormal events in the system or application program, such as errors, abnormal behaviors, illegal access, etc. can be identified, which helps to respond in a timely manner and take appropriate measures to protect the security of the system.

[0069] (2) Log analysis can help locate and solve faults or problems in the system. By analyzing log events, error messages, warnings and other indicators, the root cause of the problem can be traced and appropriate corrective measures can be taken.

[0070] (3) By analyzing log data, the performance and resource utilization of the system can be evaluated. For example, analyzing the logs of network devices can help determine bottlenecks, optimize network traffic and response time, and thus improve overall performance.

[0071] (4) Through log analysis, potential security threats, intrusion behaviors or malicious activities can be monitored and identified. By checking abnormal access, login failures, malware activities, etc. in the logs, security risks can be discovered and addressed in a timely manner.

[0072] (5) Through in-depth analysis of log data, user behavior patterns, trends and preferences can be identified. This helps to understand customer needs, optimize product design or develop marketing strategies.

[0073] Further, as shown in Figure 2 , the network traffic data (i.e. Figure 2 traffic data) is obtained, the network traffic data is input into the data processing module, the network traffic data is subjected to feature extraction and data normalization processing by the data processing module, the network traffic data processed by the data processing module is analyzed by an AI algorithm (i.e. a target data stream anomaly detection algorithm model, a data stream anomaly detection model RRCF based on random cut forest and a long short-term memory network model LSTM) to output normal traffic data and abnormal traffic data (i.e. Figure 2 normal traffic and abnormal traffic in ), and finally the normal traffic data and abnormal traffic data are input into the traffic processing module, the normal traffic data and abnormal traffic data are detected, the abnormal traffic data is subjected to abnormal point labeling processing, and abnormal labeled traffic data is obtained.

[0074] Further, as shown in Figure 3 , based on the above network log anomaly analysis method, the present application also correspondingly provides a network log anomaly analysis system, wherein the network log anomaly analysis system comprises:

[0075] A data acquisition and processing module 51 is configured to acquire network traffic data, pre-process the network traffic data, and obtain target traffic data.

[0076] A model training module 52 is configured to acquire a network traffic public data set, train a data stream anomaly detection algorithm model based on the network traffic public data set, and obtain a trained target data stream anomaly detection algorithm model.

[0077] A data analysis module 53 is configured to input the target traffic data into the target data stream anomaly detection algorithm model, analyze the target traffic data based on the target data stream anomaly detection algorithm model, and output normal traffic data and abnormal traffic data.

[0078] An abnormal data labeling module 54 is configured to label the abnormal traffic data as abnormal points, and obtain abnormal labeled traffic data.

[0079] Further, as shown in Figure 4As shown, based on the network log anomaly analysis method and system, the application further provides a terminal, which comprises a processor 10, a memory 20 and a display 30. Figure 4 Only part of the components of the terminal are shown, but it should be understood that all the shown components are not required to be implemented, and more or less components can be alternatively implemented.

[0080] The memory 20 can be an internal storage unit of the terminal in some embodiments, such as a hard disk or a memory of the terminal. The memory 20 can also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal. Further, the memory 20 can include both the internal storage unit and the external storage device of the terminal. The memory 20 is used to store application software and various data installed on the terminal, such as program codes of the terminal, etc. The memory 20 can also be used to temporarily store data that has been output or will be output. In an embodiment, the network log anomaly analysis program 40 is stored on the memory 20, which can be executed by the processor 10, so as to implement the network log anomaly analysis method in the application.

[0081] The processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, which is used to run program codes or process data stored in the memory 20, such as executing the network log anomaly analysis method, etc.

[0082] The display 30 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 30 is used to display information of the terminal and to display visualized user interfaces. The components 10-30 of the terminal communicate with each other through a system bus.

[0083] In an embodiment, the following steps are implemented when the processor 10 executes the network log anomaly analysis program 40 in the memory 20:

[0084] Obtaining network traffic data, performing data preprocessing on the network traffic data to obtain target traffic data;

[0085] obtaining a network traffic public dataset, training a data stream anomaly detection algorithm model according to the network traffic public dataset, and obtaining a trained target data stream anomaly detection algorithm model;

[0086] inputting the target traffic data into the target data stream anomaly detection algorithm model, analyzing the target traffic data based on the target data stream anomaly detection algorithm model, and outputting normal traffic data and abnormal traffic data;

[0087] performing anomaly point labeling processing on the abnormal traffic data to obtain abnormal labeled traffic data.

[0088] The data preprocessing includes feature extraction and data normalization processing.

[0089] The network traffic data is obtained, and the network traffic data is preprocessed to obtain target traffic data, specifically including:

[0090] When the network traffic data is obtained, the network traffic data is extracted for specific features, and the specific features are used to reflect the normal behavior and abnormal behavior of the network traffic.

[0091] After the network traffic data is extracted for specific features, the network traffic data is normalized to eliminate the dimensional difference between different features, and the target traffic data is obtained.

[0092] The network traffic public dataset is obtained, and the data stream anomaly detection algorithm model is trained according to the network traffic public dataset to obtain the trained target data stream anomaly detection algorithm model, specifically including:

[0093] The network traffic public dataset includes a plurality of normal data and abnormal data.

[0094] The data stream anomaly detection algorithm model is trained according to the plurality of normal data and abnormal data in the network traffic public dataset to obtain the trained target data stream anomaly detection algorithm model.

[0095] The data stream anomaly detection algorithm model includes a random cut forest-based data stream anomaly detection model and a long short-term memory network model.

[0096] The random cut forest-based data stream anomaly detection model is used for feature extraction, and the long short-term memory network model is used as a traffic classifier for detecting and classifying abnormal traffic.

[0097] The data stream anomaly detection algorithm model is trained according to a plurality of normal data and abnormal data in the network flow public data set, and a trained target data stream anomaly detection algorithm model is obtained, and the method specifically comprises the following steps:

[0098] A set of random cut trees is constructed using the random cut forest-based data stream anomaly detection model, and the random cut trees are used to detect abnormal traffic data. If a data point is in an abnormal position in multiple trees, it is considered to be abnormal traffic data.

[0099] The random cut forest-based data stream anomaly detection model is initialized, the number of random cut trees and related parameters are set, the network traffic data in the network flow public data set is input into the random cut forest-based data stream anomaly detection model for preliminary anomaly detection, and the anomaly score of each data point is output, and suspected abnormal points are screened out.

[0100] A long short-term memory network model is constructed, the network structure, hidden layer size and learning rate of the long short-term memory network model are set, the long short-term memory network model is trained using labeled abnormal traffic data to learn the pattern difference between normal traffic data and abnormal traffic data.

[0101] The suspected abnormal points output by the random cut forest-based data stream anomaly detection model are input into the trained long short-term memory network model to predict and classify network traffic data using the trained long short-term memory network model to determine whether it is abnormal traffic data.

[0102] The output results of the random cut forest-based data stream anomaly detection model and the long short-term memory network model are fused in a preset manner to obtain a fusion result, a preset threshold is set according to the fusion result, the anomaly score of a data point is compared with the preset threshold to determine whether the network traffic data is abnormal, and if the anomaly score of a data point exceeds the preset threshold, it is marked as abnormal traffic data.

[0103] The target traffic data is input into the target data stream anomaly detection algorithm model, the target traffic data is analyzed based on the target data stream anomaly detection algorithm model, and normal traffic data and abnormal traffic data are output, and the method specifically comprises the following steps:

[0104] The target traffic data is input into the random cut forest-based data stream anomaly detection model in the target data stream anomaly detection algorithm model, the random cut forest-based data stream anomaly detection model performs preliminary anomaly detection on the target traffic data, and outputs the anomaly score of each data point in the target traffic data, screens out suspected abnormal points, and obtains a first analysis result.

[0105] The suspected anomalies are input into the long short-term memory network model in the target data stream anomaly detection algorithm model. The long short-term memory network model predicts and classifies the suspected anomalies, determines whether the suspected anomalies are abnormal traffic data, and obtains the second analysis result.

[0106] The first analysis result and the second analysis result are fused to obtain a fusion result. A preset threshold is set according to the fusion result. The abnormal scores of all data points are compared with the preset threshold to determine whether the network traffic data in the target traffic data is abnormal. If the abnormal score of a certain data point exceeds the preset threshold, it is marked as abnormal traffic data, and the remaining network traffic data is marked as normal traffic data.

[0107] The network log anomaly analysis method includes:

[0108] Set preset acceptance criteria, and use the preset acceptance criteria to evaluate the recognition accuracy of the target data stream anomaly detection algorithm model;

[0109] If the preset acceptance index is greater than the preset value, it means that the recognition accuracy of the target data flow anomaly detection algorithm model meets the requirements for anomaly detection of network traffic data.

[0110] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a network log anomaly analysis program, which, when executed by a processor, implements the steps of the network log anomaly analysis method described above.

[0111] In summary, this invention provides a method, system, terminal, and storage medium for network log anomaly analysis. The method includes: acquiring network traffic data; preprocessing the network traffic data to obtain target traffic data; acquiring a publicly available network traffic dataset; training a data flow anomaly detection algorithm model based on the publicly available network traffic dataset to obtain a trained target data flow anomaly detection algorithm model; inputting the target traffic data into the target data flow anomaly detection algorithm model; analyzing the target traffic data based on the target data flow anomaly detection algorithm model; and outputting normal traffic data and abnormal traffic data; and anomaly point labeling processing of the abnormal traffic data to obtain anomaly-labeled traffic data. This invention combines a data flow anomaly detection model based on random cut forests with a long short-term memory network model to identify anomalies in network traffic data, quickly identifying abnormal traffic with high accuracy and efficiency, and providing corresponding network security protection.

[0112] It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0113] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer readable computer readable storage medium, and the program can include the processes of the above-mentioned method embodiments when executed. The computer readable storage medium can be a memory, a magnetic disc, an optical disc, etc.

[0114] It should be understood that the application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all these improvements and changes shall fall within the protection scope of the appended claims of the present application.

Claims

1. A method for anomaly analysis of network logs, characterized in that, The network log anomaly analysis method includes: Obtain network traffic data, perform data preprocessing on the network traffic data, and obtain target traffic data; Obtain a publicly available dataset of network traffic, and train a data flow anomaly detection algorithm model based on the publicly available dataset of network traffic to obtain a trained target data flow anomaly detection algorithm model; the data flow anomaly detection algorithm model includes a data flow anomaly detection model based on random cut forest and a long short-term memory network model; The step of training the data flow anomaly detection algorithm model based on various normal and abnormal data in the publicly available network traffic dataset to obtain a trained target data flow anomaly detection algorithm model specifically includes: The data flow anomaly detection model based on random cut forest is used to construct a set of randomly cut trees. The random cut trees are used to detect abnormal traffic data. If a data point is in an abnormal position in multiple trees, it is considered to be abnormal traffic data. Initialize the data flow anomaly detection model based on random forest cutting, set the number of randomly cut trees and related parameters, input the network traffic data in the public network traffic dataset into the data flow anomaly detection model based on random forest cutting for preliminary anomaly detection, and output the anomaly score of each data point to filter out suspected anomalies; A long short-term memory network model is constructed, and the network structure, hidden layer size and learning rate of the long short-term memory network model are set. The long short-term memory network model is trained using labeled abnormal traffic data to learn the pattern differences between normal traffic data and abnormal traffic data. The suspected anomaly points output by the data flow anomaly detection model based on random cut forest are input into the trained long short-term memory network model, so as to use the trained long short-term memory network model to predict and classify network traffic data and determine whether it is abnormal traffic data. The outputs of the data flow anomaly detection model based on random cut forest and the long short-term memory network model are fused using a weighted average or voting strategy to obtain a fusion result. A preset threshold is set according to the fusion result, and the anomaly score of the data point is compared with the preset threshold to determine whether the network traffic data is abnormal. If the anomaly score of a data point exceeds the preset threshold, it is marked as abnormal traffic data. The target traffic data is input into the target data stream anomaly detection algorithm model, and the target traffic data is analyzed based on the target data stream anomaly detection algorithm model to output normal traffic data and abnormal traffic data. The step of inputting the target traffic data into the target data stream anomaly detection algorithm model, analyzing the target traffic data based on the target data stream anomaly detection algorithm model, and outputting normal traffic data and abnormal traffic data specifically includes: The target traffic data is input into the random forest-based data flow anomaly detection model in the target data flow anomaly detection algorithm model. The random forest-based data flow anomaly detection model performs preliminary anomaly detection on the target traffic data and outputs the anomaly score of each data point in the target traffic data, filters out suspected anomalies, and obtains the first analysis result. The suspected anomalies are input into the long short-term memory network model in the target data stream anomaly detection algorithm model. The long short-term memory network model predicts and classifies the suspected anomalies, determines whether the suspected anomalies are abnormal traffic data, and obtains the second analysis result. The first analysis result and the second analysis result are fused to obtain a fusion result. A preset threshold is set according to the fusion result. The abnormal scores of all data points are compared with the preset threshold to determine whether the network traffic data in the target traffic data is abnormal. If the abnormal score of a certain data point exceeds the preset threshold, it is marked as abnormal traffic data. The remaining network traffic data is marked as normal traffic data. The abnormal traffic data is annotated to obtain anomaly-annotated traffic data. The specific causes of network anomalies are analyzed based on the anomaly-annotated traffic data so that network anomalies can be resolved in a timely manner based on the specific causes, and corresponding network security protection can be provided. When potential security incidents are discovered, corresponding response measures are implemented, such as blocking attackers' access, patching vulnerabilities, eliminating malicious activities, and conducting investigations to understand the source of the incident, the scope of impact, and measures to prevent similar incidents from happening again. All analysis results, actions, and findings are recorded and reports are generated to guide future security decisions, improve security measures, and meet compliance requirements.

2. The network log anomaly analysis method according to claim 1, characterized in that, The data preprocessing includes feature extraction and data normalization. The process of acquiring network traffic data and preprocessing the network traffic data to obtain target traffic data specifically includes: After acquiring network traffic data, feature extraction is performed on the network traffic data to extract specific features, which are used to reflect the normal and abnormal behavior of network traffic. After feature extraction from the network traffic data, data normalization is performed on the network traffic data to eliminate the dimensional differences between different features, thereby obtaining the target traffic data.

3. The network log anomaly analysis method according to claim 1, characterized in that, The step of obtaining a publicly available network traffic dataset and training a data flow anomaly detection algorithm model based on the publicly available network traffic dataset to obtain a trained target data flow anomaly detection algorithm model specifically includes: Obtain a publicly available dataset of network traffic, which includes various types of normal and abnormal data; The data flow anomaly detection algorithm model is trained based on various normal and abnormal data in the publicly available network traffic dataset to obtain the trained target data flow anomaly detection algorithm model.

4. The network log anomaly analysis method according to claim 3, characterized in that, The data flow anomaly detection model based on random cut forest is used for feature extraction; the long short-term memory network model is used as a traffic classifier for the detection and classification of abnormal traffic.

5. The network log anomaly analysis method according to any one of claims 1-4, characterized in that, The network log anomaly analysis method includes: Set preset acceptance criteria, and use the preset acceptance criteria to evaluate the recognition accuracy of the target data stream anomaly detection algorithm model; If the preset acceptance index is greater than the preset value, it means that the recognition accuracy of the target data flow anomaly detection algorithm model meets the requirements for anomaly detection of network traffic data.

6. A network log anomaly analysis system, characterized in that, The network log anomaly analysis system is applied to the network log anomaly analysis method according to any one of claims 1-5, and the network log anomaly analysis system comprises: The data acquisition and processing module is used to acquire network traffic data, perform data preprocessing on the network traffic data, and obtain target traffic data. The model training module is used to acquire a publicly available network traffic dataset and train the data flow anomaly detection algorithm model based on the publicly available network traffic dataset to obtain a trained target data flow anomaly detection algorithm model. The data analysis module is used to input the target traffic data into the target data stream anomaly detection algorithm model, analyze the target traffic data based on the target data stream anomaly detection algorithm model, and output normal traffic data and abnormal traffic data. The abnormal data annotation module is used to annotate the abnormal traffic data to obtain abnormal annotated traffic data.

7. A terminal, characterized in that, The terminal includes: a memory, a processor, and a network log anomaly analysis program stored in the memory and executable on the processor. When the network log anomaly analysis program is executed by the processor, it implements the steps of the network log anomaly analysis method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a network log anomaly analysis program, which, when executed by a processor, implements the steps of the network log anomaly analysis method as described in any one of claims 1-5.

Images