Distributed security situation intelligent sensing system and method

Abstract

The invention relates to a distributed security situation intelligent sensing system and method. The method comprises the following steps: each client acquires network flow data in real time and transmits the network flow data to a database in a server for storage; and inputting the network flow data into the log intelligent analysis module and the flow intelligent analysis module for intelligent analysis, obtaining an anomaly detection result, inputting the anomaly detection result into the workflow model, obtaining a final anomaly detection result, and performing visual display. The real-time acquisition of massive network flow data is realized by using each client to acquire the network flow data in real time; network flow data are input into a log intelligent analysis module and a flow intelligent analysis module for log intelligent analysis and flow intelligent analysis respectively, fault diagnosis is finally realized through a workflow model, the deep learning model is fully utilized, the accuracy of fault diagnosis is improved, meanwhile, the loss of human resources is reduced, and the fault diagnosis efficiency is improved. Data processing and analysis can be improved, and accurate fault positioning and quick response are achieved.

Description

Technical Field

[0001] The present invention relates to the field of network security and data processing technology, and in particular to a distributed security situation intelligent perception system and method. Background Art

[0002] With the rapid development of information technology, network attack methods are becoming increasingly diverse and complex. Traditional security defense methods can no longer meet the needs of modern network security. Therefore, network security systems have emerged and become an important tool for ensuring network security. Existing network security systems are usually based on distributed storage and processing technology, that is, data and computing tasks are distributed to multiple nodes for storage and processing. For example, the Hadoop-based log analysis system processes data in a cluster manner, which improves the efficiency and reliability of analysis. However, such systems often face problems such as delays or omissions in data processing and false reporting of important information. Especially when processing large-scale data, real-time performance becomes a major challenge and cannot be effectively responded to emergencies in a timely manner. In addition, the complex configuration and maintenance of distributed systems are also relatively difficult, which affects the efficiency of network security maintenance.

[0003] In terms of anomaly detection, common systems such as Snort and Suricata detect abnormal traffic and attack behaviors by capturing network traffic data and using pattern matching and rule engines. At the same time, traditional anomaly detection technologies often rely on manual rules or simple statistical models, which are difficult to deal with complex abnormal situations, and ultimately affect the timeliness and accuracy of fault location. In terms of visualization technology. By presenting log and traffic data to users in a graphical manner, the readability and analysis efficiency of the data are improved. For example, using tools such as Kibana, users can intuitively view and analyze log data. However, visualization technology also faces some challenges. For complex log and traffic data, how to effectively reduce the dimension and aggregate the data in order to reduce the complexity of visualization while maintaining information integrity is a problem that needs to be solved.

[0004] In summary, traditional network security systems often have problems such as data processing delays, missed and false alarms, time-consuming and cumbersome fault location, and high visualization complexity. Summary of the invention

[0005] Based on this, in order to solve the above technical problems, a distributed security situation intelligent perception system and method are provided, which can improve data processing and analysis, and achieve accurate fault location and rapid response.

[0006] A distributed security situation intelligent perception system, the system includes a server, several clients, a log intelligent analysis module, and a flow intelligent analysis module; the server is connected to each of the clients through a wide area network, the server is connected to a database, each of the clients is connected to the database, the log intelligent analysis module, and the flow intelligent analysis module, and the database is connected to the log intelligent analysis module and the flow intelligent analysis module; wherein:

[0007] Each of the clients collects network flow data in real time, and transmits the network flow data to the database for storage;

[0008] Each of the clients transmits the log data in the network flow data to the log intelligent analysis module, processes the log data through the LogGT model of the log intelligent analysis module, and obtains a first anomaly detection result of the log data;

[0009] Each of the clients transmits the traffic data in the network flow data to the traffic intelligent analysis module, processes the traffic data through the NetBert model of the traffic intelligent analysis module, and obtains a second anomaly detection result of the traffic data;

[0010] Inputting the first anomaly detection result and the second anomaly detection result into the workflow model to obtain a final anomaly detection result of the network flow data, and transmitting the final anomaly detection result to the database for storage;

[0011] Each of the clients reads the final anomaly detection result from the database through the server and displays it visually.

[0012] In one of the embodiments, the system further includes a client data network interface and a server data network interface;

[0013] The log intelligent analysis module and the traffic intelligent analysis module communicate with the client through the client data network interface;

[0014] The log intelligent analysis module and the traffic intelligent analysis module are connected to and communicate with the database via a server-side data network interface.

[0015] In one of the embodiments, a distributed data collection cluster is built on each of the clients; the client collects network flow data in real time through the distributed data collection cluster.

[0016] A distributed security situation intelligent perception method, the method comprising:

[0017] Each client collects network flow data in real time and transmits the network flow data to the database in the server for storage;

[0018] Each of the clients inputs the network flow data into the log intelligent analysis module and the traffic intelligent analysis module respectively, and performs anomaly detection through the LogGT model of the log intelligent analysis module to obtain a first anomaly detection result; performs anomaly detection through the NetBert model of the traffic intelligent analysis module to obtain a second anomaly detection result;

[0019] Inputting the first anomaly detection result and the second anomaly detection result into the workflow model to obtain a final anomaly detection result of the network flow data, and transmitting the final anomaly detection result to the database for storage;

[0020] The client reads the final anomaly detection result from the database through the server and displays it visually.

[0021] In one embodiment, each client collects network flow data in real time and transmits the network flow data to a database in the server for storage, including:

[0022] The distributed data collection cluster built on each of the clients collects network flow data in real time, and divides the network flow data into log data and flow data;

[0023] A log parsing algorithm is used to extract log constant data and log variable data from the log data, to complete intelligent analysis of the log data, and to obtain processed log data;

[0024] Performing data processing on the flow data to obtain numerical data, and performing text data processing on the numerical data to obtain processed flow data;

[0025] The processed log data and processed traffic data are transmitted to the database in the server for storage.

[0026] In one embodiment, anomaly detection is performed by using the LogGT model of the log intelligent analysis module to obtain a first anomaly detection result, including:

[0027] Inputting the processed log data into the LogGT model, converting each log entry in the processed log data into a high-dimensional vector through the embedding layer of the LogGT model, and extracting time series features through the GRU network to obtain a time series feature sequence;

[0028] The time series feature sequence is modeled through the Transformer Encoder layer to obtain feature values;

[0029] The feature values ​​are concatenated in the time dimension and input into the fully connected layer to obtain the final feature value;

[0030] The final feature value is mapped into the target space to obtain a first anomaly detection result.

[0031] In one of the embodiments, the GRU network includes a candidate hidden state and a current hidden state; wherein the GRU network outputs the current hidden state at a time step;

[0032] The Transformer Encoder layer is processed through a multi-head attention mechanism and residual connection, and layer normalization is performed to calculate the eigenvalues.

[0033] In one embodiment, anomaly detection is performed by using the NetBert model of the traffic intelligent analysis module to obtain a second anomaly detection result, including:

[0034] The processed traffic data is input into a custom convolutional neural network, spatial features are extracted by the custom convolutional neural network, and temporal features are captured by BiLSTM to obtain temporal feature values;

[0035] Inputting the processed traffic data into a pre-trained Bert model to obtain data feature values;

[0036] The time feature value and the data feature value are combined to obtain a joint feature vector, and the joint feature vector is mapped to a final category output to obtain a second anomaly detection result.

[0037] In one embodiment, the BiLSTM uses the features transmitted by the CNN layer as input, and performs an update operation on each time step through a forget gate, an input gate, a candidate memory unit, and an output gate;

[0038] The pre-trained Bert model converts the processed traffic data into a token ID sequence through a tokenizer and adds a tag.

[0039] In one embodiment, the method further includes inputting the first anomaly detection result and the second anomaly detection result into a workflow model to obtain a final anomaly detection result of the network flow data, and transmitting the final anomaly detection result to the database for storage, including:

[0040] Inputting the first anomaly detection result and the second anomaly detection result into a workflow model, and inferring the cause of the anomaly through the workflow model;

[0041] A network flow data fault diagnosis is performed according to the abnormal cause to obtain a final abnormality detection result of the network flow data.

[0042] The above-mentioned distributed security situation intelligent perception system and method realizes the real-time collection of massive network flow data by using each client to collect network flow data in real time; by inputting the network flow data into the log intelligent analysis module and the traffic intelligent analysis module to perform log intelligent analysis and traffic intelligent analysis respectively, and finally importing these two analysis results into the workflow model to finally realize fault diagnosis, and fully utilizes the deep learning model to improve the accuracy of fault diagnosis while reducing the loss of human resources, which can improve data processing and analysis, and realize accurate fault location and rapid response; at the same time, the final anomaly detection results are displayed through a visual interface to provide users with real-time data perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 An application environment diagram of a distributed security situation intelligent perception system in one embodiment;

[0044] Figure 2 A structural block diagram of a distributed security situation intelligent perception system in one embodiment;

[0045] Figure 3 A schematic diagram of a process of a distributed security situation intelligent perception method in one embodiment;

[0046] Figure 4 A schematic diagram of the structure of a LogGT model in one embodiment;

[0047] Figure 5 A schematic diagram of the structure of a NetBert model in one embodiment;

[0048] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe abnormal detection results, but these abnormal detection results are not limited by these terms. These terms are only used to distinguish a first abnormal detection result from another abnormal detection result. For example, without departing from the scope of this application, a first abnormal detection result may be referred to as a second abnormal detection result, and similarly, a second abnormal detection result may be referred to as a first abnormal detection result. Both the first abnormal detection result and the second abnormal detection result are abnormal detection results, but they are not the same abnormal detection result.

[0051] In one embodiment, Figure 1 , Figure 2 As shown, a distributed security situation intelligent perception system is provided, including: a server 100, a plurality of clients 200, a log intelligent analysis module 300, and a traffic intelligent analysis module 400; the server 100 is connected to communicate with each client 200 through a wide area network, the server 100 is connected to a database 110, each client 200 is connected to the database 110, the log intelligent analysis module 300, and the traffic intelligent analysis module 400, respectively, and the database 110 is connected to the log intelligent analysis module 300 and the traffic intelligent analysis module 400; wherein:

[0052] Each client 200 collects network flow data in real time and transmits the network flow data to the database 110 for storage; wherein the network flow data collected in real time includes log data and flow data for subsequent processing;

[0053] Each client 200 transmits the log data in the network flow data to the log intelligent analysis module 300, and processes the log data through the LogGT model of the log intelligent analysis module 300 to obtain a first anomaly detection result of the log data;

[0054] Each client 200 transmits the traffic data in the network flow data to the traffic intelligent analysis module 400, and processes the traffic data through the NetBert model of the traffic intelligent analysis module 400 to obtain a second anomaly detection result of the traffic data;

[0055] Input the first anomaly detection result and the second anomaly detection result into the workflow model to obtain the final anomaly detection result of the network flow data, and transmit it to the database 110 for storage;

[0056] Each client 200 reads the final anomaly detection result from the database 110 through the server 100 and displays it visually.

[0057] In one embodiment, Figure 2As shown, a distributed security situation intelligent perception system provided may also include a client data network interface and a server data network interface; the log intelligent analysis module and the traffic intelligent analysis module establish a network connection and communication with the client through the client data network interface; the log intelligent analysis module and the traffic intelligent analysis module establish a network connection and communication with the database through the server data network interface.

[0058] In this embodiment, if Figure 2 As shown, the server accesses the client through the wide area network, and the server connects to the database client to collect network flow data in real time. The collected network flow data is transmitted to the database. The database sends the network flow data to the log intelligent analysis module and the traffic intelligent analysis module through the client data network interface for anomaly detection. The anomaly detection result is returned to the database through the server data network interface, and then transmitted to the client through the server. The client generates a data view and displays it through the display interface.

[0059] In one embodiment, a distributed data collection cluster is built on each client; the client collects network flow data in real time through the distributed data collection cluster. By building a distributed data collection cluster on each client to collect network flow data in real time, real-time collection of massive network flow data can be quickly achieved, saving time.

[0060] In one embodiment, Figure 3 As shown, a distributed security situation intelligent perception method is provided, including the following steps:

[0061] Step 302: Each client collects network flow data in real time and transmits the network flow data to a database in the server for storage.

[0062] Specifically, each client can collect network flow data in real time and store the collected network flow data in a database. In one embodiment, a distributed security situation intelligent perception method provided can also include a data processing and storage process, and the specific process includes: a distributed data collection cluster built on each client collects network flow data in real time, and divides the network flow data into log data and flow data; a log parsing algorithm is used to extract log constant data and log variable data from the log data, complete intelligent analysis of the log data, and obtain processed log data; data processing is performed on the flow data to obtain numerical data, and the numerical data is processed as text data to obtain processed flow data; the processed log data and processed flow data are transmitted to the database in the server for storage.

[0063] By building a distributed data collection cluster on the client, real-time collection of network flow data can be achieved, network flow data in the current network architecture can be obtained in real time, and the collected network flow data can be written into the database for storage. Network flow data is divided into log data and traffic data. Log data involves specific protocols, while traffic data involves communication information of the network layer and the transport layer. Data preparation can be done for importing log intelligent analysis modules and traffic intelligent analysis modules.

[0064] In the process of data processing, logs can be analyzed using parsing algorithms to extract constants and variables from the original log data. The log constants generate log templates for processing log event sequences. In the log intelligent parsing part, log event sequences and log variable data are combined to achieve intelligent analysis of log data. At the same time, data format conversion, feature mapping, normalization, and one-hot encoding can be performed on the original text data of traffic data to obtain numerical data for input into the model. The text data of the data is then processed to obtain text data with consistent length, and finally the preparation of numerical data and text data for importing into the model is completed.

[0065] In this embodiment, real-time collection of network flow data and division of log data and traffic data are realized, and log parsing algorithm is used to parse logs and generate templates online to complete the parsing of structured log messages and template extraction; the original traffic data is processed to generate data content that can be recognized by the deep learning model, and finally all data preparation for system intelligent analysis is completed.

[0066] In step 304, each client inputs the network flow data into the log intelligent analysis module and the traffic intelligent analysis module respectively, and performs anomaly detection through the LogGT model of the log intelligent analysis module to obtain a first anomaly detection result; and performs anomaly detection through the NetBert model of the traffic intelligent analysis module to obtain a second anomaly detection result.

[0067] By importing the log data in the network flow data into the log intelligent analysis module, the traffic data into the traffic intelligent analysis module, and then passing the preliminary detection results into the workflow model, fault diagnosis of the network flow data is realized to obtain the final anomaly detection results, and then the final anomaly detection results are imported into the database through the server-side data network interface for storage.

[0068] Specifically, in one embodiment, a distributed security situation intelligent perception method provided may also include a process for performing anomaly detection on log data, and the specific process includes: inputting the processed log data into the LogGT model, converting each log entry in the processed log data into a high-dimensional vector through the embedding layer of the LogGT model, and extracting time series features through the GRU network to obtain a time series feature sequence; the time series feature sequence is modeled through the Transformer Encoder layer to obtain a feature value; the feature value is spliced ​​in the time dimension and input into the fully connected layer to obtain the final feature value; the final feature value is mapped to the target space to obtain a first anomaly detection result.

[0069] The structure of the LogGT model is as follows Figure 4 As shown in the figure, the LogGT model consists of two identical network paths. The first path processes the log event sequence, and the second path processes the log variable data. Specifically, the two paths of the LogGT model convert each log entry into a high-dimensional vector through the embedding layer, and extract the time series features of the two inputs of the processed data through their respective GRU networks; then the time series features are passed through the Transformer Encoder layer to perform more complex time series relationship modeling, and two feature values ​​v1 and v2 are obtained respectively; finally, the outputs of the last time step of the two Transformer Encoder outputs in the time dimension are spliced ​​together and input into the fully connected layer to obtain the target feature value mapped to the target space, and the final feature value is output to realize the anomaly detection of log data.

[0070] In one embodiment, the GRU network includes candidate hidden states and current hidden states; wherein the GRU network outputs the current hidden state at the time step; the Transformer Encoder layer is subjected to a multi-head attention mechanism and a residual connection, and layer normalization is performed to calculate the eigenvalues.

[0071] Specifically, the GRU network consists of candidate hidden states and the current hidden state h t It is composed of and outputs h at time step t t , the calculation formula can be expressed as: Among them, x t is the input vector of the current time step, h t-1 represents the previous hidden state, tanh is the Tanh activation function, W h is the input, U h Represents the weight matrix from the previous hidden state to the candidate state, Represents the element-wise Hadamard product, 1-z tis the retention ratio of the previous hidden state; then, update the gate z t and reset gate r t The formula can be expressed as: t =σ(W z x t +U z h t-1 );r t =σ(W r x t +U r h t-1 ), where σ is a sigmoid activation function, W z is x t To the weight matrix of the update gate, U z is the weight matrix from the previous hidden state to the update gate, W r is x t To reset the weight matrix of the gate, U r is the weight matrix from the previous hidden state to the reset gate.

[0072] Among them, the Transformer Encoder layer obtains a new feature representation after the multi-head attention mechanism and residual connection. The calculation formula can be expressed as: t =MultiHeadAttention(Q t ,K t ,V t ), where the input sequence is obtained by linear transformation: Q t =X t W Q ,K t =X t W k ,V t =X t W v , T t It is the new feature vector obtained after the input sequence is processed by TransformerEncoder at time step t; Q t is the query vector, K t is the key vector, V t is a vector of values, x t is the input sequence, W Q , W k , W v is a trainable weight matrix.

[0073] In this embodiment, the attention weight can be calculated by the dot product attention mechanism and the weighted sum vector can be obtained. The specific calculation formula is: The attention score is calculated by parallel calculation of multiple attention casts, and the calculation formula is: MultiHead(Q t,K t ,V t )=Concat(head1,…,head h )W o ;in, head i is the output of the i-th attention head, Concat is the concatenation operation, is the weight matrix of the ith attention head, W o is the output matrix after the multi-head attention mechanism; finally, the output of the multi-head attention mechanism can be residually connected with the original input and layer normalized.

[0074] In one embodiment, a distributed security situation intelligent perception method provided may also include a process for performing anomaly detection on traffic data, and the specific process includes: inputting the processed traffic data into a custom convolutional neural network, extracting spatial features through the custom convolutional neural network, and capturing time features through BiLSTM to obtain time feature values; inputting the processed traffic data into a pre-trained Bert model to obtain data feature values; merging the time feature values ​​and data feature values ​​to obtain a joint feature vector, mapping the joint feature vector to the final category output, and obtaining a second anomaly detection result.

[0075] In the process of detecting anomalies in traffic data, a model that is a mixture of a custom convolutional neural network and BiLSTM, namely the NetBert model, can be used for processing. The structure of the NetBert model is as follows: Figure 5 As shown in the figure, the first path of the NetBert model inputs the processed traffic numerical data, and first uses a custom convolutional neural network to extract the spatial features in the time series data. In this part, two layers of convolution and maximum pooling layers are used to gradually reduce the data dimension and enhance the features that the model focuses on; then the data is passed to a bidirectional LSTM to capture the temporal features in the context data and obtain the v1 feature value; the second path of the NetBert model inputs the processed traffic text data, imports the pre-trained Bert model, obtains the CLS tag of each text sequence and outputs it, and obtains the data feature value v2; the v1 and v2 feature values ​​are merged to obtain a joint feature vector, which is then sent to the fully connected layer (FC) to map the fused features to the final category output, and finally realize the anomaly detection of the traffic to obtain the second anomaly detection result.

[0076] In one embodiment, BiLSTM uses the features transmitted by the CNN layer as input, and updates each time step through the forget gate, input gate, candidate memory unit, and output gate; the pre-trained Bert model converts the processed traffic data into a token ID sequence through a tokenizer and adds a tag.

[0077] The processing of each time step of the BiLSTM can include using the features x passed by the CNN layer t As input, the specific formula can be expressed as: Forget gate f t :f t =σ(W f ·[h t-1 ,x t ]+b f );Input gate i t and candidate memory cells i t =σ(W i ·[h t-1 ,x t ]+b i ); Next, the output of the forget gate and the input gate can be used to update the cell state: Output gate o t And the final output h t :o t =σ(W o [h t-1 ,x t ]+b o );h t =o t *tanh(C t ), where W i is the weight of the input gate, W c is the bias of the input gate, b i is the weight of the candidate memory unit, b c is the bias of the candidate memory cell, C t-1 is the state of the memory unit at the previous time step, W o is the weight of the output gate, b o Bias for the output gate.

[0078] In this embodiment, the working mechanism of the Bert model is as follows: the input text is first converted into a sequence of tokenIDs through a tokenizer; then, these IDs are converted into a format that the BERT model can process, including adding special CLS and SEP tags.

[0079] Step 306: Input the first anomaly detection result and the second anomaly detection result into the workflow model to obtain the final anomaly detection result of the network flow data, and transmit it to the database for storage.

[0080] Specifically, in one embodiment, a distributed security situation intelligent perception method provided may also include a process of obtaining a final anomaly detection result, and the specific process includes: inputting a first anomaly detection result and a second anomaly detection result into a workflow model, and inferring the cause of the anomaly through the workflow model; performing network flow data fault diagnosis according to the cause of the anomaly, and obtaining the final anomaly detection result of the network flow data.

[0081] Among them, after obtaining the first anomaly detection result and the second anomaly detection result, the log sequence restored by the LogGT model in the detection can be used to reproduce the program execution order and scenario using the workflow model. The execution principle of the workflow model is that when the prediction of this step is different from the actual value of the input data here, it is considered to be an anomaly. Therefore, the final anomaly detection result is imported into the workflow model, and the cause of the anomaly is automatically inferred based on the content of the log records before and after the network flow data, and finally the fault diagnosis of the network flow data is realized to obtain the final anomaly detection result output.

[0082] Step 308: The client reads the final anomaly detection result from the database through the server and displays it visually.

[0083] The client device reads the anomaly detection result data in the database through the wide area network, and realizes the visualization of data through the Web terminal developed by Spring Boot and JSP technology, providing users with a good interface and dynamic perception of anomaly detection and fault data.

[0084] In this embodiment, the real-time collection of massive network flow data is achieved by using a cluster architecture built on the client device. The divided log data is parsed by a log parsing algorithm and then passed to the log intelligent analysis, while the traffic data is passed to the traffic intelligent analysis. Finally, the two analysis results are imported into the workflow model to ultimately realize fault diagnosis. The deep learning model is fully utilized to improve the accuracy of fault diagnosis while reducing the loss of human resources. At the same time, a good visual interface is used to provide users with real-time data perception.

[0085] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0086] In one embodiment, in order to verify a provided distributed security situation intelligent perception system, a comparative experiment for the distributed security situation intelligent perception system is provided.

[0087] In order to verify the effectiveness of the LogGT model and the NetBert model in anomaly detection, comparative experiments were conducted on the HDFS dataset and the kddtest dataset generated by combining the UNSW-NB15 dataset and the NLS-KDD 99 dataset. The HDFS dataset contains various fields of the log file, with a total of 9 parameters; the kddtest dataset contains about 490,000 network connections, each of which records a sequence of TCP packets in a specific time period, involving data transmission from the source IP to the destination IP using protocols such as TCP or UDP. The original data of the first dataset is text data, which is preprocessed by the log parsing algorithm to achieve the division of constant and variable data in the original data; the original data of the second dataset is numerical data, which is prepared to be input into the numerical processing path of the traffic intelligent processing part through data format conversion, feature mapping, normalization processing and one-hot encoding, and then these data are converted into text data of consistent length to complete the text processing path of the traffic intelligent processing part.

[0088] The first dataset is imported into the LogGT model for log anomaly detection, and the second dataset is imported into the NetBert model for traffic anomaly detection. The experimental results of these two models are compared with the traditional log and traffic anomaly detection models.

[0089] The test evaluation process is as follows:

[0090] In order to measure the performance of the LogGT model and NetBert model in log and traffic anomaly detection, the following indicators are used for evaluation: precision, recall, F1 score, accuracy (ACC), and false positive rate (FPR). The calculation formula is as follows:

[0091] Among them, TP represents the positive sample predicted by the model as the positive class, TN represents the negative sample predicted by the model as the negative class, FP represents the negative sample predicted by the model as the positive class, and FN represents the positive sample predicted by the model as the negative class.

[0092] The experimental results are:

[0093] The comparative experimental results of the two innovative deep learning models in this application on the first and second data sets are shown in Tables 1 and 2:

[0094] Table 1 Performance comparison of log anomaly detection models on HDFS dataset

[0095]

[0096] Table 2 Performance comparison of traffic anomaly detection models on the kddtest dataset

[0097]

[0098] As can be seen from Table 1, the Transformer model is easily affected by noise and performs poorly when analyzing abnormal data; the Autoencoder model cannot accurately capture important patterns and features in log data; the Attn.BiLSTM model's attention mechanism is easily affected by noise when analyzing log data. By comparing the LogGT model with other models, it can be found that the LogGT model is superior to other models in both precision and recall, and has the best overall performance.

[0099] As can be seen from Table 2, CNN cannot extract the temporal features in the second dataset well; SimpleRNN performs poorly in capturing long-term dependencies and is prone to gradient vanishing or exploding problems; GRN is easily affected by noise and category imbalance in the data; CNN+GRN is prone to information loss when processing high-dimensional data; CNN+SimpleRNN, although improved in some performance, still cannot fully extract the temporal dependency features in the data. By comparing the evaluation indicators of this application with the above models, it can be seen that this application has overcome the shortcomings of the above models, fully extracted the spatiotemporal features of traffic data, and achieved high-accuracy anomaly detection results.

[0100] Therefore, the application is reasonably designed and has high test accuracy. The real-time data collected by the client is stored, and then the network flow data is sent to the log intelligent analysis module and the traffic intelligent analysis module through the client data network interface for anomaly detection. The anomaly detection results are returned to the database through the server data network interface, and then transmitted to the client device through the server. The client device uses data visualization technology to generate dynamic and interactive data views. It can monitor and analyze network flow data in real time, and realize rapid detection and response to abnormal situations through intelligent analysis.

[0101] In one embodiment, a computer device is provided. The computer device may be a server or a client. The internal structure diagram thereof may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0102] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0103] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0104] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A distributed security situation intelligent perception system, characterized in that: The system includes a server, several clients, a log intelligent analysis module, and a traffic intelligent analysis module; the server is connected to each of the clients through a wide area network, the server is connected to a database, each of the clients is connected to the database, the log intelligent analysis module, and the traffic intelligent analysis module, and the database is connected to the log intelligent analysis module and the traffic intelligent analysis module; wherein: Each of the clients collects network flow data in real time, and transmits the network flow data to the database for storage; Each of the clients transmits the log data in the network flow data to the log intelligent analysis module, processes the log data through the LogGT model of the log intelligent analysis module, and obtains a first anomaly detection result of the log data; Each of the clients transmits the traffic data in the network flow data to the traffic intelligent analysis module, processes the traffic data through the NetBert model of the traffic intelligent analysis module, and obtains a second anomaly detection result of the traffic data; Inputting the first anomaly detection result and the second anomaly detection result into the workflow model to obtain a final anomaly detection result of the network flow data, and transmitting the final anomaly detection result to the database for storage; Each of the clients reads the final anomaly detection result from the database through the server and displays it visually.

2. The distributed security situation intelligent perception system according to claim 1 is characterized in that: The system also includes a client data network interface and a server data network interface; The log intelligent analysis module and the traffic intelligent analysis module communicate with the client through the client data network interface; The log intelligent analysis module and the traffic intelligent analysis module are connected to and communicate with the database via a server-side data network interface.

3. The distributed security situation intelligent perception system according to claim 1 is characterized in that: A distributed data collection cluster is built on each of the clients; the clients collect network flow data in real time through the distributed data collection cluster.

4. A distributed security situation intelligent perception method, characterized in that: The method comprises: Each client collects network flow data in real time and transmits the network flow data to the database in the server for storage; Each of the clients inputs the network flow data into the log intelligent analysis module and the traffic intelligent analysis module respectively, and performs anomaly detection through the LogGT model of the log intelligent analysis module to obtain a first anomaly detection result; performs anomaly detection through the NetBert model of the traffic intelligent analysis module to obtain a second anomaly detection result; Inputting the first anomaly detection result and the second anomaly detection result into the workflow model to obtain a final anomaly detection result of the network flow data, and transmitting the final anomaly detection result to the database for storage; The client reads the final anomaly detection result from the database through the server and displays it visually.

5. The distributed security situation intelligent perception method according to claim 4 is characterized in that: Each client collects network flow data in real time and transmits the network flow data to the database in the server for storage, including: The distributed data collection cluster built on each of the clients collects network flow data in real time, and divides the network flow data into log data and flow data; A log parsing algorithm is used to extract log constant data and log variable data from the log data, to complete intelligent analysis of the log data, and to obtain processed log data; Performing data processing on the flow data to obtain numerical data, and performing text data processing on the numerical data to obtain processed flow data; The processed log data and processed traffic data are transmitted to the database in the server for storage.

6. The distributed security situation intelligent perception method according to claim 5 is characterized in that: Anomaly detection is performed through the LogGT model of the log intelligent analysis module to obtain a first anomaly detection result, including: Inputting the processed log data into the LogGT model, converting each log entry in the processed log data into a high-dimensional vector through the embedding layer of the LogGT model, and extracting time series features through the GRU network to obtain a time series feature sequence; The time series feature sequence is modeled through the Transformer Encoder layer to obtain feature values; The feature values ​​are concatenated in the time dimension and input into the fully connected layer to obtain the final feature value; The final feature value is mapped into the target space to obtain a first anomaly detection result.

7. The distributed security situation intelligent perception method according to claim 6 is characterized in that: The GRU network includes a candidate hidden state and a current hidden state; wherein the GRU network outputs the current hidden state at a time step; The Transformer Encoder layer is processed through a multi-head attention mechanism and residual connection, and layer normalization is performed to calculate the eigenvalues.

8. The distributed security situation intelligent perception method according to claim 5 is characterized in that: Anomaly detection is performed through the NetBert model of the traffic intelligent analysis module to obtain a second anomaly detection result, including: The processed traffic data is input into a custom convolutional neural network, spatial features are extracted by the custom convolutional neural network, and temporal features are captured by BiLSTM to obtain temporal feature values; Inputting the processed traffic data into a pre-trained Bert model to obtain data feature values; The time feature value and the data feature value are combined to obtain a joint feature vector, and the joint feature vector is mapped to a final category output to obtain a second anomaly detection result.

9. The distributed security situation intelligent perception method according to claim 8 is characterized in that: The BiLSTM uses the features transmitted by the CNN layer as input, and updates each time step through the forget gate, input gate, candidate memory unit, and output gate; The pre-trained Bert model converts the processed traffic data into a token ID sequence through a tokenizer and adds a tag.

10. The distributed security situation intelligent perception method according to claim 4 is characterized in that: Inputting the first anomaly detection result and the second anomaly detection result into a workflow model to obtain a final anomaly detection result of the network flow data, and transmitting the final anomaly detection result to the database for storage, including: Inputting the first anomaly detection result and the second anomaly detection result into a workflow model, and inferring the cause of the anomaly through the workflow model; A network flow data fault diagnosis is performed according to the abnormal cause to obtain a final abnormality detection result of the network flow data.

Images