Distributed security situation intelligent perception system and method

Abstract

The present invention relates to a distributed security situation intelligent perception system and method. The method comprises: each client collects network flow data in real time and transmits it to a database in a server for storage; the network flow data is input into a log intelligent analysis module and a flow intelligent analysis module for intelligent analysis, anomaly detection results are obtained and input into a workflow model, and the final anomaly detection results are obtained and visualized. By using each client to collect network flow data in real time, real-time collection of massive network flow data is achieved; by inputting network flow data into a log intelligent analysis module and a flow intelligent analysis module for log intelligent analysis and flow intelligent analysis, fault diagnosis is finally achieved through a workflow model, and deep learning models are fully utilized to improve the accuracy of fault diagnosis while reducing human resource loss, thereby improving data processing and analysis, and achieving accurate fault location and rapid response.

Description

Technical Field

[0001] The present invention relates to the field of network security and data processing technology, and in particular to a distributed security situation intelligent perception system and method. Background Art

[0002] With the rapid development of information technology, cyberattack methods are becoming increasingly diverse and complex. Traditional security defenses are no longer able to meet the needs of modern network security. Consequently, network security systems have emerged as a crucial tool for ensuring network security. Existing network security systems typically rely on distributed storage and processing technologies, distributing data and computing tasks across multiple nodes for storage and processing. For example, Hadoop-based log analysis systems process data in a clustered manner, improving analysis efficiency and reliability. However, such systems often face challenges such as data processing delays, omissions, and false positives. This is particularly true when processing large amounts of data, where real-time performance becomes a major challenge, hindering timely and effective response to emergencies. Furthermore, the complex configuration and maintenance of distributed systems are relatively difficult, impacting the efficiency of network security maintenance.

[0003] In terms of anomaly detection, common systems such as Snort and Suricata capture network traffic data and use pattern matching and rule engines to detect abnormal traffic and attack behavior. At the same time, traditional anomaly detection technologies often rely on manual rules or simple statistical models, making them difficult to handle complex anomalies, ultimately affecting the timeliness and accuracy of fault location. Visualization technology improves data readability and analysis efficiency by presenting log and traffic data to users in a graphical format. For example, using tools such as Kibana, users can intuitively view and analyze log data. However, visualization technology also faces some challenges. For complex log and traffic data, how to effectively reduce its dimensionality and aggregate it to reduce visualization complexity while maintaining information integrity is a problem that needs to be solved.

[0004] In summary, traditional network security systems often have problems such as data processing delays, missed and false alarms, time-consuming and cumbersome fault location, and high visualization complexity. Summary of the Invention

[0005] Based on this, in order to solve the above technical problems, a distributed security situation intelligent perception system and method are provided, which can improve data processing and analysis, and achieve accurate fault location and rapid response.

[0006] A distributed security situation intelligent perception system, comprising a server, several clients, a log intelligent analysis module, and a traffic intelligent analysis module; the server is connected to and communicates with each of the clients via a wide area network; the server is connected to a database; each of the clients is connected to the database, the log intelligent analysis module, and the traffic intelligent analysis module; the database is connected to the log intelligent analysis module and the traffic intelligent analysis module; wherein:

[0007] Each of the clients collects network flow data in real time and transmits the network flow data to the database for storage;

[0008] Each of the clients transmits log data in the network flow data to the log intelligent analysis module, processes the log data using the LogGT model of the log intelligent analysis module, and obtains a first anomaly detection result of the log data;

[0009] Each of the clients transmits traffic data in the network flow data to the traffic intelligent analysis module, processes the traffic data using the NetBert model of the traffic intelligent analysis module, and obtains a second anomaly detection result of the traffic data;

[0010] Inputting the first anomaly detection result and the second anomaly detection result into a workflow model to obtain a final anomaly detection result of the network flow data, and transmitting the final anomaly detection result to the database for storage;

[0011] Each of the clients reads the final anomaly detection result from the database through the server and displays it visually.

[0012] In one embodiment, the system further includes a client data network interface and a server data network interface;

[0013] The log intelligent analysis module and the traffic intelligent analysis module communicate with the client through the client data network interface;

[0014] The log intelligent analysis module and the traffic intelligent analysis module establish network connection communication with the database through the server data network interface.

[0015] In one embodiment, a distributed data collection cluster is built on each of the clients; the clients collect network flow data in real time through the distributed data collection cluster.

[0016] A distributed security situation intelligent perception method, the method comprising:

[0017] Each client collects network flow data in real time and transmits the network flow data to the database in the server for storage;

[0018] Each of the clients inputs the network flow data into a log intelligent analysis module and a traffic intelligent analysis module, respectively, and performs anomaly detection using the LogGT model of the log intelligent analysis module to obtain a first anomaly detection result; and performs anomaly detection using the NetBert model of the traffic intelligent analysis module to obtain a second anomaly detection result;

[0019] Inputting the first anomaly detection result and the second anomaly detection result into a workflow model to obtain a final anomaly detection result of the network flow data, and transmitting the final anomaly detection result to the database for storage;

[0020] The client reads the final anomaly detection result from the database through the server and displays it visually.

[0021] In one embodiment, each client collects network flow data in real time and transmits the network flow data to a database in the server for storage, including:

[0022] The distributed data collection cluster built on each of the clients collects network flow data in real time and divides the network flow data into log data and traffic data;

[0023] Using a log parsing algorithm to extract log constant data and log variable data from the log data, completing intelligent analysis of the log data, and obtaining processed log data;

[0024] Performing data processing on the flow data to obtain numerical data, and performing text data processing on the numerical data to obtain processed flow data;

[0025] The processed log data and processed traffic data are transmitted to the database in the server for storage.

[0026] In one embodiment, anomaly detection is performed using the LogGT model of the log intelligent analysis module to obtain a first anomaly detection result, including:

[0027] Inputting the processed log data into the LogGT model, converting each log entry in the processed log data into a high-dimensional vector through the embedding layer of the LogGT model, and extracting time series features through the GRU network to obtain a time series feature sequence;

[0028] The time series feature sequence is modeled through the Transformer Encoder layer to obtain feature values;

[0029] The eigenvalues ​​are concatenated in the time dimension and input into the fully connected layer to obtain the final eigenvalues;

[0030] The final eigenvalue is mapped into the target space to obtain a first anomaly detection result.

[0031] In one embodiment, the GRU network includes a candidate hidden state and a current hidden state; wherein the GRU network outputs the current hidden state at a time step;

[0032] The Transformer Encoder layer undergoes a multi-head attention mechanism and residual connection, and performs layer normalization processing to calculate the eigenvalues.

[0033] In one embodiment, anomaly detection is performed using the NetBert model of the traffic intelligent analysis module to obtain a second anomaly detection result, including:

[0034] Input the processed traffic data into a custom convolutional neural network, extract spatial features through the custom convolutional neural network, and capture temporal features through BiLSTM to obtain temporal feature values;

[0035] Input the processed traffic data into the pre-trained Bert model to obtain data feature values;

[0036] The time feature value and the data feature value are combined to obtain a joint feature vector, and the joint feature vector is mapped to the final category output to obtain a second anomaly detection result.

[0037] In one embodiment, the BiLSTM uses the features transmitted by the CNN layer as input and performs an update operation on each time step through a forget gate, an input gate, a candidate memory unit, and an output gate;

[0038] The pre-trained Bert model converts the processed traffic data into a token ID sequence through a tokenizer and adds a tag.

[0039] In one embodiment, the method further includes inputting the first anomaly detection result and the second anomaly detection result into a workflow model to obtain a final anomaly detection result for the network flow data, and transmitting the final anomaly detection result to the database for storage, including:

[0040] Inputting the first anomaly detection result and the second anomaly detection result into a workflow model, and inferring the cause of the anomaly through the workflow model;

[0041] A network flow data fault diagnosis is performed based on the abnormal cause to obtain a final abnormality detection result of the network flow data.

[0042] The above-mentioned distributed security situation intelligent perception system and method realizes the real-time collection of massive network flow data by using each client to collect network flow data in real time; by inputting the network flow data into the log intelligent analysis module and the traffic intelligent analysis module for log intelligent analysis and traffic intelligent analysis respectively, and finally importing these two analysis results into the workflow model to finally realize fault diagnosis, making full use of the deep learning model to improve the accuracy of fault diagnosis while reducing the loss of human resources, which can improve data processing and analysis, and realize accurate fault location and rapid response; at the same time, the final anomaly detection results are displayed through a visual interface to provide users with real-time data perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a diagram of an application environment of a distributed security situation intelligent perception system in one embodiment;

[0044] Figure 2 A structural block diagram of a distributed security situation intelligent perception system in one embodiment;

[0045] Figure 3 1 is a flow chart of a distributed security situation intelligent perception method according to an embodiment;

[0046] Figure 4 A schematic diagram of the structure of a LogGT model in one embodiment;

[0047] Figure 5 A schematic diagram of the structure of a NetBert model in one embodiment;

[0048] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0050] It is understood that the terms "first," "second," and the like used herein may be used herein to describe anomaly detection results, but these anomaly detection results are not limited by these terms. These terms are merely used to distinguish a first anomaly detection result from another anomaly detection result. For example, without departing from the scope of this application, a first anomaly detection result may be referred to as a second anomaly detection result, and similarly, a second anomaly detection result may be referred to as a first anomaly detection result. Both the first anomaly detection result and the second anomaly detection result are anomaly detection results, but they are not the same anomaly detection result.

[0051] In one embodiment, Figure 1 、 Figure 2 As shown, a distributed security situation intelligent perception system is provided, including: a server 100, several clients 200, a log intelligent analysis module 300, and a traffic intelligent analysis module 400; the server 100 is connected to and communicates with each client 200 via a wide area network, the server 100 is connected to a database 110, each client 200 is connected to the database 110, the log intelligent analysis module 300, and the traffic intelligent analysis module 400, respectively, and the database 110 is connected to the log intelligent analysis module 300 and the traffic intelligent analysis module 400; wherein:

[0052] Each client 200 collects network flow data in real time and transmits the network flow data to the database 110 for storage; wherein the network flow data collected in real time includes log data and traffic data for subsequent processing;

[0053] Each client 200 transmits the log data in the network flow data to the log intelligent analysis module 300, and processes the log data through the LogGT model of the log intelligent analysis module 300 to obtain a first anomaly detection result of the log data;

[0054] Each client 200 transmits traffic data in the network flow data to the traffic intelligent analysis module 400, and processes the traffic data through the NetBert model of the traffic intelligent analysis module 400 to obtain a second anomaly detection result of the traffic data;

[0055] Input the first anomaly detection result and the second anomaly detection result into the workflow model to obtain the final anomaly detection result of the network flow data, and transmit it to the database 110 for storage;

[0056] Each client 200 reads the final anomaly detection result from the database 110 through the server 100 and displays it visually.

[0057] In one embodiment, Figure 2As shown, a distributed security situation intelligent perception system provided may also include a client data network interface and a server data network interface; the log intelligent analysis module and the traffic intelligent analysis module establish a network connection and communicate with the client through the client data network interface; the log intelligent analysis module and the traffic intelligent analysis module establish a network connection and communicate with the database through the server data network interface.

[0058] In this embodiment, if Figure 2 As shown, the server accesses the client through the wide area network, and the server connects to the database to collect network flow data in real time. The collected network flow data is transmitted to the database. The database sends the network flow data to the log intelligent analysis module and the traffic intelligent analysis module through the client data network interface for anomaly detection. The anomaly detection results are returned to the database through the server data network interface and then transmitted to the client through the server. The client generates a data view and displays it through the display interface.

[0059] In one embodiment, a distributed data collection cluster is built on each client; the client uses the distributed data collection cluster to collect network flow data in real time. By building a distributed data collection cluster on each client to collect network flow data in real time, it is possible to quickly achieve real-time collection of massive network flow data, saving time.

[0060] In one embodiment, Figure 3 As shown, a distributed security situation intelligent perception method is provided, including the following steps:

[0061] In step 302, each client collects network flow data in real time and transmits the network flow data to a database in the server for storage.

[0062] Specifically, each client can collect network flow data in real time and store the collected network flow data in a database. In one embodiment, a distributed security situation intelligent perception method provided can also include a data processing and storage process, specifically including: a distributed data collection cluster built on each client collects network flow data in real time and divides the network flow data into log data and flow data; a log parsing algorithm is used to extract log constant data and log variable data from the log data, completing intelligent analysis of the log data to obtain processed log data; data processing is performed on the flow data to obtain numerical data, and the numerical data is processed into text data to obtain processed flow data; the processed log data and processed flow data are transmitted to the database on the server for storage.

[0063] By building a distributed data collection cluster on the client, we can achieve real-time collection of network flow data, obtain network flow data within the current network architecture in real time, and write the collected network flow data into a database for storage. Network flow data is divided into log data and traffic data. Log data involves specific protocols, while traffic data involves communication information between the network layer and the transport layer. This prepares data for import into the log intelligent analysis module and the traffic intelligent analysis module.

[0064] During data processing, log parsing algorithms can be used to analyze logs, extracting constants and variables from the raw log data. Log templates are generated from the log constants to process log event sequences. Intelligent log parsing combines log event sequences and log variable data to implement intelligent analysis of log data. Furthermore, raw traffic data can be formatted, feature mapped, normalized, and one-hot encoded to obtain numerical data for the model input. This text data is then processed to obtain text data of consistent length, ultimately completing the preparation of both numerical and text data for input into the model.

[0065] In this embodiment, real-time collection of network flow data and division of log data and traffic data are realized, and log parsing and template generation are carried out online through log parsing algorithms to complete the parsing of structured log messages and template extraction; the original traffic data is processed to generate data content that can be recognized by the deep learning model, and finally all data preparation for system intelligent analysis is completed.

[0066] In step 304, each client inputs the network flow data into the log intelligent analysis module and the traffic intelligent analysis module respectively, and performs anomaly detection using the LogGT model of the log intelligent analysis module to obtain a first anomaly detection result; and performs anomaly detection using the NetBert model of the traffic intelligent analysis module to obtain a second anomaly detection result.

[0067] By importing the log data in the network flow data into the log intelligent analysis module, the traffic data into the traffic intelligent analysis module, and then passing the preliminary detection results into the workflow model, fault diagnosis of the network flow data is realized to obtain the final anomaly detection result, and then the final anomaly detection result is imported into the database through the server-side data network interface for storage.

[0068] Specifically, in one embodiment, a distributed security situation intelligent perception method provided may also include a process of performing anomaly detection on log data, and the specific process includes: inputting the processed log data into the LogGT model, converting each log entry in the processed log data into a high-dimensional vector through the embedding layer of the LogGT model, and extracting time series features through the GRU network to obtain a time series feature sequence; the time series feature sequence is subjected to time series relationship modeling through the Transformer Encoder layer to obtain eigenvalues; the eigenvalues ​​are spliced ​​in the time dimension and input into the fully connected layer to obtain the final eigenvalues; the final eigenvalues ​​are mapped to the target space to obtain a first anomaly detection result.

[0069] The structure of the LogGT model is as follows Figure 4 As shown in the figure, the LogGT model consists of two identical network paths: the first path processes the log event sequence, and the second path processes the log variable data. Specifically, the two paths of the LogGT model convert each log entry into a high-dimensional vector through an embedding layer. The two inputs of the processed data are then passed through their own GRU networks to extract time series features. The time series features are then passed through the Transformer Encoder layer to perform more complex temporal relationship modeling, resulting in two eigenvalues ​​v1 and v2. Finally, the outputs of the two Transformer Encoders at the last time step in the time dimension are concatenated and input into a fully connected layer, which maps the target eigenvalues ​​to the target space and outputs the final eigenvalues, thus enabling anomaly detection in log data.

[0070] In one embodiment, the GRU network includes candidate hidden states and a current hidden state; wherein the GRU network outputs the current hidden state at a time step; the Transformer Encoder layer undergoes a multi-head attention mechanism and a residual connection, and performs layer normalization processing to calculate the feature value.

[0071] Specifically, the GRU network consists of candidate hidden states and the current hidden state h t and outputs h at time step t t , the calculation formula can be expressed as: Among them, x t is the input vector of the current time step, h t-1 Represents the previous hidden state, tanh is the Tanh activation function, W h is the input, U h Represents the weight matrix from the previous hidden state to the candidate state, Represents the element-wise Hadamard product, 1-z tis the retention ratio of the previous hidden state; then, update the gate z t and reset gate r t The formula can be expressed as: t =σ(W z x t +U z h t-1 );r t =σ(W r x t +U r h t-1 ); where σ is the sigmoid activation function, W z is x t To the weight matrix of the update gate, U z is the weight matrix from the previous hidden state to the update gate, W r is x t To the weight matrix of the reset gate, U r is the weight matrix from the previous hidden state to the reset gate.

[0072] Among them, the Transformer Encoder layer obtains a new feature representation after the multi-head attention mechanism and residual connection. The calculation formula can be expressed as: T t =MultiHeadAttention(Q t ,K t ,V t ); where the input sequence is obtained by linear transformation: Q t =X t W Q ,K t =X t W k ,V t =X t W v , T t It is the new feature vector obtained after the input sequence is processed by TransformerEncoder at time step t; Q t is the query vector, K t is the key vector, V t is a vector of values, x t is the input sequence, W Q 、W k 、W v is a trainable weight matrix.

[0073] In this embodiment, the attention weight and weighted sum vector can be calculated by the dot product attention mechanism. The specific calculation formula is: The attention score is calculated in parallel by multiple attention casts, and the calculation formula is: MultiHead(Q t,K t ,V t )=Concat(head1,…,head h )W o ;in, head i is the output of the i-th attention head, Concat is the splicing operation, is the weight matrix of the i-th attention head, W o is the output matrix after the multi-head attention mechanism; finally, the output of the multi-head attention mechanism can be residually connected with the original input and layer normalized.

[0074] In one embodiment, a distributed security situation intelligent perception method provided may also include a process for performing anomaly detection on traffic data, the specific process including: inputting the processed traffic data into a custom convolutional neural network, extracting spatial features through the custom convolutional neural network, and capturing time features through BiLSTM to obtain time feature values; inputting the processed traffic data into a pre-trained Bert model to obtain data feature values; merging the time feature values ​​and data feature values ​​to obtain a joint feature vector, mapping the joint feature vector to the final category output, and obtaining a second anomaly detection result.

[0075] In the process of detecting anomalies in traffic data, a model that is a hybrid of a custom convolutional neural network and BiLSTM, namely the NetBert model, can be used for processing. The structure of the NetBert model is as follows: Figure 5 As shown in the figure, the first path of the NetBert model inputs the processed traffic numerical data, and first uses a custom convolutional neural network to extract the spatial features in the time series data. In this part, two layers of convolution and maximum pooling layers are used to gradually reduce the data dimension and enhance the features that the model focuses on; then the data is passed to a bidirectional LSTM to capture the temporal features in the context data and obtain the v1 feature value; the second path of the NetBert model inputs the processed traffic text data, imports the pre-trained Bert model, obtains the CLS tag of each text sequence and outputs it to obtain the data feature value v2; the v1 and v2 feature values ​​are merged to obtain a joint feature vector, which is then sent to the fully connected layer (FC) to map the fused features to the final category output, and finally realize traffic anomaly detection to obtain the second anomaly detection result.

[0076] In one embodiment, the BiLSTM uses the features passed by the CNN layer as input and updates each time step through the forget gate, input gate, candidate memory unit, and output gate. The pre-trained BERT model converts the processed traffic data into a token ID sequence through a tokenizer and adds tags.

[0077] The processing of each time step of BiLSTM can include using the features x passed by the CNN layer t As input, the specific formula can be expressed as: forget gate f t :f t =σ(W f ·[h t-1 ,x t ]+b f ); input gate i t and candidate memory cells i t =σ(W i ·[h t-1 ,x t ]+b i ); The outputs of the forget gate and input gate can then be used to update the cell state: Output gate o t And the final output h t :o t =σ(W o [h t-1 ,x t ]+b o );h t =o t *tanh(C t ); where W i is the weight of the input gate, W c is the bias of the input gate, b i is the weight of the candidate memory unit, b c is the bias of the candidate memory unit, C t-1 is the state of the memory unit at the previous time step, W o is the weight of the output gate, b o Bias for the output gate.

[0078] In this embodiment, the working mechanism of the BERT model is as follows: the input text is first converted into a sequence of token IDs through a tokenizer; then, these IDs are converted into a format that the BERT model can process, including adding special CLS and SEP tags.

[0079] In step 306 , the first anomaly detection result and the second anomaly detection result are input into the workflow model to obtain a final anomaly detection result for the network flow data, and the final anomaly detection result is transmitted to a database for storage.

[0080] Specifically, in one embodiment, a distributed security situation intelligent perception method provided may also include a process of obtaining a final anomaly detection result, the specific process including: inputting a first anomaly detection result and a second anomaly detection result into a workflow model, and inferring the cause of the anomaly through the workflow model; performing network flow data fault diagnosis based on the cause of the anomaly to obtain a final anomaly detection result for the network flow data.

[0081] After obtaining the first and second anomaly detection results, the log sequence recovered using the LogGT model can be used to recreate the program execution sequence and scenario using the workflow model. The workflow model operates on the principle that an anomaly is considered to have occurred when the predicted value at that step differs from the actual value of the input data at that point. Therefore, the final anomaly detection result is imported into the workflow model, and the cause of the anomaly is automatically inferred based on the content of the preceding and following log records of the network flow data, ultimately achieving fault diagnosis of the network flow data and outputting the final anomaly detection result.

[0082] In step 308, the client reads the final anomaly detection result from the database through the server and displays it visually.

[0083] The client device reads the anomaly detection result data in the database through the wide area network, and realizes the visualization of data through the web terminal developed by Spring Boot and JSP technology, providing users with a good interface and dynamic perception of anomaly detection and fault data.

[0084] In this embodiment, a cluster architecture built on client devices is used to realize real-time collection of massive network flow data. The divided log data is parsed by a log parsing algorithm and then passed to the log intelligent analysis, while the traffic data is passed to the traffic intelligent analysis. Finally, the two analysis results are imported into the workflow model to finally realize fault diagnosis. The deep learning model is fully utilized to improve the accuracy of fault diagnosis while reducing the loss of human resources. At the same time, a good visual interface is used to provide users with real-time data perception.

[0085] It should be understood that, although the various steps in the above flow chart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above flow chart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0086] In one embodiment, in order to verify a provided distributed security situation intelligent perception system, a comparative experiment for the distributed security situation intelligent perception system is provided.

[0087] To verify the effectiveness of the LogGT and NetBert models in anomaly detection, comparative experiments were conducted on the HDFS dataset and the kddtest dataset, which is a combination of the UNSW-NB15 and NLS-KDD 99 datasets. The HDFS dataset contains various fields from log files, totaling nine parameters; the kddtest dataset contains approximately 490,000 network connections, each of which records a sequence of TCP packets within a specific time period, involving data transmission from a source IP address to a destination IP address using protocols such as TCP or UDP. The raw data of the first dataset is text data, which is preprocessed using a log parsing algorithm to separate constant and variable data within the raw data. The raw data of the second dataset is numerical data. Through data format conversion, feature mapping, normalization, and one-hot encoding, it is prepared for input into the numerical processing path of the intelligent traffic processing component. This data is then converted into text data of consistent length, completing the text processing path of the intelligent traffic processing component.

[0088] The first dataset is imported into the LogGT model for log anomaly detection, and the second dataset is imported into the NetBert model for traffic anomaly detection. The experimental results of these two models are compared with traditional log and traffic anomaly detection models.

[0089] The trial evaluation process is as follows:

[0090] In order to measure the performance of the LogGT model and NetBert model in log and traffic anomaly detection, the following indicators are used for evaluation: precision, recall, F1 score, accuracy rate (ACC), and false positive rate (FPR). The calculation formula is as follows:

[0091] Among them, TP represents the positive sample predicted by the model as the positive class, TN represents the negative sample predicted by the model as the negative class, FP represents the negative sample predicted by the model as the positive class, and FN represents the positive sample predicted by the model as the negative class.

[0092] The experimental results are:

[0093] The comparative experimental results of the two innovative deep learning models in this application on the first and second datasets are shown in Tables 1 and 2:

[0094] Table 1 Performance comparison of log anomaly detection models on HDFS dataset

[0095]

[0096] Table 2 Performance comparison of traffic anomaly detection models on the kddtest dataset

[0097]

[0098] Table 1 shows that the Transformer model is susceptible to noise and performs poorly when analyzing abnormal data. The Autoencoder model fails to accurately capture important patterns and features in log data. The Attn.BiLSTM model's attention mechanism is susceptible to noise when analyzing log data. Comparing the LogGT model with other models reveals that it outperforms them in both precision and recall, achieving the best overall performance.

[0099] As can be seen from Table 2, CNN is unable to effectively extract temporal features from the second dataset; SimpleRNN performs poorly in capturing long-term dependencies and is prone to vanishing or exploding gradients; GRN is susceptible to noise and class imbalance in the data; CNN+GRN is prone to information loss when processing high-dimensional data; and CNN+SimpleRNN, despite some performance improvements, still cannot fully extract temporal dependency features from the data. By comparing the evaluation metrics of this application with those of the aforementioned models, it can be seen that this application has effectively overcome the shortcomings of the aforementioned models, fully extracting the spatiotemporal features of traffic data and achieving highly accurate anomaly detection results.

[0100] Therefore, this application is well-designed and has high test accuracy. It stores real-time data collected by the client and then feeds the network flow data into the log intelligent analysis module and the traffic intelligent analysis module through the client data network interface for anomaly detection. The anomaly detection results are returned to the database through the server data network interface and then transmitted to the client device through the server. The client device uses data visualization technology to generate dynamic and interactive data views. It can monitor and analyze network flow data in real time and quickly detect and respond to anomalies through intelligent analysis.

[0101] In one embodiment, a computer device is provided. The computer device can be a server or a client. The internal structure diagram thereof can be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. The display screen of the computer device can be a liquid crystal display or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0102] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0103] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0104] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A distributed security situation intelligent perception method, characterized in that: The method comprises: Each client collects network flow data in real time and transmits the network flow data to the database in the server for storage; Each of the clients inputs the network flow data into the log intelligent analysis module and the traffic intelligent analysis module respectively, and performs anomaly detection through the LogGT model of the log intelligent analysis module to obtain a first anomaly detection result, wherein the LogGT model consists of two identical network paths, the first path processes the log event sequence, and the second path processes the log variable data; the two paths of the LogGT model convert each log entry into a high-dimensional vector through the embedding layer, and extract the time series features of the two inputs of the processed data through their respective GRU networks; then the time series features are passed through the Transformer Encoder layer to perform more complex time series relationship modeling, and two eigenvalues ​​v1 and v2 are obtained respectively; finally, the two Transformers are used to transform the two paths. The encoder outputs the output of the last time step in the time dimension, which are spliced ​​together and input into the fully connected layer to obtain target feature values ​​mapped to the target space, and the final feature values ​​are output to realize anomaly detection of the log data; anomaly detection is performed through the NetBert model of the traffic intelligent analysis module to obtain a second anomaly detection result, including: inputting the processed traffic data into a custom convolutional neural network, extracting spatial features through the custom convolutional neural network, and capturing time features through BiLSTM to obtain time feature values; inputting the processed traffic data into a pre-trained Bert model to obtain data feature values; merging the time feature values ​​and data feature values ​​to obtain a joint feature vector, and mapping the joint feature vector to the final category output to obtain a second anomaly detection result; Inputting the first anomaly detection result and the second anomaly detection result into a workflow model to obtain a final anomaly detection result of the network flow data, and transmitting the final anomaly detection result to the database for storage; The client reads the final anomaly detection result from the database through the server and displays it visually.

2. The distributed security situation intelligent perception method according to claim 1 is characterized in that: Each client collects network flow data in real time and transmits the network flow data to the database in the server for storage, including: The distributed data collection cluster built on each of the clients collects network flow data in real time and divides the network flow data into log data and traffic data; Using a log parsing algorithm to extract log constant data and log variable data from the log data, completing intelligent analysis of the log data, and obtaining processed log data; Performing data processing on the flow data to obtain numerical data, and performing text data processing on the numerical data to obtain processed flow data; The processed log data and processed traffic data are transmitted to the database in the server for storage.

3. The distributed security situation intelligent perception method according to claim 1 is characterized in that: The GRU network includes a candidate hidden state and a current hidden state; wherein the GRU network outputs the current hidden state at a time step; The Transformer Encoder layer undergoes a multi-head attention mechanism and residual connection, and performs layer normalization processing to calculate the eigenvalues.

4. The distributed security situation intelligent perception method according to claim 1 is characterized in that: The BiLSTM uses the features passed by the CNN layer as input and performs an update operation on each time step through the forget gate, input gate, candidate memory unit, and output gate; The pre-trained Bert model converts the processed traffic data into a token ID sequence through a tokenizer and adds a tag.

5. The distributed security situation intelligent perception method according to claim 1 is characterized in that: Inputting the first anomaly detection result and the second anomaly detection result into a workflow model to obtain a final anomaly detection result of the network flow data, and transmitting the final anomaly detection result to the database for storage, including: Inputting the first anomaly detection result and the second anomaly detection result into a workflow model, and inferring the cause of the anomaly through the workflow model; A network flow data fault diagnosis is performed based on the abnormal cause to obtain a final abnormality detection result of the network flow data.

6. A distributed security situation intelligent perception system, applied to the distributed security situation intelligent perception method according to any one of claims 1 to 5, characterized in that: The system includes a server, several clients, a log intelligent analysis module, and a traffic intelligent analysis module; the server is connected to each of the clients via a wide area network, the server is connected to a database, each of the clients is connected to the database, the log intelligent analysis module, and the traffic intelligent analysis module, and the database is connected to the log intelligent analysis module and the traffic intelligent analysis module; wherein: Each of the clients collects network flow data in real time and transmits the network flow data to the database for storage; Each of the clients transmits log data in the network flow data to the log intelligent analysis module, processes the log data using the LogGT model of the log intelligent analysis module, and obtains a first anomaly detection result of the log data; Each of the clients transmits traffic data in the network flow data to the traffic intelligent analysis module, processes the traffic data using the NetBert model of the traffic intelligent analysis module, and obtains a second anomaly detection result of the traffic data; Inputting the first anomaly detection result and the second anomaly detection result into a workflow model to obtain a final anomaly detection result of the network flow data, and transmitting the final anomaly detection result to the database for storage; Each of the clients reads the final anomaly detection result from the database through the server and displays it visually.

7. The distributed security situation intelligent perception system according to claim 6, characterized in that: The system also includes a client data network interface and a server data network interface; The log intelligent analysis module and the traffic intelligent analysis module communicate with the client through the client data network interface; The log intelligent analysis module and the traffic intelligent analysis module establish network connection communication with the database through the server data network interface.

8. The distributed security situation intelligent perception system according to claim 6, characterized in that: A distributed data collection cluster is built on each of the clients; the clients collect network flow data in real time through the distributed data collection cluster.

Images