Network traffic anomaly detection system based on artificial intelligence

Abstract

The invention discloses a network traffic anomaly detection system based on artificial intelligence, and relates to the technical field of network security, and the system comprises the following components: a plurality of agent modules, a cooperative communication module, a decision fusion module and a model training and updating module. According to the invention, by integrating a plurality of agent modules, the multi-dimensional features of the network traffic can be comprehensively and deeply analyzed, and each agent module adopts advanced algorithms and technologies, such as a multi-dimensional feature mapping algorithm, a behavior pattern mining algorithm, a protocol semantic parsing algorithm and a periodic fluctuation decomposition algorithm. The algorithms can accurately capture the abnormal behavior of the network flow, and meanwhile, the decision fusion module adopts an adaptive dynamic weight fusion algorithm, comprehensively considers the detection result of each agent, obtains a final abnormal detection conclusion, and further improves the detection accuracy.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to an artificial intelligence-based network traffic anomaly detection system. Background Art

[0002] With the rapid development of the Internet, network traffic has shown explosive growth, and the complexity of the network environment has also increased. In this context, network traffic anomaly detection has become an important part of ensuring network security. The network traffic anomaly detection system based on artificial intelligence has emerged. It uses advanced artificial intelligence technology and algorithms to monitor and analyze network traffic in real time to identify and respond to potential network threats.

[0003] Traditional technologies have some significant shortcomings. Traditional technologies often lack comprehensive and in-depth analysis capabilities and find it difficult to capture the multi-dimensional characteristics of network traffic, which limits the accuracy and efficiency of detection. When faced with complex and changing network environments, the decision-making and fusion strategies of traditional systems are often too simple, making it difficult to accurately capture abnormal behavior of network traffic.

[0004] In summary, although traditional technology has certain application value in network traffic anomaly detection, its shortcomings limit its accuracy and efficiency in practical applications. Therefore, it is particularly important to develop an artificial intelligence-based network traffic anomaly detection system. Summary of the invention

[0005] The purpose of the present invention is to make up for the shortcomings of the existing technology and provide an artificial intelligence-based network traffic anomaly detection system, which can become an important way to solve the current network traffic anomaly detection problem by optimizing the collaboration mechanism between intelligent agents, improving information sharing and communication efficiency, introducing advanced decision-making and fusion algorithms, and strengthening model updating and training capabilities.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a network traffic anomaly detection system based on artificial intelligence, the system comprising the following components: multiple intelligent agent modules, a collaborative communication module, a decision fusion module and a model training and updating module; The multiple agent modules are composed of traffic feature agent, user behavior agent, application layer protocol agent and time series agent. The traffic feature agent uses a multi-dimensional feature mapping algorithm to analyze basic features. The algorithm maps the original features to a high-dimensional space and uses a kernel function to Perform a nonlinear transformation, where and represents two sample data points in the original feature space, is the kernel parameter, express and The user behavior agent uses a behavior pattern mining algorithm based on the Markov decision process to construct a state transition matrix , indicating that from the state Transfer to state The probability of a failure is calculated through historical data statistics. The application layer protocol agent uses a protocol semantic parsing algorithm to parse the application layer protocol data and identify protocol types and abnormal protocol behaviors. The algorithm splits the protocol data into semantic units. The time series agent uses a periodic fluctuation decomposition algorithm to analyze the changing trend of network traffic over time and capture periodic and sudden traffic changes. The algorithm decomposes the time series into long-term trends. , periodic component and random fluctuations , ,in Through local weighted regression calculation, The main periodic components are extracted by Fourier transform. is the residual; the collaborative communication module: the agents collaborate through message passing and a shared knowledge base, the message passing adopts a communication protocol based on a publish-subscribe model, and its storage content includes normal network traffic patterns and historical abnormal cases; The decision fusion module: adopts an adaptive dynamic weight fusion algorithm to obtain the final anomaly detection conclusion, fuses the detection results of each intelligent agent, obtains the final anomaly detection conclusion, and assigns different weights to different intelligent agents according to their detection accuracy and reliability; The model training and updating module uses historical network traffic data to train the intelligent agent detection model, adopts the stochastic gradient descent-momentum acceleration algorithm combined with the adaptive learning rate adjustment strategy, and at the same time, the system monitors the newly emerging network traffic data and abnormal cases in real time, and dynamically updates the detection model through the incremental learning algorithm based on knowledge distillation. The algorithm transfers the knowledge of new data to the student model through the teacher model, and learns new data while maintaining the original knowledge of the model.

[0007] Furthermore, when the traffic feature agent performs a multidimensional feature mapping algorithm, it performs feature selection on the mapped high-dimensional feature space and uses a feature selection method based on mutual information to calculate the mutual information between each feature and the abnormal label. , the formula is ,in It is a feature and tags The joint probability distribution of and They are and The marginal probability distribution of is obtained, the features are sorted according to the mutual information value, and the feature subset with higher mutual information value is selected to remove redundant and irrelevant features to improve the detection efficiency and accuracy. In practical applications, by setting the mutual information threshold, the features that contribute more to anomaly detection are screened out to reduce the amount of calculation and noise interference.

[0008] Furthermore, when the user behavior agent uses the behavior pattern mining algorithm, it performs personalized modeling for different user groups. First, the users are classified and divided into different groups according to their attributes. For each group, an independent Markov decision process model is constructed. When calculating the state transition probability, the data of working time and non-working time are counted separately, so that the model can better fit the actual behavior patterns of different groups and improve the pertinence of anomaly detection.

[0009] Furthermore, when the application layer protocol agent adopts the protocol semantic parsing algorithm, it combines the recurrent neural network RNN ​​in deep learning for semantic understanding, takes the semantic unit sequence of the protocol data as the input of the RNN, and captures the contextual relationship between the semantic units through the cyclic calculation of the hidden layer. The hidden layer state update formula of the RNN is: ,in is the semantic unit of the current input, is the hidden layer state at the previous moment, is the weight matrix, is the bias term, The activation function is used. In the output layer, the hidden layer state is mapped to the protocol type and abnormal judgment result through the fully connected layer. The RNN model is trained using a large amount of annotated protocol data, so that it can automatically learn the complex patterns in the protocol semantics and improve the ability to identify abnormal protocol behaviors. Furthermore, when the time series agent uses the periodic fluctuation decomposition algorithm, it further analyzes the decomposed periodic components and random fluctuations. , wavelet transform is used to analyze the energy distribution of different frequency components and determine the frequency and amplitude of the main periodic components. The wavelet transform formula is: ,in is the periodic component time series, is the wavelet basis function, is the scale parameter, is the translation parameter, for random fluctuations , through the Gaussian mixture model GMM, random fluctuations are decomposed into the superposition of multiple Gaussian distributions, the parameters of each Gaussian distribution are determined, and the abnormal changes in network traffic can be judged more accurately through in-depth analysis of periodic components and random fluctuations.

[0010] Furthermore, the shared knowledge base of the collaborative communication module adopts data encryption and access control mechanisms during data storage and query processes. The data encryption adopts an asymmetric encryption algorithm to encrypt the normal mode of network traffic and historical abnormal case data stored in the knowledge base, and generate a public-private key pair. The public key is used for data encryption, and the private key is used for data decryption. When the data is stored, the public key is used to encrypt the data and then store it. When the data is queried, the private key is used to decrypt the query result. Access control adopts a role-based access control model, assigns different roles to different agents and system users, defines corresponding access rights for each role, and ensures the security and confidentiality of data in the shared knowledge base through data encryption and access control mechanisms.

[0011] Furthermore, the decision fusion module considers the correlation between agents when adopting an adaptive dynamic weight fusion algorithm; Calculate the Pearson correlation coefficient between agents , the formula is ,in and Agent and Agent In the The results of the test, and Agent and Agent The average value of the test results is used to calculate the dynamic weight When the correlation adjustment factor is introduced , when the agent and Agent When the correlation is high, the weights of the two should be appropriately reduced to avoid excessive influence of repeated information on decision-making. The adjusted weight formula is: , so that the decision fusion results can more comprehensively reflect the effective information of different intelligent agents.

[0012] Furthermore, the model training and updating module also introduces a momentum term during the training process. , the formula is: , parameter update formula: , using the momentum term at the current moment Multiply by the learning rate To update the parameters ,in is the momentum coefficient, for The learning rate at the moment is calculated as: , is the initial learning rate, is the attenuation coefficient, for Furthermore, when the model training and updating module adopts the incremental learning algorithm based on knowledge distillation, a knowledge fusion strategy is introduced. When the teacher model transfers knowledge to the student model, not only the soft label of the prediction result is transferred, but also the feature representation of the middle layer of the model is transferred. The middle layer features of the teacher model and the student model are fused through the feature fusion layer. The formula is: ,in is the fused feature, and are the intermediate layer features of the teacher model and the student model respectively, For the fusion coefficient, the optimal value is determined through experiments, and the fused features are used for subsequent training and updating, so that the student model can better absorb the knowledge of the teacher model. At the same time, it is optimized based on the characteristics of the new data to improve the model's generalization ability and adaptability to new data.

[0013] Compared with the existing technology, this artificial intelligence-based network traffic anomaly detection system has the following beneficial effects: 1. By integrating multiple intelligent agent modules, the system can comprehensively and deeply analyze the multi-dimensional characteristics of network traffic. Each intelligent agent module adopts advanced algorithms and technologies, such as multi-dimensional feature mapping algorithm, behavior pattern mining algorithm, protocol semantic parsing algorithm and periodic fluctuation decomposition algorithm. These algorithms can accurately capture abnormal behavior of network traffic. At the same time, the decision fusion module adopts an adaptive dynamic weight fusion algorithm to comprehensively consider the detection results of each intelligent agent and draw the final anomaly detection conclusion, which further improves the detection accuracy. In addition, the system also reduces the interference of redundant information and irrelevant features and improves the detection efficiency through feature selection, personalized modeling and in-depth analysis and optimization.

[0014] 2. The system trains the intelligent agent detection model by using historical network traffic data, and adopts the stochastic gradient descent-momentum acceleration algorithm combined with the adaptive learning rate adjustment strategy to optimize the model parameters and improve the generalization ability of the model. At the same time, the system monitors the emerging network traffic data and abnormal cases in real time, and dynamically updates the detection model through the incremental learning algorithm based on knowledge distillation to ensure that the model can always adapt to the changes in network traffic. In addition, the system also introduces a knowledge fusion strategy. When the teacher model transfers knowledge to the student model, it not only transfers the soft label of the prediction result, but also transfers the feature representation of the middle layer of the model, which further improves the performance of the student model. These characteristics enable the system to flexibly cope with various complex network environments and has strong adaptability and scalability.

[0015] Other advantages, objectives and features of the present invention will be set forth in part in the following description and, in part, will be apparent to those skilled in the art based on an examination of the following or may be taught from the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 This is the process operation diagram of the network traffic anomaly detection system based on artificial intelligence; Figure 2 It is a process operation diagram of multiple intelligent agent modules. DETAILED DESCRIPTION

[0018] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0019] Embodiment 1 Embodiment 1 describes a large enterprise park with multiple office areas, R&D centers and data centers. The network structure is complex, the daily network traffic is huge and diverse, and there are many application scenarios in the park, such as employees accessing internal business systems during daily office hours and R&D personnel testing new applications. To ensure the security of the enterprise network, it is necessary to monitor the network traffic in real time and promptly detect abnormal traffic behaviors, such as malware propagation and data theft attacks. The traffic feature agent collects the original features of the source IP, destination IP, port number, traffic size, and number of data packets of the network traffic, and uses a multi-dimensional feature mapping algorithm through the kernel function Map it to a high-dimensional space, and then use the mutual information-based feature selection method to calculate the mutual information between each feature and the abnormal label , filter out key features, and the user behavior agent divides users into different groups according to employee positions and department attributes. For the office staff group, a behavior pattern mining algorithm based on the Markov decision process is constructed to count the access frequency and access content data during working hours and non-working hours, and calculate the state transition matrix , the application layer protocol agent parses HTTP, HTTPS, and SMTP application layer protocol data, splits the protocol data into semantic units and inputs them into the recurrent neural network (RNN). The RNN hidden layer state update formula is , a large amount of labeled data is used to train the model to identify abnormal protocol behavior, and the time series agent decomposes the periodic fluctuations of network traffic. The formula is ,in Through local weighted regression calculation, The main periodic components are extracted by Fourier transform. is the remaining residual, and then the wavelet transform formula is used Analyze the periodic components Using Gaussian mixture model GMM to predict random fluctuations Modeling.

[0020] Each intelligent agent communicates through message transmission and shared knowledge base collaboration through a communication protocol based on the publish-subscribe model. The shared knowledge base stores normal network traffic patterns and historical abnormal case data, uses an asymmetric encryption algorithm to encrypt data, and uses public-private key pairs to ensure data security. The role-based access control model assigns roles to different intelligent agents and system users and defines access rights.

[0021] The decision fusion module adopts an adaptive dynamic weight fusion algorithm. Assume that the number of agents is , the detection results of each agent are , dynamic weight According to the formula Calculate, where , the optimal weight adjustment coefficient is determined by the particle swarm optimization algorithm, and the Pearson correlation effect between the agents is calculated. , introducing the correlation adjustment factor , the adjusted weight formula is , the final anomaly detection result .

[0022] The model training and update module uses the company's historical network traffic data, adopts the stochastic gradient descent-momentum acceleration algorithm combined with the adaptive learning rate adjustment strategy to train the intelligent agent detection model, and introduces the momentum term in the training process. , while monitoring new traffic data and abnormal cases in real time. Through the incremental learning algorithm based on knowledge distillation and combined with the knowledge fusion strategy, the new data knowledge is transferred from the teacher model to the student model. The feature fusion formula is , update the detection model.

[0023] Example 2 This example describes a cloud computing data center that provides cloud services to many enterprises and processes massive amounts of network traffic every day. To ensure service quality, it is necessary to accurately monitor and detect anomalies in the internal network traffic of the data center to promptly detect traffic anomalies that may affect service stability.

[0024] The traffic feature agent analyzes the unique features of cloud service traffic, such as the traffic interaction pattern between virtual machines, uses a multi-dimensional feature mapping algorithm to mine potential feature associations, and determines key features through feature selection. The user behavior agent classifies and models the usage habits of different cloud tenants, and builds a Markov decision process model that conforms to the behavior patterns of each tenant. The application layer protocol agent parses special protocols related to cloud services, such as the OpenStack API call protocol, and uses deep learning models to identify abnormal protocol operations. The time series agent decomposes the time series of data center traffic, analyzes periodic components and random fluctuations, and understands the law of traffic changes.

[0025] The agents collaborate through message passing and a shared knowledge base. The shared knowledge base stores normal data center traffic patterns and historical abnormal cases. The decision fusion module determines the dynamic weight based on the performance of each agent in historical detection tasks and the complexity of the task. It adjusts the weight considering the correlation between agents and draws accurate anomaly detection conclusions.

[0026] The model training and update module uses a large amount of historical traffic data accumulated in the data center to train the detection model, and adopts the stochastic gradient descent-momentum acceleration algorithm and adaptive learning rate adjustment strategy to improve training efficiency. At the same time, the incremental learning algorithm based on knowledge distillation is combined with the knowledge fusion strategy to dynamically update the model according to the newly emerging traffic data and abnormal cases to adapt to the changes in cloud service traffic.

[0027] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. The network traffic anomaly detection system based on artificial intelligence is characterized by: The system includes the following components: multiple agent modules, collaborative communication module, decision fusion module and model training and updating module; The multiple agent modules are composed of traffic feature agent, user behavior agent, application layer protocol agent and time series agent. The traffic feature agent uses a multi-dimensional feature mapping algorithm to analyze basic features. The algorithm maps the original features to a high-dimensional space and uses a kernel function to Perform a nonlinear transformation, where and represents two sample data points in the original feature space, is the kernel parameter, express and The user behavior agent uses a behavior pattern mining algorithm based on the Markov decision process to construct a state transition matrix , indicating that from the state Transfer to state The probability of a failure is calculated through historical data statistics. The application layer protocol agent uses a protocol semantic parsing algorithm to parse the application layer protocol data and identify protocol types and abnormal protocol behaviors. The algorithm splits the protocol data into semantic units. The time series agent uses a periodic fluctuation decomposition algorithm to analyze the changing trend of network traffic over time and capture periodic and sudden traffic changes. The algorithm decomposes the time series into long-term trends. , periodic component and random fluctuations , ,in By local weighted regression calculation, The main periodic components are extracted by Fourier transform. is the residual; the collaborative communication module: the agents collaborate through message passing and a shared knowledge base, the message passing adopts a communication protocol based on a publish-subscribe model, and its storage content includes normal network traffic patterns and historical abnormal cases; The decision fusion module: adopts an adaptive dynamic weight fusion algorithm to obtain the final anomaly detection conclusion, fuses the detection results of each intelligent agent, obtains the final anomaly detection conclusion, and assigns different weights to different intelligent agents according to their detection accuracy and reliability; The model training and updating module uses historical network traffic data to train the intelligent agent detection model, adopts the stochastic gradient descent-momentum acceleration algorithm combined with the adaptive learning rate adjustment strategy, and at the same time, the system monitors the newly emerging network traffic data and abnormal cases in real time, and dynamically updates the detection model through the incremental learning algorithm based on knowledge distillation. The algorithm transfers the knowledge of new data to the student model through the teacher model, and learns new data while maintaining the original knowledge of the model.

2. The network traffic anomaly detection system based on artificial intelligence according to claim 1 is characterized in that: When the traffic feature agent performs a multi-dimensional feature mapping algorithm, it performs feature selection on the mapped high-dimensional feature space and uses a feature selection method based on mutual information to calculate the mutual information between each feature and the abnormal label. , the formula is ,in It is a feature and tags The joint probability distribution of and They are and The marginal probability distribution of is obtained, the features are sorted according to the mutual information value, the feature subset with higher mutual information value is selected, and redundant and irrelevant features are removed. In practical applications, the features that contribute more to anomaly detection are screened out by setting the mutual information threshold.

3. The network traffic anomaly detection system based on artificial intelligence according to claim 1 is characterized in that: When the user behavior agent uses the behavior pattern mining algorithm, it performs personalized modeling for different user groups. First, the users are classified and divided into different groups according to their attributes. For each group, an independent Markov decision process model is constructed. When calculating the state transition probability, the data of working time and non-working time are counted separately to make the model more in line with the actual behavior patterns of different groups.

4. The network traffic anomaly detection system based on artificial intelligence according to claim 1 is characterized in that: When the application layer protocol agent adopts the protocol semantic parsing algorithm, it combines the recurrent neural network RNN ​​in deep learning to perform semantic understanding, takes the semantic unit sequence of the protocol data as the input of the RNN, and captures the contextual relationship between the semantic units through the cyclic calculation of the hidden layer. The hidden layer state update formula of the RNN is: ,in is the semantic unit of the current input, is the hidden layer state at the previous moment, is the weight matrix, is the bias term, As the activation function, in the output layer, the hidden layer state is mapped to the protocol type and abnormal judgment result through the fully connected layer. The RNN model is trained using a large amount of annotated protocol data so that it can automatically learn the complex patterns in the protocol semantics.

5. The network traffic anomaly detection system based on artificial intelligence according to claim 1 is characterized in that: When the time series agent uses the periodic fluctuation decomposition algorithm, it further analyzes the decomposed periodic components and random fluctuations. , wavelet transform is used to analyze the energy distribution of different frequency components and determine the frequency and amplitude of the main periodic components. The wavelet transform formula is: ,in is the periodic component time series, is the wavelet basis function, is the scale parameter, is the translation parameter, for random fluctuations , the Gaussian mixture model GMM is used to model the random fluctuations, decompose them into the superposition of multiple Gaussian distributions, and determine the parameters of each Gaussian distribution.

6. The network traffic anomaly detection system based on artificial intelligence according to claim 1 is characterized in that: The shared knowledge base of the collaborative communication module adopts data encryption during the data storage and query process. The data encryption adopts an asymmetric encryption algorithm to encrypt the normal mode of network traffic and historical abnormal case data stored in the knowledge base, generate a public-private key pair, the public key is used for data encryption, and the private key is used for data decryption. When storing data, the public key is used to encrypt the data and then store it. When querying the data, the private key is used to decrypt the query result. Access control adopts a role-based access control model, assigning different roles to different agents and system users, and defining corresponding access rights for each role.

7. The network traffic anomaly detection system based on artificial intelligence according to claim 1 is characterized in that: The decision fusion module considers the correlation between intelligent agents when adopting the adaptive dynamic weight fusion algorithm; Calculate the Pearson correlation coefficient between agents , the formula is ,in and Agent and Agent In the The results of the test, and Agent and Agent The average value of the test results is used to calculate the dynamic weight When the correlation adjustment factor is introduced , when the agent and Agent When the correlation is high, the weights of the two should be appropriately reduced to avoid excessive influence of repeated information on decision-making. The adjusted weight formula is: , so that the decision fusion results can more comprehensively reflect the effective information of different intelligent agents.

8. The network traffic anomaly detection system based on artificial intelligence according to claim 1 is characterized in that: The model training and updating module also introduces momentum term during the training process , the formula is: , parameter update formula: , using the momentum term at the current moment Multiply by the learning rate To update the parameters ,in is the momentum coefficient, for The learning rate at the moment is calculated as: , is the initial learning rate, is the attenuation coefficient, for Gradient of time.

9. The network traffic anomaly detection system based on artificial intelligence according to claim 1 is characterized in that: The model training and updating module introduces a knowledge fusion strategy when adopting an incremental learning algorithm based on knowledge distillation. When the teacher model transfers knowledge to the student model, it not only transfers the soft label of the prediction result, but also transfers the feature representation of the middle layer of the model. The middle layer features of the teacher model and the student model are fused through the feature fusion layer. The formula is: ,in is the fused feature, and are the intermediate layer features of the teacher model and the student model respectively, For the fusion coefficient, the optimal value is determined through experiments, and the fused features are used for subsequent training and updating, so that the student model can better absorb the knowledge of the teacher model and optimize it in combination with the characteristics of the new data.

Images