Method and device for detecting abnormal points and abnormal clusters based on time-series network traffic

Abstract

The application discloses a time series network traffic integrated anomaly point and anomaly cluster detection method and device, relates to the field of network security, and comprises the following steps: marking target network traffic data to obtain a network traffic data set; an initial network traffic anomaly detection model is constructed and trained; isolated forest algorithm and local outlier factor algorithm of the trained anomaly detection model are used to detect global anomaly points and local anomaly points respectively; density clustering algorithm and K-means clustering algorithm are used to detect non-spherical cluster anomaly clusters and spherical cluster anomaly clusters respectively; the confidence weight of the initial model parameters is determined according to the anomaly points and the anomaly clusters, and the initial model parameters are adjusted to obtain target model parameters, so that a target network traffic anomaly detection model is constructed based on the target model parameters to detect the network traffic to be detected. Through the detection of network traffic anomaly points and anomaly clusters by various detection algorithms, the robustness and generalization of time series network traffic anomaly detection in complex network attack scenarios are improved.

Description

Technical Field

[0001] This invention relates to the field of network security, and in particular to a method and apparatus for detecting anomalies and clusters based on time-series network traffic integration. Background Technology

[0002] Critical information infrastructure is a vital national asset. Damage, loss of function, or data leakage can lead not only to financial losses but also severely impact the stable operation of the economy and society. Therefore, utilizing technological means to maintain network security and safeguard national cyberspace sovereignty is essential. With the increasing complexity, automation, and intelligence of cyberattacks, new attack types are constantly emerging, showing a trend of expanding scale and diversification. This makes traditional single-mode models limited and challenging in detecting anomalies in network traffic. The importance of anomaly detection lies in its ability to help promptly detect and identify malicious attacks, such as intrusions and DoS (Denial of Service) attacks, enabling corresponding defensive measures and protecting network security. However, traditional single-mode models are often susceptible to noise interference in complex and ever-changing network environments, failing to effectively handle large-scale network traffic data, leading to decreased detection accuracy and efficiency.

[0003] Temporal network traffic anomaly detection plays a crucial role in real-time monitoring and analysis of network data traffic, and in the timely detection of various network attacks. To overcome these challenges, numerous deep learning-based network traffic anomaly detection methods have been developed. However, global traffic packet mining methods based on convolutional attention mechanisms are prone to errors in detecting local anomalies. Stacked denoising autoencoder algorithms, long short-term memory network algorithms, anomaly intrusion detection models based on residual networks, and methods based on inverse training autoencoders are highly dependent on data. Differences in traffic feature distributions under different network attack scenarios lead to poor anomaly detection performance. Deep learning models also exhibit poor interpretability in anomaly detection and require large amounts of data for training. When using random forests for network traffic anomaly detection, data imbalance leads to poor performance, and the numerous features and dimensions of network traffic data make it difficult to model a good random forest. Network anomaly detection methods based on density peak clustering have certain problems in terms of model density parameter sensitivity and generalization. Time-series Outlier Detection System (TODS) relies heavily on prior knowledge of the network domain, and its anomaly detection algorithm is sensitive to parameter selection. Furthermore, TODS requires high-quality time-series data, but in real-world scenarios, the quality of network traffic data varies significantly, potentially leading to poor anomaly detection performance. In summary, current network traffic anomaly detection methods are limited by the constraints of their model algorithms. Therefore, designing a more general network traffic anomaly detection method to improve anomaly detection effectiveness remains a challenge in this field. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method and apparatus for detecting anomalies and clusters in time-series network traffic. This method utilizes multiple detection algorithms to specifically detect anomalies and clusters in network traffic, improving the robustness and generalization of time-series network traffic anomaly detection in complex network attack scenarios. Furthermore, by integrating the detection methods for anomalies and clusters in network traffic data, combined with parameter tuning and integration methods, it can effectively detect network traffic anomalies. The specific solution is as follows:

[0005] Firstly, this application provides a method for detecting anomalies and clusters in time-series network traffic, including:

[0006] The initial network traffic data of the current scene is collected, and the collected initial network traffic data is preprocessed according to a preset data processing flow. Then, the target network traffic data obtained after preprocessing is labeled using a data labeling tool to obtain the network traffic dataset of the current scene.

[0007] An initial network traffic anomaly detection model is constructed based on a preset anomaly detection algorithm, and the initial network traffic anomaly detection model is trained using the training set in the network traffic dataset to obtain a trained anomaly detection model.

[0008] The target isolated forest anomaly detection algorithm and the target local outlier anomaly detection algorithm of the trained anomaly detection model are used to detect global and local anomalies in the labeled data of the network traffic dataset, respectively, and the network traffic anomalies in the current scene are obtained based on the global and local anomalies.

[0009] The target density clustering algorithm and the target K-means clustering algorithm of the trained anomaly detection model are used respectively to detect non-spherical cluster anomaly clusters and spherical cluster anomaly clusters of network traffic data clusters in the labeled data of the network traffic dataset, so as to obtain the network traffic anomaly clusters of the current scenario based on the non-spherical cluster anomaly clusters and the spherical cluster anomaly clusters.

[0010] The confidence weights of the initial model parameters of the initial network traffic anomaly detection model are determined based on the detected network traffic anomaly points and network traffic anomaly clusters. The initial model parameters of the initial network traffic anomaly detection model are then adjusted based on the confidence weights and a preset model adjustment rule to obtain target model parameters. The target network traffic anomaly detection model constructed based on the target model parameters is then used to detect the network traffic to be detected.

[0011] Optionally, before training the initial network traffic anomaly detection model using the training set in the network traffic dataset to obtain the trained anomaly detection model, the method further includes:

[0012] Determine the proportion of abnormal data in the network traffic data of the tagged data, and process the network traffic data of the tagged data using statistical methods such as box plots, normal distribution, and standard scores based on the proportion of abnormal data;

[0013] The anomaly ratio of the initial isolated forest anomaly detection algorithm is determined based on the processed network traffic data, and the target isolated forest anomaly detection algorithm of the initial network traffic anomaly detection model is determined based on the anomaly ratio.

[0014] Optionally, before training the initial network traffic anomaly detection model using the training set in the network traffic dataset to obtain the trained anomaly detection model, the method further includes:

[0015] Based on the processed network traffic data, the anomaly ratio and the number of anomaly clusters of the initial local outlier anomaly detection algorithm are determined, and the target local outlier anomaly detection algorithm of the initial network traffic anomaly detection model is determined based on the anomaly ratio and the number of anomaly clusters.

[0016] Optionally, the step of using the target isolated forest anomaly detection algorithm and the target local outlier anomaly detection algorithm of the trained anomaly detection model to detect global and local anomalies in the labeled data of the network traffic dataset includes:

[0017] The network traffic data in the labeled data of the network traffic dataset is divided into several groups of target data based on a preset data partitioning rule;

[0018] The target isolated forest anomaly detection algorithm and the target local outlier anomaly detection algorithm of the trained anomaly detection model are used to detect the corresponding global anomalies and local anomalies in the several sets of target data in parallel.

[0019] The target data sets are either time window data sets or subsequence data sets.

[0020] Optionally, before training the initial network traffic anomaly detection model using the training set in the network traffic dataset to obtain the trained anomaly detection model, the method further includes:

[0021] Determine the neighborhood density of network traffic data in the tagged data, and process the network traffic data of the tagged data using statistical methods such as box plots, normal distribution, and standard scores based on the neighborhood density;

[0022] The scanning radius and number of cluster samples of the initial density clustering algorithm are determined based on the processed network traffic data, and the target density clustering algorithm of the initial network traffic anomaly detection model is determined based on the scanning radius and the number of cluster samples.

[0023] Optionally, before training the initial network traffic anomaly detection model using the training set in the network traffic dataset to obtain the trained anomaly detection model, the method further includes:

[0024] The proportion of abnormal data in the network traffic data of the labeled data is determined, and the number of clusters in the initial K-means clustering algorithm is determined based on the proportion of abnormal data, the list of preset clusters corresponding to preset attack scenarios and the target density clustering algorithm, so as to determine the target K-means clustering algorithm of the initial network traffic anomaly detection model based on the number of clusters.

[0025] Optionally, the step of determining the confidence weights of the initial model parameters of the initial network traffic anomaly detection model based on the detected network traffic anomaly points and network traffic anomaly clusters, and adjusting the initial model parameters of the initial network traffic anomaly detection model according to the confidence weights based on a preset model adjustment rule to obtain the target model parameters, includes:

[0026] Based on the detected network traffic anomalies, the F-beta value and confusion matrix of the initial network traffic anomaly detection model are determined, and the first model parameters of the initial network traffic anomaly detection model are determined based on the F-beta value and the confusion matrix.

[0027] The first confidence weight of the initial network traffic anomaly detection model is determined based on the F-beta value and the confusion matrix, and the initial model parameters of the initial network traffic anomaly detection model are adjusted based on the first confidence weight to obtain the second model parameters;

[0028] Based on the detected network traffic anomaly clusters, the accuracy, recall, F1 score, and cluster profile coefficient of the initial network traffic anomaly detection model are determined to determine the third model parameters of the initial network traffic anomaly detection model.

[0029] The second confidence weight of the initial network traffic anomaly detection model is determined based on the accuracy, recall, F1 score, and cluster profile coefficient of the initial network traffic anomaly detection model, and the initial model parameters of the initial network traffic anomaly detection model are adjusted based on the second confidence weight to obtain the fourth model parameters.

[0030] The target model parameters are determined based on the first model parameters, the second model parameters, the third model parameters, and the fourth model parameters;

[0031] Accordingly, the detection of network traffic to be detected using the target network traffic anomaly detection model constructed based on the target model parameters includes:

[0032] The target network traffic anomaly detection model is used to determine network traffic anomaly points and network traffic anomaly clusters of the network traffic to be detected, and to determine a first anomaly score of the network traffic anomaly points and a second anomaly score of the network traffic anomaly clusters, so as to determine the target anomaly score of the network traffic to be detected based on the first anomaly score, the second anomaly score, the first confidence weight and the second confidence weight.

[0033] Secondly, this application provides a device for detecting anomalies and clusters in time-series network traffic, comprising:

[0034] The data processing module is used to collect initial network traffic data of the current scene, preprocess the collected initial network traffic data based on a preset data processing process, and use a data annotation tool to annotate the target network traffic data obtained after preprocessing to obtain the network traffic dataset of the current scene.

[0035] The model training module is used to construct an initial network traffic anomaly detection model based on a preset anomaly detection algorithm, and to train the initial network traffic anomaly detection model using the training set in the network traffic dataset to obtain a trained anomaly detection model.

[0036] The anomaly detection module is used to detect global and local anomalies in the labeled data of the network traffic dataset using the target isolated forest anomaly detection algorithm and the target local outlier anomaly detection algorithm of the trained anomaly detection model, respectively, and to obtain the network traffic anomalies in the current scene based on the global and local anomalies.

[0037] An anomaly cluster detection module is used to detect non-spherical cluster anomaly clusters and spherical cluster anomaly clusters in the labeled data of the network traffic dataset using the target density clustering algorithm and the target K-means clustering algorithm of the trained anomaly detection model, respectively, so as to obtain the network traffic anomaly clusters in the current scenario based on the non-spherical cluster anomaly clusters and the spherical cluster anomaly clusters.

[0038] The parameter adjustment module is used to determine the confidence weight of the initial model parameters of the initial network traffic anomaly detection model based on the detected network traffic anomaly points and network traffic anomaly clusters, and adjust the initial model parameters of the initial network traffic anomaly detection model according to the confidence weight and a preset model adjustment rule to obtain target model parameters, so as to use the target network traffic anomaly detection model constructed based on the target model parameters to detect the network traffic to be detected.

[0039] Optionally, the model training module further includes:

[0040] The second data processing unit is used to determine the neighborhood density of network traffic data in the tagged data, and to process the network traffic data of the tagged data using statistical methods such as box plots, normal distribution, and standard scores based on the neighborhood density.

[0041] The third algorithm determination unit is used to determine the scanning radius and the number of cluster samples of the initial density clustering algorithm based on the processed network traffic data, so as to determine the target density clustering algorithm of the initial network traffic anomaly detection model based on the scanning radius and the number of cluster samples.

[0042] Optionally, the model training module further includes:

[0043] The fourth algorithm determination unit is used to determine the proportion of abnormal data in the network traffic data of the labeled data, and to determine the number of clusters in the initial K-means clustering algorithm based on the proportion of abnormal data, the list of preset cluster numbers corresponding to preset attack scenarios and the target density clustering algorithm, so as to determine the target K-means clustering algorithm of the initial network traffic anomaly detection model based on the number of clusters.

[0044] Thirdly, this application provides an electronic device, which includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the aforementioned method for detecting anomalies and clusters based on time-series network traffic integration.

[0045] Fourthly, this application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned method for detecting anomalies and clusters based on time-series network traffic integration.

[0046] This application collects initial network traffic data of the current scene, preprocesses the collected initial network traffic data according to a preset data processing flow, and labels the preprocessed target network traffic data using a data labeling tool to obtain a network traffic dataset for the current scene; constructs an initial network traffic anomaly detection model according to a preset anomaly detection algorithm, and trains the initial network traffic anomaly detection model using the training set in the network traffic dataset to obtain a trained anomaly detection model; uses the target isolated forest anomaly detection algorithm and the target local outlier anomaly detection algorithm of the trained anomaly detection model to detect global and local anomalies in the labeled data of the network traffic dataset, respectively, and obtains the current scene's... Network traffic anomalies are identified. The training anomaly detection model utilizes both the target density clustering algorithm and the target K-means clustering algorithm to detect non-spherical and spherical anomaly clusters in the labeled data of the network traffic dataset. Based on these non-spherical and spherical anomaly clusters, the network traffic anomaly clusters of the current scenario are obtained. The confidence weights of the initial model parameters of the initial network traffic anomaly detection model are determined based on the detected network traffic anomalies and the network traffic anomaly clusters. The initial model parameters of the initial network traffic anomaly detection model are then adjusted according to the confidence weights and a preset model adjustment rule to obtain target model parameters. These target model parameters are then used to detect the network traffic to be detected using the target network traffic anomaly detection model constructed based on the target model parameters. By employing multiple detection algorithms to specifically detect anomalies and clusters in network traffic, the robustness and generalization of time-series network traffic anomaly detection in complex network attack scenarios are improved. Furthermore, by integrating detection methods for network traffic anomalies and clusters, combined with parameter tuning and integration methods, network traffic anomaly detection can be effectively achieved. Compared to current anomaly detection methods, combining different anomaly detection algorithms enables more comprehensive analysis and monitoring of network traffic, improves sensitivity to abnormal behavior, and allows for timely responses, which is beneficial for protecting network security and stability. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 This application provides a flowchart of a method for detecting anomalies and clusters based on time-series network traffic integration;

[0049] Figure 2 This application provides an implementation flowchart of a method for detecting anomalies and clusters based on time-series network traffic integration;

[0050] Figure 3 This application provides a schematic diagram of network traffic anomaly detection results;

[0051] Figure 4 This application provides an iterative schematic diagram of a network traffic anomaly detection model.

[0052] Figure 5 A flowchart illustrating a specific network traffic anomaly detection model training method provided in this application;

[0053] Figure 6 A schematic diagram of a time-series network traffic integrated anomaly point and anomaly cluster detection device provided in this application;

[0054] Figure 7 This application provides a structural diagram of an electronic device. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] The increasing scale and variety of cyberattacks have led to limitations and challenges in traditional single-model network traffic anomaly detection. Traditional single-model approaches are often susceptible to noise and ineffective in handling large-scale network traffic data in complex and dynamic network environments, resulting in decreased detection accuracy and efficiency. This application addresses these challenges by employing multiple detection algorithms to specifically detect anomalies and clusters in network traffic, improving the robustness and generalization of time-series network traffic anomaly detection in complex attack scenarios. Furthermore, by integrating anomaly detection methods for network traffic data and combining parameter tuning and ensemble techniques, it achieves effective anomaly detection in network traffic.

[0057] See Figure 1 As shown in the figure, this invention discloses a method for detecting anomalies and anomaly clusters based on time-series network traffic integration, including:

[0058] Step S11: Collect initial network traffic data of the current scene, preprocess the collected initial network traffic data based on a preset data processing flow, and use a data annotation tool to annotate the target network traffic data obtained after preprocessing to obtain the network traffic dataset of the current scene.

[0059] In this embodiment, the initial network traffic data for the current scenario is first collected according to the pre-configured data acquisition layer. This network traffic data includes time-series network traffic, device assets, and external knowledge base ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) indicator data. The collected indicator data features are then preprocessed using a pre-defined data processing workflow, including data cleaning, handling missing values, attribute reduction, data dimensionality reduction, and normalization. It should be noted that to further achieve automated annotation, the network traffic, device assets, and external knowledge base ATT&CK indicators need to be integrated. Specifically, the Docano data annotation tool can be used to label the data to obtain the network traffic dataset for the current scenario, and the labeled network traffic dataset is then divided into a training set, labeled data, and a test set. The ATT&CK matrix is ​​a framework for describing and classifying threat behaviors. This matrix lists various types of tactics and techniques that attackers may use, providing detailed descriptions and examples to help security professionals understand and combat these threats.

[0060] Step S12: Construct an initial network traffic anomaly detection model based on a preset anomaly detection algorithm, and train the initial network traffic anomaly detection model using the training set in the network traffic dataset to obtain a trained anomaly detection model.

[0061] In this embodiment, an initial network traffic anomaly detection model can be constructed using the Isolation Forest algorithm, the Local Outlier Factor algorithm, the Density-Based Spatial Clustering of Applications with Noise algorithm, and the K-means clustering algorithm. The initial network traffic anomaly detection model is then trained using the training set in the network traffic dataset to obtain the trained anomaly detection model. The algorithm corresponding to the trained anomaly detection model is the determined target anomaly detection algorithm.

[0062] Step S13: Use the target isolated forest anomaly detection algorithm and the target local outlier anomaly detection algorithm of the trained anomaly detection model to detect global and local anomalies in the labeled data of the network traffic dataset, and obtain the network traffic anomalies of the current scene based on the global and local anomalies.

[0063] In this embodiment, in the detection of anomalies in network traffic data, the target isolated forest anomaly detection algorithm and the target local outlier anomaly detection algorithm of the trained anomaly detection model can be used to detect global and local anomalies in the labeled data of the network traffic dataset, and the network traffic anomalies in the current scenario can be obtained based on the global and local anomalies.

[0064] For global outlier detection, the IForset algorithm can be used; for local outlier detection, the LOF algorithm is chosen. It should be noted that when detecting global and local outliers in labeled network traffic data within a network traffic dataset, the labeled network traffic data can first be divided into several target data groups based on a predefined data partitioning rule. Then, the Isolation Forest and Local Outlier Detection algorithms of the trained anomaly detection model are used to detect corresponding global and local outliers in these target data groups in parallel. These target data groups can be time window data or subsequence data. By dividing the data into multiple time windows or subsequences, and then applying the IForset and LOF algorithms in parallel for anomaly detection, the IForset algorithm can quickly detect overall anomalies globally, while the LOF algorithm is more suitable for detecting local anomalies. By processing each time window or subsequence in parallel, the anomaly detection process can be accelerated, computational resources can be fully utilized, and the efficiency of time-series network traffic anomaly detection can be improved. At the same time, by combining the advantages of IFOorset and LOF algorithms, we can better capture global and local anomalies and improve the accuracy of detection.

[0065] Step S14: Detect non-spherical cluster anomaly clusters and spherical cluster anomaly clusters in the labeled data of the network traffic dataset using the target density clustering algorithm and the target K-means clustering algorithm of the trained anomaly detection model, respectively, so as to obtain the network traffic anomaly clusters of the current scenario based on the non-spherical cluster anomaly clusters and the spherical cluster anomaly clusters.

[0066] In this embodiment, the DBScan algorithm is selected for detecting non-spherical cluster anomalies, and the K-means algorithm is selected for detecting spherical cluster anomalies. The target density clustering algorithm and the target K-means clustering algorithm of the trained anomaly detection model are used to detect non-spherical and spherical cluster anomalies in the labeled network traffic data of the network traffic dataset, respectively, to determine the network traffic anomalies in the current scenario.

[0067] Understandably, the DBScan algorithm, a density-based clustering algorithm, is used to identify non-spherical cluster anomalies in network traffic. It determines clusters by defining neighborhood density and core objects, grouping data points with attainable density into the same cluster. DBScan can effectively detect clusters of arbitrary shapes and is suitable for irregularly distributed network traffic data. In contrast, the K-means algorithm, a distance-based clustering algorithm, can be used to capture spherical cluster anomalies. It divides data points into clusters with similar characteristics and optimizes the clustering results by minimizing the squared distance between data points within a cluster, thus enabling K-means to detect spherical cluster anomalies. Therefore, combining DBScan for capturing non-spherical cluster anomalies with K-means for capturing spherical cluster anomalies can improve the accuracy and adaptability of network traffic anomaly detection.

[0068] like Figure 2 As shown, in this embodiment, in conjunction with step S13 above, to address network traffic anomaly detection under different network attack scenarios, anomaly detection methods (IForset, LOF) and anomaly cluster detection methods (DBScan, K-means) are integrated. Modeling is performed based on the training set, labeled data, and test set divided by the data processing layer. In the training set, the above algorithms are used to model corresponding anomalies or anomaly data. Then, model parameters can be initialized through data distribution to evaluate the modeling effect in labeled data.

[0069] Step S15: Determine the confidence weight of the initial model parameters of the initial network traffic anomaly detection model based on the detected network traffic anomaly points and network traffic anomaly clusters, and adjust the initial model parameters of the initial network traffic anomaly detection model according to the confidence weight and a preset model adjustment rule to obtain target model parameters, so as to use the target network traffic anomaly detection model constructed based on the target model parameters to detect the network traffic to be detected.

[0070] In this embodiment, the initial model parameters of the initial network traffic anomaly detection model are adjusted using labeled data from the network traffic dataset to obtain the final detection model. The confidence weights of the initial model parameters can be determined based on the detected anomalies and anomaly clusters, and the initial model parameters are adjusted according to these confidence weights. Specifically:

[0071] First, based on the detected network traffic anomalies, the F-beta value and confusion matrix of the initial network traffic anomaly detection model are determined. The first model parameters of the initial model are then determined based on these F-beta values ​​and the confusion matrix. Next, the first confidence weights of the initial model are determined based on the F-beta values ​​and the confusion matrix, and the initial model parameters are adjusted based on these first confidence weights to obtain the second model parameters. For anomaly detection, modeling is performed using a training set, labeled data, and a test set partitioned by the data processing layer. In the training set, the IForset algorithm is used to model global anomalies, and the LOF algorithm is used to model local anomalies. Then, the model parameters are initialized using the data distribution, and the modeling effect is evaluated in the labeled data. By comparing the anomalies identified by the model with those automatically labeled, the F-beta value and confusion matrix are calculated to determine the parameters to retain for the current anomaly detection model. Then, the confidence weights under the current parameters are obtained using the F-beta algorithm, and the optimal anomaly model parameters are selected through iterative optimization. This allows for adaptive adjustment of the confidence weights for different algorithm models.

[0072] For anomaly clusters, the accuracy, recall, F1 score, and cluster profile coefficient of the initial network traffic anomaly detection model are first determined based on the detected network traffic anomaly clusters to establish the third model parameter. Then, the second confidence weight of the initial network traffic anomaly detection model is determined based on the accuracy, recall, F1 score, and cluster profile coefficient, and the initial model parameters are adjusted based on the second confidence weight to obtain the fourth model parameter. Based on these parameters, the target model parameters are determined according to the first, second, third, and fourth model parameters. For anomaly cluster detection, the DBScan algorithm is used to detect non-spherical clusters in the training set, while the K-means algorithm is used to detect spherical clusters, and the modeling effect is evaluated. By comparing the anomalies identified by the model with those automatically labeled, the accuracy, recall, F1 score, and cluster profile coefficient are calculated to determine the parameters to retain for the current anomaly cluster detection model. Then, the confidence weights of different algorithm models are adaptively adjusted according to the accuracy, recall, and F1 score algorithms, and the optimal anomaly cluster model parameters are selected through iterative optimization of the model.

[0073] It is understandable that, after determining the target model parameters, the target network traffic anomaly detection model can be used to identify network traffic anomalies and network traffic anomaly clusters in the network traffic to be detected, and to determine the first anomaly score of the network traffic anomalies and the second anomaly score of the network traffic anomaly clusters. Based on the first anomaly score, the second anomaly score, the first confidence weight, and the second confidence weight, the target anomaly score of the network traffic to be detected can be determined. It is also understood that, since network traffic anomalies in this embodiment are determined by corresponding global and local anomaly detection algorithms, the aforementioned first anomaly score includes two corresponding anomaly scores. Correspondingly, the aforementioned second anomaly score, the first confidence weight, and the second confidence weight also include two corresponding anomaly scores or weights. Specifically, the aforementioned target anomaly score K is:

[0074]

[0075] Where n is the number of anomaly scores and confidence weights, which is 4 in this embodiment. Z i Z0 represents the anomaly score output by the target network traffic anomaly detection model. Since this embodiment utilizes pairwise integration of four algorithms, from the perspective of anomaly points, Z0 represents the anomaly score generated by the IForset algorithm, and Z1 represents the anomaly score generated by the LOF algorithm. From the perspective of anomaly clusters, Z2 represents the anomaly score generated by the DBScan algorithm, and Z3 represents the anomaly score generated by the K-means algorithm. Similarly, W... i These are the confidence weights for detecting anomalies in network traffic.

[0076] It should be pointed out that, as Figure 3As shown, this embodiment employs the Bagging (Bootstrap Aggregating) ensemble method, combining anomaly detection and anomaly cluster detection algorithms for time-series network traffic detection. The Bagging algorithm generates multiple different sub-training sets by sampling the training set with replacement (bootstrap sampling). A base classifier is then trained for each sub-training set. Finally, the prediction results of the base classifiers are ensembled. By constructing multiple base classifiers and aggregating the prediction results, the stability and generalization ability of the model can be improved. Furthermore, an adaptive parameter tuning method is used to ensure that each algorithm achieves optimal performance. When calculating the target anomaly score, since the anomaly scores Z generated by different anomaly point and anomaly cluster algorithms have different meanings, they need to be standardized and normalized. First, the anomaly scores Z are standardized and normalized to ensure they have the same scale and range. This makes the anomaly scores from different algorithms comparable and facilitates weighted summation. Then, the confidence weights W are normalized using the F1 score from the labeled data. This normalization process limits the weights to a reasonable range, ensuring the sum of the algorithm model weights equals 1. This better controls the contribution of different algorithms to the final result. Finally, the anomaly score Z and the confidence weights W are weighted and summed to obtain the final target anomaly score K. The target anomaly score K comprehensively considers the anomaly detection results of each algorithm and their confidence weights, thus enabling a more accurate assessment of anomalies in time-series network traffic. This ensemble method allows for more comprehensive and accurate detection of anomalies in time-series network traffic. By integrating the advantages of multiple models or algorithms through ensemble learning, it can improve anomaly detection performance, reduce false positive rates, and effectively handle large-scale network traffic data. Furthermore, as... Figure 4 As shown, in order to obtain the optimal anomaly detection algorithm and anomaly cluster detection algorithm, an iterative and adaptive method is adopted. By continuously iterating and adjusting the parameters of the algorithm, the optimal configuration of each algorithm is found to improve the performance of anomaly detection. Furthermore, the Bagging method of ensemble learning is used to combine the prediction results of multiple anomaly detection algorithms, thereby reducing the bias and variance of individual algorithms and improving the accuracy and stability of network traffic anomaly detection.

[0077] In another embodiment, it is understood that Boost ensemble can also be used in the integrated anomaly detection and anomaly cluster detection processes. For anomaly detection, the IForset algorithm is used for global anomaly detection in network traffic. The features from the IForset training process are cloned into two parts: one part serves as the feature input for local anomaly detection in the LOF algorithm, and the other part serves as the adaptive parameter optimization for the IForset algorithm. Similarly, the features from the LOF algorithm training process are cloned into two parts: one part serves as the anomaly score for anomaly detection, and the other part serves as the adaptive parameter optimization for the LOF algorithm itself. For anomaly cluster detection, the DBScan algorithm is used for non-spherical cluster detection in network traffic. The features from the DBScan training process are cloned into two parts: one part serves as the adaptive parameter optimization for the DBScan algorithm, and the other part serves as the feature input for local anomaly detection in the K-means algorithm. Since the K-means algorithm requires a pre-defined k value (number of clusters), the neighborhood of the core cluster number k in the K-means algorithm is set according to the number of clusters in the DBScan algorithm. Then, the clustering effect is evaluated based on the silhouette coefficient, thereby finally selecting the optimal k value. The K-Means algorithm outputs the anomaly score.

[0078] Through the above technical solution, this embodiment integrates anomaly point and anomaly cluster methods for time-series network traffic data anomaly detection under different network attack scenarios. The data acquisition layer collects network traffic, device asset, and external knowledge base ATT&CK metrics data for different network scenarios. The collected metrics data undergoes data cleaning, missing value handling, dimensionality reduction, and normalization. The processed metrics data is then automatically labeled, dividing the data into training, labeled, and test sets to exhibit different characteristics of network traffic under different attack scenarios. Regarding network traffic anomaly point detection, the IForset algorithm is used to capture anomalies in global network traffic data, and the LOF algorithm is used to capture anomalies in local network traffic data. Regarding network traffic anomaly cluster detection, the DBScan algorithm is used to capture non-spherical cluster anomalies in network traffic data, and K-means is used to capture spherical cluster anomalies in network traffic data. This method integrates network traffic data anomalies and anomaly clusters. During the iteration process, it adaptively obtains the optimal parameters of the current algorithm from the training set, and simultaneously adaptively obtains the algorithm confidence weights from the results of labeled data. It retrieves the optimal parameters and confidence scores of each algorithm in the optimal parameter iteration layer, and combines multiple algorithms using a Bagging ensemble approach to obtain a network traffic anomaly score. The anomaly score confidence weights are then iteratively optimized to finally detect network traffic anomalies. This improves the robustness and generalization of time-series network traffic anomaly detection in complex network attack scenarios. By combining parameter tuning and ensemble methods, it can effectively detect network traffic anomalies.

[0079] As can be seen from the previous embodiment, this application can specifically detect anomalies and abnormal clusters in network traffic using various detection algorithms. Next, this embodiment will describe in detail the process of training a network traffic anomaly detection model based on the aforementioned algorithms. See also... Figure 5 As shown in the figure, an embodiment of the present invention discloses a method for training a network traffic anomaly detection model, including:

[0080] Step S21: Determine the proportion of abnormal data in the network traffic data of the tagged data, and process the network traffic data of the tagged data using statistical methods such as box plots, normal distribution, and standard scores based on the proportion of abnormal data; determine the abnormal ratio of the initial isolated forest anomaly detection algorithm based on the processed network traffic data, and determine the target isolated forest anomaly detection algorithm of the initial network traffic anomaly detection model based on the abnormal ratio.

[0081] In this embodiment, the IForset algorithm can be used for global outlier detection. First, based on the proportion of outliers in the dataset, box plots, normal distributions, and Z-score statistics can be used to preprocess the data and determine the core parameter of the IForset algorithm, Contamination (outlier ratio). When using the IForset algorithm to detect global feature outliers in the network traffic dataset, a feature x (e.g., network traffic, device assets, and ATT&CK tactical features) is randomly selected. A cut point is randomly chosen between the minimum and maximum values ​​of this feature, dividing the training samples into a left and right subtree. This process is repeated for both the left and right subtrees until the maximum depth H is reached or only a single sample point remains. During this process, c(n) is obtained by calculating the average search length of failed searches in the binary search tree. Based on the IForset forest setting of the outlier score threshold, the expected value of the path length h(x) of node x across multiple isolated trees is calculated, and finally, the outlier score s(x,n) of node x is obtained. It is understood that if a node's outlier score is greater than the outlier score threshold, the node is determined as a global outlier in the network traffic data. The abnormal score s(x,n) is calculated using the following formula:

[0082]

[0083] In Isolation Forest, since it is trained by cutting subsamples into trees, a tree is generated for each subsample. h(x) is the height of x in each tree, E(h(x)) is the average height of x in each tree, and c(n) is the average path length (or height) of all samples given a sample size n, used to standardize the path length h(x) of sample x. For anomaly score judgment, for example: when the score is close to 1, the path length is very small, and the data point is easily isolated, resulting in an anomaly; when the score is less than 0.5, the path length increases, resulting in a normal data point; if all observations have anomalies around 0.5, then there is no anomaly. Isolation Forest calculates the anomaly score for each tree and averages it across isolated trees. Typically, anomalies have shorter path lengths because they are easier to segment within the tree compared to normal points, allowing for anomaly isolation in fewer steps than normal observations. Based on these steps, the IFOorset algorithm can be trained for global feature anomaly detection, accurately identifying global anomalies in network traffic data.

[0084] Step S22: Determine the anomaly ratio and the number of anomaly clusters of the initial local outlier anomaly detection algorithm based on the processed network traffic data, and determine the target local outlier anomaly detection algorithm of the initial network traffic anomaly detection model based on the anomaly ratio and the number of anomaly clusters.

[0085] In this embodiment, the LOF algorithm is selected for local outlier detection. First, based on the proportion of outlier data in the dataset, box plots, normal distribution, and Z-score statistical methods are used to pre-initialize and set the core parameters of the LOF algorithm: Contamination (outlier ratio) and n_neighbors (number of outlier clusters). The LOF algorithm is then used to detect local outliers in the network traffic dataset. The network traffic, equipment assets, and ATT&CK tactical features in the dataset are fused. Based on the selected Euclidean distance, the distance between each data sample and its k neighboring data samples is calculated. Based on this distance, the local reachability density (lrd(p)) is calculated for each data sample p, where rdist is the reachability distance between data samples. Then, the local outlier factor (lof(p)) threshold can be obtained from the result of the local reachability density calculation.

[0086]

[0087]

[0088] If a sample point has a relatively low density, but its surrounding neighboring sample points have a higher density, then that sample point is considered an outlier. LOF considers the relative density difference between this point and other points in its neighborhood; the larger the density difference value, the more likely it is a local anomaly in network traffic. The LOF algorithm can identify local anomalies that do not conform to normal patterns, promptly detecting potential security threats so that appropriate defensive measures can be taken.

[0089] In summary, anomaly detection is achieved using both the global IForset algorithm and the local LOF algorithm. The IForset algorithm efficiently detects overall anomalies, while the LOF algorithm considers the relationships between data points within a local scope, using local density to assess the degree of anomaly. By combining the advantages of both global and local anomaly detection, accurate and reliable network traffic anomaly detection can be achieved in various network attack scenarios.

[0090] Step S23: Determine the neighborhood density of network traffic data in the labeled data, and process the network traffic data of the labeled data using statistical methods such as box plots, normal distribution, and standard scores based on the neighborhood density; determine the scanning radius and the number of cluster samples of the initial density clustering algorithm based on the processed network traffic data, and determine the target density clustering algorithm of the initial network traffic anomaly detection model based on the scanning radius and the number of cluster samples.

[0091] In this embodiment, the DBScan algorithm is used to detect non-spherical cluster anomalies. First, based on the data domain density, the core parameters eps (radius) and min_samples (number of cluster samples) of the DBScan algorithm are initialized using the box plot, normal distribution, and Z-score methods. Then, the target density clustering algorithm of the initial network traffic anomaly detection model is determined based on the radius and the number of cluster samples. In subsequent detection, the DBScan algorithm can be used to perform non-spherical cluster anomaly detection on the network traffic dataset. The sample data features (network traffic, equipment assets, and ATT&CK tactical features) are vectorized, and an unclassified data point is selected as the current core point. A circle is drawn around the core point, and the number of all data points in the ε-neighborhood of the current core point is calculated. If the minimum number of data points is less than MinPts, the current core point is marked as an anomaly and marked as visited, and a new cluster is created. For all unvisited data points in the ε-neighborhood of the current core point, if the point is a core point, all data points in its ε-neighborhood are added to the new cluster. The above operation is repeated until there are no unvisited core points in the ε-neighborhood of the current core point, and until all data points have been visited. Each cluster obtained is the clustering result, and the unclassified data points are clustered according to the network traffic data features and then marked as anomaly clusters.

[0092] The DBSCAN clustering algorithm can divide sample points in a dataset into different clusters and identify anomalies. It is robust to noisy data and is helpful in classifying the neighborhood density around a discovered network attack or point into different intrusions.

[0093] Step S24: Determine the proportion of abnormal data in the network traffic data of the tagged data, and determine the number of clusters in the initial K-means clustering algorithm based on the proportion of abnormal data, the list of preset clusters corresponding to the preset attack scenario and the target density clustering algorithm, so as to determine the target K-means clustering algorithm of the initial network traffic anomaly detection model based on the number of clusters.

[0094] In this embodiment, for the detection of spherical cluster anomalies, the K-means algorithm is selected. First, based on the proportion of data anomalies and different network attack scenarios (such as DDoS attacks, Trojan attacks, and vulnerability exploits), the core parameter k (number of clusters) of the DBScan algorithm is initialized, combining the number of clusters in the DBScan algorithm and the list of k values ​​for different network attack scenarios. Then, based on the number of clusters, the target K-means clustering algorithm for the initial network traffic anomaly detection model is determined. During subsequent anomaly detection, k data points are randomly selected as initial cluster centers. For each data point, the distance between it and each cluster center is calculated, and the data point is assigned to the cluster to which the nearest cluster center belongs. For each cluster, the average value of all its data points is calculated, and this average value is used as the new cluster center. This process continues until the cluster centers no longer change or a predetermined number of iterations is reached, ultimately forming k-1 anomaly clusters.

[0095] The K-means algorithm divides the sample points in the dataset into K non-overlapping clusters. Through iterative optimization, it assigns sample points to the nearest cluster and updates the centroid of each cluster, minimizing the distance between a sample point and its cluster centroid. This method can effectively identify anomalous clusters of spherical clusters, allowing for the timely detection of potential security threats and the implementation of appropriate mitigation measures.

[0096] In summary, this study employs the DBScan algorithm for non-spherical clusters and the K-means algorithm for spherical clusters to capture anomaly clusters in network traffic. The DBScan algorithm efficiently detects non-spherical cluster anomaly clusters, while the K-means algorithm quickly clusters spherical clusters. By combining the density characteristics of both spherical and non-spherical clusters to assess the degree of anomaly, accurate and reliable detection of anomaly clusters in network traffic can be achieved under various network attack scenarios.

[0097] Step S25: Obtain the trained anomaly detection model based on the target isolated forest anomaly detection algorithm, the target local outlier anomaly detection algorithm, the target density clustering algorithm, and the target K-means clustering algorithm.

[0098] Through the above technical solutions, this embodiment employs the IForset algorithm to capture global anomalies in the data and combines it with the LOF algorithm to capture local anomalies, thereby achieving the detection of anomalies in time-series network traffic. Furthermore, the DBScan algorithm is introduced to capture non-spherical cluster anomalies in the data, and the K-means algorithm is combined to capture spherical cluster anomalies in the data, thus achieving the detection of anomaly clusters in time-series network traffic. By combining different anomaly detection algorithms, network traffic can be analyzed and monitored more comprehensively, sensitivity to abnormal behavior can be improved, and timely responses can be made to protect network security and stability.

[0099] See Figure 7 As shown in the illustration, this application also discloses a device for detecting anomalies and clusters based on time-series network traffic integration, comprising:

[0100] The data processing module 11 is used to collect initial network traffic data of the current scene, preprocess the collected initial network traffic data based on a preset data processing process, and use a data annotation tool to annotate the target network traffic data obtained after preprocessing to obtain the network traffic dataset of the current scene.

[0101] The model training module 12 is used to construct an initial network traffic anomaly detection model according to a preset anomaly detection algorithm, and to train the initial network traffic anomaly detection model using the training set in the network traffic dataset to obtain a trained anomaly detection model.

[0102] The anomaly detection module 13 is used to detect global and local anomalies in the labeled data of the network traffic dataset using the target isolated forest anomaly detection algorithm and the target local outlier anomaly detection algorithm of the trained anomaly detection model, respectively, and to obtain the network traffic anomalies of the current scene based on the global and local anomalies.

[0103] The anomaly cluster detection module 14 is used to detect non-spherical cluster anomaly clusters and spherical cluster anomaly clusters in the labeled data of the network traffic dataset using the target density clustering algorithm and the target K-means clustering algorithm of the trained anomaly detection model, respectively, so as to obtain the network traffic anomaly clusters in the current scenario based on the non-spherical cluster anomaly clusters and the spherical cluster anomaly clusters.

[0104] The parameter adjustment module 15 is used to determine the confidence weight of the initial model parameters of the initial network traffic anomaly detection model based on the detected network traffic anomaly points and the network traffic anomaly clusters, and adjust the initial model parameters of the initial network traffic anomaly detection model according to the confidence weight and a preset model adjustment rule to obtain target model parameters, so as to use the target network traffic anomaly detection model constructed based on the target model parameters to detect the network traffic to be detected.

[0105] In this embodiment, initial network traffic data of the current scene is collected, and the collected initial network traffic data is preprocessed according to a preset data processing flow. The preprocessed target network traffic data is then labeled using a data labeling tool to obtain a network traffic dataset. An initial network traffic anomaly detection model is constructed based on a preset anomaly detection algorithm, and the initial network traffic anomaly detection model is trained using the training set in the network traffic dataset to obtain a trained anomaly detection model. The target isolated forest anomaly detection algorithm and the target local outlier anomaly detection algorithm of the trained anomaly detection model are used to detect global and local anomalies in the labeled network traffic data of the network traffic dataset, respectively. Based on the global and local anomalies, the current scene is determined. Network traffic anomalies are identified. The target density clustering algorithm and target K-means clustering algorithm of the trained anomaly detection model are used to detect non-spherical and spherical anomaly clusters in the labeled network traffic data of the network traffic dataset. These non-spherical and spherical anomaly clusters are then used to determine the network traffic anomaly clusters in the current scenario. Based on the detected network traffic anomalies and clusters, the confidence weights of the initial model parameters of the initial network traffic anomaly detection model are determined. The initial model parameters are then adjusted according to the confidence weights and a preset model adjustment rule to obtain the target model parameters. Finally, the target network traffic anomaly detection model constructed based on the target model parameters is used to detect the network traffic to be detected. By employing multiple detection algorithms to specifically detect anomalies and clusters in network traffic, the robustness and generalization of time-series network traffic anomaly detection in complex network attack scenarios are improved. Furthermore, by integrating detection methods for network traffic anomalies and clusters, combined with parameter tuning and integration methods, network traffic anomaly detection can be effectively achieved. By combining different anomaly detection algorithms, network traffic can be analyzed and monitored more comprehensively, improving sensitivity to abnormal behavior and enabling timely responses, which is beneficial for protecting network security and stability.

[0106] In some specific embodiments, the model training module 12 further includes:

[0107] The first data processing unit is used to determine the proportion of abnormal data in the network traffic data of the tagged data, and to process the network traffic data of the tagged data using statistical methods such as box plots, normal distribution, and standard scores based on the proportion of abnormal data.

[0108] The first algorithm determination unit is used to determine the anomaly ratio of the initial isolated forest anomaly detection algorithm based on the processed network traffic data, so as to determine the target isolated forest anomaly detection algorithm of the initial network traffic anomaly detection model based on the anomaly ratio.

[0109] In some specific embodiments, the model training module 12 further includes:

[0110] The second algorithm determination unit is used to determine the anomaly ratio and the number of anomaly clusters of the initial local outlier anomaly detection algorithm based on the processed network traffic data, so as to determine the target local outlier anomaly detection algorithm of the initial network traffic anomaly detection model based on the anomaly ratio and the number of anomaly clusters.

[0111] In some specific embodiments, the anomaly detection module 13 specifically includes:

[0112] The data partitioning unit is used to divide the network traffic data in the labeled data of the network traffic dataset into several groups of target data based on a preset data partitioning rule;

[0113] The anomaly detection unit is used to detect, in parallel, the target isolated forest anomaly detection algorithm and the target local outlier anomaly detection algorithm in the trained anomaly detection model, respectively, the corresponding global anomalies and local anomalies in the several sets of target data; wherein, the several sets of target data are several sets of time window data or several sets of subsequence data.

[0114] In some specific embodiments, the model training module 12 further includes:

[0115] The second data processing unit is used to determine the neighborhood density of network traffic data in the tagged data, and to process the network traffic data of the tagged data using statistical methods such as box plots, normal distribution, and standard scores based on the neighborhood density.

[0116] The third algorithm determination unit is used to determine the scanning radius and the number of cluster samples of the initial density clustering algorithm based on the processed network traffic data, so as to determine the target density clustering algorithm of the initial network traffic anomaly detection model based on the scanning radius and the number of cluster samples.

[0117] In some specific embodiments, the model training module 12 further includes:

[0118] The fourth algorithm determination unit is used to determine the proportion of abnormal data in the network traffic data of the labeled data, and to determine the number of clusters in the initial K-means clustering algorithm based on the proportion of abnormal data, the list of preset cluster numbers corresponding to preset attack scenarios and the target density clustering algorithm, so as to determine the target K-means clustering algorithm of the initial network traffic anomaly detection model based on the number of clusters.

[0119] In some specific embodiments, the parameter adjustment module 15 specifically includes:

[0120] The first parameter determination unit is used to determine the F-beta value and confusion matrix of the initial network traffic anomaly detection model based on the detected network traffic anomaly points, so as to determine the first model parameters of the initial network traffic anomaly detection model based on the F-beta value and the confusion matrix.

[0121] The second parameter determination unit is used to determine the first confidence weight of the initial network traffic anomaly detection model based on the F-beta value and the confusion matrix, and to adjust the initial model parameters of the initial network traffic anomaly detection model based on the first confidence weight to obtain the second model parameters.

[0122] The third parameter determination unit is used to determine the accuracy, recall, F1 score, and cluster profile coefficient of the initial network traffic anomaly detection model based on the detected network traffic anomaly clusters, so as to determine the third model parameters of the initial network traffic anomaly detection model.

[0123] The fourth parameter determination unit is used to determine the second confidence weight of the initial network traffic anomaly detection model based on the accuracy, recall, F1 score and cluster profile coefficient of the initial network traffic anomaly detection model, and adjust the initial model parameters of the initial network traffic anomaly detection model based on the second confidence weight to obtain the fourth model parameters.

[0124] The fifth parameter determination unit is used to determine the target model parameters based on the first model parameters, the second model parameters, the third model parameters, and the fourth model parameters;

[0125] The traffic monitoring unit is used to determine network traffic anomaly points and network traffic anomaly clusters of the network traffic to be detected using the target network traffic anomaly detection model, and to determine a first anomaly score of the network traffic anomaly points and a second anomaly score of the network traffic anomaly clusters, so as to determine the target anomaly score of the network traffic to be detected based on the first anomaly score, the second anomaly score, the first confidence weight and the second confidence weight.

[0126] Furthermore, embodiments of this application also disclose an electronic device, Figure 7 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0127] Figure 7This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the network traffic anomaly detection method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be a computer.

[0128] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0129] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0130] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the network traffic anomaly detection method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0131] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned network traffic anomaly detection method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0132] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0133] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0135] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0136] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for detecting anomalies and clusters in time-series network traffic, characterized in that, include: The initial network traffic data of the current scene is collected, and the collected initial network traffic data is preprocessed according to a preset data processing flow. The target network traffic data obtained after preprocessing is labeled using a data labeling tool to obtain the network traffic dataset of the current scene. An initial network traffic anomaly detection model is constructed based on a preset anomaly detection algorithm, and the initial network traffic anomaly detection model is trained using the training set in the network traffic dataset to obtain a trained anomaly detection model. The target isolated forest anomaly detection algorithm and the target local outlier anomaly detection algorithm of the trained anomaly detection model are used to detect global and local anomalies in the labeled data of the network traffic dataset, respectively, and the network traffic anomalies in the current scene are obtained based on the global and local anomalies. The target density clustering algorithm and the target K-means clustering algorithm of the trained anomaly detection model are used respectively to detect non-spherical cluster anomaly clusters and spherical cluster anomaly clusters of network traffic data clusters in the labeled data of the network traffic dataset, so as to obtain the network traffic anomaly clusters of the current scenario based on the non-spherical cluster anomaly clusters and the spherical cluster anomaly clusters. The confidence weights of the initial model parameters of the initial network traffic anomaly detection model are determined based on the detected network traffic anomaly points and network traffic anomaly clusters. The initial model parameters of the initial network traffic anomaly detection model are then adjusted based on the confidence weights and a preset model adjustment rule to obtain target model parameters. The target network traffic anomaly detection model constructed based on the target model parameters is then used to detect the network traffic to be detected. The step of determining the confidence weights of the initial model parameters of the initial network traffic anomaly detection model based on the detected network traffic anomaly points and network traffic anomaly clusters, and adjusting the initial model parameters of the initial network traffic anomaly detection model according to the confidence weights and a preset model adjustment rule to obtain the target model parameters, includes: Based on the detected network traffic anomalies, the F-beta value and confusion matrix of the initial network traffic anomaly detection model are determined, and the first model parameters of the initial network traffic anomaly detection model are determined based on the F-beta value and the confusion matrix. The first confidence weight of the initial network traffic anomaly detection model is determined based on the F-beta value and the confusion matrix, and the initial model parameters of the initial network traffic anomaly detection model are adjusted based on the first confidence weight to obtain the second model parameters; Based on the detected network traffic anomaly clusters, the accuracy, recall, F1 score, and cluster profile coefficient of the initial network traffic anomaly detection model are determined to determine the third model parameters of the initial network traffic anomaly detection model. The second confidence weight of the initial network traffic anomaly detection model is determined based on the accuracy, recall, F1 score, and cluster profile coefficient of the initial network traffic anomaly detection model, and the initial model parameters of the initial network traffic anomaly detection model are adjusted based on the second confidence weight to obtain the fourth model parameters. The target model parameters are determined based on the first model parameters, the second model parameters, the third model parameters, and the fourth model parameters; Accordingly, the detection of network traffic to be detected using the target network traffic anomaly detection model constructed based on the target model parameters includes: The target network traffic anomaly detection model is used to determine network traffic anomaly points and network traffic anomaly clusters of the network traffic to be detected, and to determine a first anomaly score of the network traffic anomaly points and a second anomaly score of the network traffic anomaly clusters, so as to determine the target anomaly score of the network traffic to be detected based on the first anomaly score, the second anomaly score, the first confidence weight and the second confidence weight.

2. The method for detecting anomalies and clusters based on time-series network traffic integration according to claim 1, characterized in that, Before training the initial network traffic anomaly detection model using the training set in the network traffic dataset to obtain the trained anomaly detection model, the method further includes: Determine the proportion of abnormal data in the network traffic data of the tagged data, and process the network traffic data of the tagged data using statistical methods such as box plots, normal distribution, and standard scores based on the proportion of abnormal data; The anomaly ratio of the initial isolated forest anomaly detection algorithm is determined based on the processed network traffic data, and the target isolated forest anomaly detection algorithm of the initial network traffic anomaly detection model is determined based on the anomaly ratio.

3. The method for detecting anomalies and clusters based on time-series network traffic integration according to claim 2, characterized in that, Before training the initial network traffic anomaly detection model using the training set in the network traffic dataset to obtain the trained anomaly detection model, the method further includes: Based on the processed network traffic data, the anomaly ratio and the number of anomaly clusters of the initial local outlier anomaly detection algorithm are determined, and the target local outlier anomaly detection algorithm of the initial network traffic anomaly detection model is determined based on the anomaly ratio and the number of anomaly clusters.

4. The method for detecting anomalies and clusters based on time-series network traffic integration according to claim 3, characterized in that, The steps of detecting global and local outliers in the labeled network traffic data of the network traffic dataset using the target isolated forest anomaly detection algorithm and the target local outlier anomaly detection algorithm of the trained anomaly detection model respectively include: The network traffic data in the labeled data of the network traffic dataset is divided into several groups of target data based on a preset data partitioning rule; The target isolated forest anomaly detection algorithm and the target local outlier anomaly detection algorithm of the trained anomaly detection model are used to detect the corresponding global anomalies and local anomalies in the several sets of target data in parallel. The target data sets are either time window data sets or subsequence data sets.

5. The method for detecting anomalies and clusters based on time-series network traffic integration according to claim 1, characterized in that, Before training the initial network traffic anomaly detection model using the training set in the network traffic dataset to obtain the trained anomaly detection model, the method further includes: Determine the neighborhood density of network traffic data in the tagged data, and process the network traffic data of the tagged data using statistical methods such as box plots, normal distribution, and standard scores based on the neighborhood density; The scanning radius and number of cluster samples of the initial density clustering algorithm are determined based on the processed network traffic data, and the target density clustering algorithm of the initial network traffic anomaly detection model is determined based on the scanning radius and the number of cluster samples.

6. The method for detecting anomalies and clusters based on time-series network traffic integration according to claim 5, characterized in that, Before training the initial network traffic anomaly detection model using the training set in the network traffic dataset to obtain the trained anomaly detection model, the method further includes: The proportion of abnormal data in the network traffic data of the labeled data is determined, and the number of clusters in the initial K-means clustering algorithm is determined based on the proportion of abnormal data, the list of preset clusters corresponding to preset attack scenarios and the target density clustering algorithm, so as to determine the target K-means clustering algorithm of the initial network traffic anomaly detection model based on the number of clusters.

7. A device for detecting anomalies and clusters in time-series network traffic, characterized in that, include: The data processing module is used to collect initial network traffic data of the current scene, preprocess the collected initial network traffic data based on a preset data processing process, and use a data annotation tool to annotate the target network traffic data obtained after preprocessing to obtain the network traffic dataset of the current scene. The model training module is used to construct an initial network traffic anomaly detection model based on a preset anomaly detection algorithm, and to train the initial network traffic anomaly detection model using the training set in the network traffic dataset to obtain a trained anomaly detection model. The anomaly detection module is used to detect global and local anomalies in the labeled data of the network traffic dataset using the target isolated forest anomaly detection algorithm and the target local outlier anomaly detection algorithm of the trained anomaly detection model, respectively, and to obtain the network traffic anomalies in the current scene based on the global and local anomalies. An anomaly cluster detection module is used to detect non-spherical cluster anomaly clusters and spherical cluster anomaly clusters in the labeled data of the network traffic dataset using the target density clustering algorithm and the target K-means clustering algorithm of the trained anomaly detection model, respectively, so as to obtain the network traffic anomaly clusters in the current scenario based on the non-spherical cluster anomaly clusters and the spherical cluster anomaly clusters. The parameter adjustment module is used to determine the confidence weight of the initial model parameters of the initial network traffic anomaly detection model based on the detected network traffic anomaly points and network traffic anomaly clusters, and adjust the initial model parameters of the initial network traffic anomaly detection model according to the confidence weight and a preset model adjustment rule to obtain target model parameters, so as to use the target network traffic anomaly detection model constructed based on the target model parameters to detect the network traffic to be detected. The parameter adjustment module includes: The first parameter determination unit is used to determine the F-beta value and confusion matrix of the initial network traffic anomaly detection model based on the detected network traffic anomaly points, so as to determine the first model parameters of the initial network traffic anomaly detection model based on the F-beta value and the confusion matrix. The second parameter determination unit is used to determine the first confidence weight of the initial network traffic anomaly detection model based on the F-beta value and the confusion matrix, and to adjust the initial model parameters of the initial network traffic anomaly detection model based on the first confidence weight to obtain the second model parameters. The third parameter determination unit is used to determine the accuracy, recall, F1 score, and cluster profile coefficient of the initial network traffic anomaly detection model based on the detected network traffic anomaly clusters, so as to determine the third model parameters of the initial network traffic anomaly detection model. The fourth parameter determination unit is used to determine the second confidence weight of the initial network traffic anomaly detection model based on the accuracy, recall, F1 score and cluster profile coefficient of the initial network traffic anomaly detection model, and adjust the initial model parameters of the initial network traffic anomaly detection model based on the second confidence weight to obtain the fourth model parameters. The fifth parameter determination unit is used to determine the target model parameters based on the first model parameters, the second model parameters, the third model parameters, and the fourth model parameters; The traffic monitoring unit is used to determine network traffic anomaly points and network traffic anomaly clusters of the network traffic to be detected using the target network traffic anomaly detection model, and to determine a first anomaly score of the network traffic anomaly points and a second anomaly score of the network traffic anomaly clusters, so as to determine the target anomaly score of the network traffic to be detected based on the first anomaly score, the second anomaly score, the first confidence weight and the second confidence weight.

8. The device for detecting anomalies and clusters based on time-series network traffic integration according to claim 7, characterized in that, The model training module also includes: The second data processing unit is used to determine the neighborhood density of network traffic data in the tagged data, and to process the network traffic data of the tagged data using statistical methods such as box plots, normal distribution, and standard scores based on the neighborhood density. The third algorithm determination unit is used to determine the scanning radius and the number of cluster samples of the initial density clustering algorithm based on the processed network traffic data, so as to determine the target density clustering algorithm of the initial network traffic anomaly detection model based on the scanning radius and the number of cluster samples.

9. The device for detecting anomalies and clusters based on time-series network traffic integration according to claim 8, characterized in that, The model training module also includes: The fourth algorithm determination unit is used to determine the proportion of abnormal data in the network traffic data of the labeled data, and to determine the number of clusters in the initial K-means clustering algorithm based on the proportion of abnormal data, the list of preset cluster numbers corresponding to preset attack scenarios and the target density clustering algorithm, so as to determine the target K-means clustering algorithm of the initial network traffic anomaly detection model based on the number of clusters.

Images