A Network Anomaly Behavior Detection Method Based on Dynamic Data Balancing and Generation

Abstract

The present invention relates to a method for detecting network abnormal behaviors based on dynamic data balancing and generation, including: acquiring original network traffic data, inputting the original network traffic data into a dynamic abnormal data balancing model to obtain an enhanced data set; marking the original network traffic data based on the abnormal data balancing model, performing selective undersampling on the marked data, adjusting the ratio of normal data to abnormal data to obtain a balanced data set, generating new abnormal data according to the balanced data set, and merging it with the balanced data set to obtain an enhanced data set; inputting the enhanced data set into a TimeDiT denoising model for denoising to obtain a denoised target data set, and inputting it into an LSTM-Transformer weighted training model for detecting network abnormal behaviors, weighting the abnormal data in the target data set in combination with a weighted loss function to obtain a detection result. The present invention can improve the detection efficiency and accuracy of network traffic abnormal behaviors.

Description

Technical Field

[0001] The present invention relates to the technical fields of network security and data analysis, and particularly to a method for detecting network abnormal behaviors based on dynamic data balancing and generation. Background Art

[0002] In modern information society, the network has become an important part of people's daily life and work. However, with the increase in network scale and complexity, network abnormal behaviors (such as network attacks, data leakage, etc.) have become increasingly complex and diverse. Effective methods for detecting network abnormal behaviors are of great significance for ensuring network security. In current research, traditional rule-based and statistical detection methods are no longer able to cope with complex network environments, so it is necessary to rely on advanced machine learning and deep learning technologies to improve the accuracy and efficiency of detection.

[0003] Network data has characteristics such as complex format, high dimension, and large data volume. Network traffic data includes various attributes such as various protocols, source addresses, destination addresses, port numbers, timestamps, etc., and is also accompanied by a large amount of noise and redundant information. These characteristics make it difficult for traditional data processing methods to effectively process and analyze network data while ensuring real-time performance and accuracy.

[0004] Traditional methods for detecting abnormal behaviors have obvious limitations in dealing with problems such as high-dimensional network traffic data, noise interference, and data imbalance. In order to overcome these technical challenges.

[0005] In the field of detecting network abnormal behaviors, the existing technical methods have the following main disadvantages:

[0006] 1. Weak noise processing ability: Network traffic data often contains a large amount of noise, such as irrelevant normal traffic data, random fluctuations caused by network latency, etc. Traditional anomaly detection methods (such as those based on statistics or simple filtering mechanisms) perform poorly in dealing with this noise, easily leading to inaccurate classification results. Noise will interfere with the extraction of features by the model, making it difficult for the model to find effective abnormal behavior signals in the noise.

[0007] Influence of weak noise processing ability: Noise will reduce the robustness of the model, increase the false alarm rate, and make the performance of the model unstable in actual applications.

[0008] 2. Insufficient handling of data imbalance problem: In most network traffic datasets, normal behaviors account for the vast majority, while the samples of abnormal behaviors are usually very scarce. When traditional machine learning models face this imbalanced data, they tend to learn the patterns of normal behaviors and ignore the minority of abnormal behaviors, resulting in insufficient recognition ability for abnormal behaviors. Even some commonly used data augmentation methods (such as oversampling, SMOTE, etc.) cannot effectively generate complex non-linear samples.

[0009] Influence of handling data imbalance problem: Data imbalance leads to a high false negative rate, and the model cannot effectively detect potential network threats.

[0010] 3. Limited ability to capture temporal information: Abnormal behaviors in network traffic often have temporal dependencies, but many existing anomaly detection methods only focus on static features and ignore time series information. In this case, classification models are difficult to identify long-term continuous attack patterns (such as APT attacks) or short-term sudden abnormal behaviors (such as DDoS attacks). Traditional methods such as SVM, decision trees, etc. cannot effectively model temporal information, resulting in inaccurate detection of abnormal behaviors.

[0011] Influence of limited ability to capture temporal information: Ignoring temporal dependencies makes it difficult for existing models to capture the time patterns of network behaviors, especially when dealing with long-term or sudden attack behaviors, the model performance is poor. Summary of the Invention

[0012] To solve the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a network abnormal behavior detection method based on dynamic data balance and generation, which can improve the detection efficiency and accuracy of network abnormal behaviors by solving the problems of data imbalance, scarce abnormal data and insufficient diversity in network traffic.

[0013] The present invention aims to:

[0014] 1. Improve the balance of the dataset: By using an abnormal data marking module to mark abnormal data, a dynamic analysis layer to selectively undersample normal data, and combining FlowGAN to generate more diverse abnormal data, dynamically adjust the ratio of normal and abnormal data, so as to effectively solve the problem that normal data in network traffic is much more than abnormal data.

[0015] 2. Expand the diversity of abnormal data: Through FlowGAN, generate more diverse abnormal samples, enrich the abnormal dataset, so that the detection model can learn more abnormal behavior patterns, especially in the case of scarce abnormal data, further improve the model's ability to handle complex abnormal behaviors.

[0016] 3. Reduce noise interference: Through TimeDiT noise reduction processing, eliminate the noise in the network traffic data, enhance the data quality, make it easier for the model to learn key features from the data, and reduce false alarms and missed alarms caused by noise interference.

[0017] 4. Improve the detection ability for complex time-series abnormal behaviors: Through weighted training of LSTM-Transformer, combine the short-term dependence processing ability of LSTM and the global dependence processing ability of Transformer to enhance the model's recognition ability for network abnormal behaviors under complex time-series dependencies. At the same time, use a weighted loss function to solve the problem of data imbalance, improve the model's attention to abnormal data, and thus improve the overall detection performance.

[0018] In summary, the present invention realizes efficient and accurate detection of abnormal behaviors in network traffic, especially in the scenarios of data imbalance, scarce abnormal data, data noise interference, and complex time-series abnormal behaviors, and has significant technical advantages.

[0019] To achieve the above object, the present invention provides the following solutions:

[0020] A network abnormal behavior detection method based on dynamic data balance and generation, including:

[0021] Obtain the original network traffic data, input the original network traffic data into the dynamic abnormal data balance model, and obtain an enhanced data set;

[0022] Based on the abnormal data balance model, mark the original network traffic data, perform selective undersampling on the marked data, adjust the ratio of normal data to abnormal data, obtain a balanced data set, generate new abnormal data according to the balanced data set, and merge it with the balanced data set to obtain an enhanced data set;

[0023] Input the enhanced data set into the TimeDiT noise reduction model for denoising, obtain the denoised target data set, and input it into the LSTM-Transformer weighted training model for network abnormal behavior detection, and weight the abnormal data in the target data set by combining a weighted loss function to obtain the detection result.

[0024] Optionally, marking the original network traffic data includes:

[0025] Gradually separate the data points in the original network traffic data, and generate an abnormal score through the isolation depth of the data points;

[0026] Mark the original network traffic data according to the abnormal score; the marking is used to generate classification labels for abnormal data and normal data.

[0027] Optionally, obtaining the balanced data set includes:

[0028] Performing clustering analysis on the normal data in the labeled data, dividing the normal data into multiple data subsets, selecting representative samples in each data subset for preservation, and at the same time using distance metric analysis to preserve the normal data related to the abnormal data and dynamically adjust the ratio of the normal data to the abnormal data to obtain the balanced data set.

[0029] Optionally, obtaining the enhanced data set includes:

[0030] Preprocessing the balanced data set; the preprocessing includes: filtering, truncating / padding, and normalization processing;

[0031] Inputting the preprocessed balanced data set into the FlowGAN sub-model to generate new abnormal data, and merging the new abnormal data with the balanced data set to obtain the enhanced data set.

[0032] Optionally, the FlowGAN sub-model includes:

[0033] A generator for analyzing the real data distribution in the preprocessed balanced data set and generating forged data;

[0034] A discriminator for judging the data category; the categories include: real data and forged data;

[0035] A training adversary for introducing intensity control perturbations during the process of the discriminator judging the data category, calculating the distance between the real data and the forged data, and controlling the abnormal intensity of the forged data, thereby generating the new abnormal data.

[0036] Optionally, during the game process of the generator and the discriminator, by introducing intensity control perturbations, the formula for generating the new abnormal data is:

[0037]

[0038] where D(x) represents the probability output of the discriminator for the sample x belonging to the real data distribution p data of, represents the judgment of the discriminator on the real sample x, D(G(z)) represents the probability output of the discriminator for the generated sample G(z) belonging to the real data, represents the judgment of the discriminator on the generated sample G(z), Intensity(G(z)) represents the abnormal intensity of the generated sample, It represents the difference between the anomaly intensity of the generated sample G(z) and the set target intensity s, and λ represents the balance between controlling the authenticity of the generated sample and the anomaly intensity control.

[0039] Optionally, obtaining the denoised target data set includes:

[0040] Inputting the augmented data set into the TimeDiT denoising model, gradually increasing the noise and denoising through the diffusion reverse process to obtain the denoised target data set.

[0041] Optionally, obtaining the detection result includes:

[0042] Inputting the denoised target data set into the LSTM layer in the LSTM-Transformer weighted training model to capture short-term dependencies and obtain short-term dependency features; the capturing of short-term dependencies is: saving useful short-term information and removing useless data;

[0043] Through the self-attention mechanism of the Transformer layer in the LSTM-Transformer weighted training model, capturing the global context relationship in the time series to obtain long-term dependency features;

[0044] Based on the short-term dependency features and long-term dependency features, combining with a weighted loss function to weight the abnormal data in the target data set for network abnormal behavior detection to obtain the detection result.

[0045] Optionally, the weighted loss function includes: FocalLoss weighted loss function.

[0046] Optionally, the method further includes:

[0047] Establishing an evaluation metric, evaluating the detection ability of the LSTM-Transformer weighted training model through the evaluation metric, and adjusting the hyperparameters of the LSTM-Transformer weighted training model based on the evaluation result;

[0048] Wherein, the evaluation metric includes: precision, recall rate, and F1 score.

[0049] The beneficial effects of the present invention are:

[0050] 1. Significantly improving the processing ability of the data imbalance problem: The present invention uses an anomaly data marking module to mark the abnormal data through the DADB layer (Dynamic Anomaly Data Balancer), combines the selective undersampling of the dynamic analysis layer and FlowGAN to generate abnormal data, realizing the dynamic balance of normal and abnormal data in the data set. Compared with traditional methods, it can more effectively solve the situation where normal data is far more than abnormal data and improve the detection ability of abnormal data.

[0051] 2. Expand the diversity of abnormal data: By introducing the FlowGAN module, the present invention can not only generate more abnormal data samples, solve the problem of scarce abnormal data, but also generate diverse abnormal behavior patterns. Compared with the limited abnormal data sets relied on by traditional methods, the present invention can enhance the detection ability of the model when facing new and complex abnormal behaviors, and significantly improve the generalization performance.

[0052] 3. Reduce false positives and false negatives: Through the noise reduction processing of TimeDiT (Time Series Diffusion Model), the present invention effectively removes the noise interference in network traffic, retains the key features in the data, enables the model to more accurately identify abnormal behaviors, and reduces the situations of false positives and false negatives. This technology significantly improves the stability and accuracy of the detection system.

[0053] 4. Enhance the detection ability for complex time-series abnormal behaviors: The present invention combines the processing ability of LSTM for short-term dependencies and the capturing ability of Transformer for long-term dependencies through the LSTM-Transformer weighted training module. This module can effectively model and detect complex time-series patterns (including short-term and long-term abnormal behaviors) in network traffic. Compared with traditional models, it can better capture long-term potential abnormal behaviors, such as slow attacks and data leakage behaviors, greatly improving the comprehensiveness of detection.

[0054] 5. Optimize the adaptability of the model to unbalanced data sets: The present invention adopts a weighted loss function (such as FocalLoss) to address the problem of unbalanced data sets. By assigning higher weights to abnormal data, the model's attention to abnormal data during training is increased, thereby improving the detection accuracy of abnormal behaviors. This weighting mechanism enables the model to have higher adaptability and robustness when facing unbalanced data sets.

[0055] 6. Efficient data processing and model training: Through the selective undersampling technique, the present invention reduces the redundancy of normal data, significantly reduces the scale of the data set, and improves the efficiency of data processing. At the same time, the abnormal data generated by FlowGAN also improves the data quality and training efficiency. Compared with traditional network traffic detection methods, the present invention can complete data processing and model training in a shorter time, adapting to the real-time detection requirements in large-scale network environments.

[0056] 7. Flexible adaptation to different network environments: Each module of the present invention (such as the DADB layer, FlowGAN, TimeDiT, LSTM-Transformer weighted training) has a high degree of adjustability and can be flexibly adjusted according to different network environments and detection requirements. Therefore, this solution is not only applicable to typical network traffic anomaly detection, but also capable of adapting to more complex scenarios (such as encrypted traffic, slow attacks, etc.), with strong adaptability and wide application.

[0057] 8. Higher model generalization ability: The present invention generates diverse abnormal data samples through FlowGAN, enabling the model to learn more potential abnormal behavior patterns, thereby enhancing the model's generalization ability. This enables the model to not only handle known abnormal behaviors, but also exhibit stronger detection capabilities when faced with new types of attacks and unknown threats.

[0058] Summary: Through the combination of various modules, the present invention solves the common problems of data imbalance, scarce abnormal data, noise interference, and complex temporal dependencies in network traffic anomaly detection, significantly improving the accuracy and efficiency of detection, reducing false positives and false negatives, and enhancing the robustness and generalization ability of the model in practical applications, making the present invention have significant advantages in various network environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0060] Figure 1 It is a schematic diagram of a network abnormal behavior detection method based on dynamic data balance and generation according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0061] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0062] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the drawings and specific embodiments.

[0063] Such as Figure 1As shown in the figure, a network anomaly behavior detection method based on dynamic data balance and generation in this embodiment includes: obtaining original network traffic data, inputting the original network traffic data into a dynamic anomaly data balance model to obtain an enhanced data set; marking the original network traffic data based on the anomaly data balance model, performing selective undersampling on the marked data, adjusting the ratio of normal data to abnormal data to obtain a balanced data set, generating new abnormal data according to the balanced data set, and merging it with the balanced data set to obtain an enhanced data set; inputting the enhanced data set into the TimeDiT denoising model for denoising to obtain a denoised target data set, and inputting it into the LSTM-Transformer weighted training model for network anomaly behavior detection, weighting the abnormal data in the target data set by combining a weighted loss function to obtain a detection result.

[0064] Specifically, this embodiment proposes a new method based on the DADB layer (Dynamic Anomaly Data Balancer), TimeDiT denoising, and the weighted training model of LSTM-Transformer. This technical solution solves the problems of unbalanced sample data, scarce and insufficiently diverse abnormal data through the DADB layer, solves the problem of a large amount of sample noise through TimeDiT denoising, and finally combines the weighted training model of LSTM-Transformer to output a classification result, aiming to effectively address the problems of high-dimensional data, noise, data imbalance, scarce and insufficiently diverse abnormal data, and temporal dependence in network anomaly behavior detection, with significant creativity, novelty, and practicality.

[0065] For the problems of unbalanced network traffic data, scarcity of abnormal data, and insufficient diversity, this embodiment uses the DADB layer for processing. The DADB layer (Dynamic Abnormal Data Balancer) is a key component in this embodiment for solving the problems of data imbalance, scarcity of abnormal data, and insufficient diversity in network traffic data. It consists of three parts: abnormal data marking, dynamic analysis layer, and FlowGAN (Flow generative adversarial network) data generation. Through this series of steps, it effectively annotates, adjusts, and expands abnormal data to improve the accuracy and coverage of abnormal behavior detection. Abnormal data marking is used to initially mark and screen abnormal data in the dataset, helping to identify potential abnormal points in the data, and outputting the marked dataset, including labels of normal data and abnormal data, for subsequent processing by the dynamic analysis layer. The dynamic analysis layer processes the marked dataset through selective undersampling technology, dynamically intervening to balance the proportion of normal and abnormal data, reducing redundant normal data while maintaining the diversity of normal data. Then it is input into FlowGAN to generate new abnormal data samples to expand the scale and diversity of the abnormal dataset, alleviating the problem of scarce abnormal data. The abnormal data generated by FlowGAN is combined with the original abnormal data to form a balanced and high-quality normal and abnormal dataset for subsequent model training.

[0066] For the noise interference in network traffic, this embodiment adopts noise reduction processing based on TimeDiT (Time-aware Diffusion Transformer). TimeDiT combines the diffusion model with the Transformer, utilizes the time-dependent information in the time series, and removes the noise and redundant features in the data. Compared with traditional noise reduction methods, TimeDiT can better retain the temporal information of the data and reduce the impact of noise on subsequent feature extraction and classification. This method has significant novelty in the field of network abnormal behavior detection and can significantly improve the robustness of the model to noisy data.

[0067] For the temporal dependence in network traffic, this embodiment performs temporal dependence modeling by combining LSTM-Transformer weighted training, using LSTM to process local time dependence and the self-attention mechanism of the Transformer to capture long-distance dependence in network traffic. This combination can not only handle the changes in short-term behavior but also cope with complex long-term dependence behaviors. Compared with traditional static feature classification methods, it can better capture the time patterns of abnormal behaviors, thus improving the overall performance of the classifier. And because a weighted loss function (such as FocalLoss) is incorporated, the model will pay more attention to abnormal data during training, balancing the class bias in the imbalanced dataset, and thus improving the detection ability for abnormal behaviors.

[0068] Furthermore, marking the original network traffic data includes: gradually separating the data points in the original network traffic data, generating an anomaly score based on the isolation depth of the data points; marking the original network traffic data according to the anomaly score; and the marking is used to generate classification labels for abnormal data and normal data.

[0069] Specifically, the DADB layer is the key module of this embodiment, aiming to solve problems such as data imbalance, scarce abnormal data, and insufficient diversity in network traffic data. The DADB layer comprehensively realizes the dynamic balance and expansion of the dataset through abnormal data marking, selective undersampling of the dynamic analysis layer, and abnormal data generation by FlowGAN (Flow generative adversarial network), providing a high-quality abnormal dataset for subsequent model training.

[0070] The abnormal data marking module gradually separates the data points by randomly dividing the dataset, and generates an anomaly score based on the isolation depth of the data points. The easier it is to isolate a data point, the higher the possibility of its abnormality. According to the set anomaly score threshold, the data points are marked as normal or abnormal, which is used to preliminarily mark and screen the abnormal data in the dataset. It can quickly and efficiently mark the potential abnormal behaviors in the data, providing a basis for subsequent processing.

[0071] The specific steps are as follows:

[0072] 1. Input the original network traffic data.

[0073] 2. Randomly divide the dataset, gradually separate the data points, and generate an anomaly score based on the isolation depth of the data points.

[0074] 3. Output the marked dataset, including classification labels for abnormal and normal data.

[0075] Furthermore, obtaining a balanced dataset includes: performing clustering analysis on the normal data in the marked data, dividing the normal data into multiple data subsets, selecting representative samples in each data subset for preservation, and at the same time using distance metric analysis to preserve the normal data related to the abnormal data, and dynamically adjusting the ratio of normal data to abnormal data to obtain a balanced dataset.

[0076] Specifically, the dynamic analysis layer balances the ratio of normal data and abnormal data in the dataset through selective undersampling technology, maintaining the diversity of the dataset. Cluster analysis is used to cluster the normal data, dividing the normal data into multiple subsets, and representative samples are selected and retained in each subset to ensure the diversity of normal behaviors in the dataset. At the same time, through distance metric analysis, normal data that is relatively similar to the abnormal data is retained, reducing redundant normal data, and interference is carried out during the analysis process to dynamically adjust the ratio of normal data to abnormal data. While maintaining the diversity of normal data, the data imbalance is reduced, providing a balanced dataset for the FlowGAN module to generate more useful abnormal data.

[0077] The specific steps are as follows:

[0078] 1. Input the labeled normal and abnormal datasets.

[0079] 2. Adjust the original dataset through dynamic analysis and clustering, and retain representative samples.

[0080] 3. Interfere with the ratio of normal samples to abnormal samples, and output the normal and abnormal datasets processed by selective undersampling, including diverse normal data.

[0081] Furthermore, obtaining the enhanced dataset includes: preprocessing the balanced dataset, inputting the preprocessed balanced dataset into the FlowGAN sub-model to generate new abnormal data; the preprocessing includes: filtering, truncating / padding, and normalization processing; merging the new abnormal data with the balanced dataset to obtain the enhanced dataset.

[0082] Specifically, FlowGAN is a data augmentation method based on the generative adversarial network (GAN), specifically designed to address the problem of unbalanced network traffic data. The FlowGAN data generation module generates more diverse abnormal data, expanding the abnormal dataset and alleviating the problem of scarce abnormal data. It can generate more high-quality abnormal data, expanding the diversity of abnormal behaviors in the dataset and enhancing the detection ability of the model in complex network environments.

[0083] The specific steps are as follows:

[0084] 1. Input the normal and abnormal datasets processed by the dynamic analysis layer.

[0085] 2. Input format standardization: The data first undergoes preprocessing of the PCAP file, including filtering, truncating / padding, and normalization processing.

[0086] 3. FlowGAN model training: Conduct generator training, discriminator training, and game training.

[0087] 4. Data balancing: Generate new abnormal data and then merge it with the balanced dataset.

[0088] 5. Output an enhanced dataset containing the original abnormal data and the abnormal data generated by FlowGAN.

[0089] Furthermore, the FlowGAN sub-model includes: a generator for analyzing the real data distribution in the pre-processed balanced dataset and generating forged data; a discriminator for judging the data category; the categories include: real data and forged data; a training adversary for introducing intensity control perturbations during the process of the discriminator judging the data category, calculating the distance between the real data and the forged data, and controlling the abnormal intensity of the forged data, thereby generating the new abnormal data.

[0090] Specifically, preprocess the balanced dataset, input the pre-processed balanced dataset into the FlowGAN sub-model, and generate new abnormal data through adversarial games. The generator learns the real data distribution and generates a small number of category traffic data similar to the real samples. The discriminator is responsible for discriminating whether the input data comes from the real data distribution or the forged data generated by the generator. During the adversarial training process, the generator and the discriminator are continuously optimized through games to generate more realistic traffic samples. At the same time, different from traditional adversarial training, intensity control perturbations are innovatively introduced during the game process. By calculating the distance between the generated sample and the normal sample (such as the Mahalanobis distance), the abnormal intensity of the generated sample is controlled by parameters. The judgment criterion of the game is not only restricted by the adversarial authenticity between the generator and the discriminator, but also restricted by the custom intensity control perturbations. Finally, abnormal data samples are output through the above modules. The specific formula is as follows:

[0091]

[0092] where D(x) is the probability output of the discriminator that the sample x belongs to the real data distribution p data of the probability output, represents the discriminator's judgment of the real sample x, D(G(z)) is the probability output of the discriminator that the generated sample G(z) belongs to the real data, represents the discriminator's judgment of the generated sample G(z), Intensity(G(z)) is the abnormal intensity of the generated sample, which can be calculated by distance metrics (such as the Mahalanobis distance) or other methods, represents the difference between the abnormal intensity of the generated sample G(z) and the set target intensity s. The weight parameter λ controls the balance between the authenticity of the generated sample and the abnormal intensity control, and can be determined by experimental adjustment. The improved game point includes:

[0093] 1. The samples generated by the generator are realistic enough that the discriminator cannot easily distinguish between real samples and generated samples (D(G(z)) ≈ 0.5).

[0094] 2. The anomaly intensity of the generated samples meets the target requirements (|s - Intensity(G(z))| ≤ ∈);

[0095] where ∈ is the tolerance range, that is, whether the difference between the actual intensity of the generated samples and the specified intensity s is within the tolerance range.

[0096] Furthermore, obtaining the denoised target dataset includes: inputting the enhanced dataset into the TimeDiT denoising model, gradually increasing the noise and denoising through the diffusion reverse process to obtain the denoised target dataset.

[0097] Specifically, the TimeDiT denoising module: uses TimeDiT (Time-aware Diffusion Transformer) to denoise network traffic data through the time diffusion mechanism. The denoising process of TimeDiT utilizes the time-dependent information in network traffic. By gradually removing the noise, the key information in the data becomes clearer. It can improve the overall quality of the dataset, ensure that subsequent model training can focus on learning key temporal features, and reduce false alarms and missed detections.

[0098] The specific steps are as follows:

[0099] 1. Input the dataset processed by the FlowGAN generation and dynamic analysis layer.

[0100] 2. Diffusion process: Gradually increase the noise and denoise through the diffusion reverse process.

[0101] 3. Output a high-quality dataset denoised by TimeDiT, including normal and abnormal data.

[0102] Furthermore, obtaining the detection result includes: inputting the denoised target dataset into the LSTM layer in the LSTM-Transformer weighted training model to capture short-term dependencies and obtain short-term dependency features; capturing short-term dependencies means: saving useful short-term information and removing useless data; through the self-attention mechanism of the Transformer layer in the LSTM-Transformer weighted training model, capturing the global context relationship in the time series and obtaining long-term dependency features; based on the short-term dependency features and long-term dependency features, combining the weighted loss function to weight the abnormal data in the target dataset for network abnormal behavior detection to obtain the detection result; among them, the weighted loss function includes: the FocalLoss weighted loss function.

[0103] Specifically, the LSTM-Transformer weighted training module: The LSTM-Transformer weighted training module can capture short-term and long-term dependency features in the data, solve the data imbalance problem through a weighted loss function, and improve the detection ability for abnormal data. LSTM layer: Processes short-term dependencies and captures short-term behavior changes in network traffic. Through its memory units, LSTM can retain useful short-term information while discarding unimportant data. Transformer layer: Processes long-term dependencies through the self-attention mechanism, captures the global context relationship in the time series, and is especially suitable for detecting abnormal behaviors (such as data leakage) that require long-term accumulation to be observed. Weighted loss function: Uses weighted loss functions such as Focal Loss to assign higher weights to abnormal data and balance the classification weights of normal and abnormal data. This method ensures that the model pays more attention to abnormal behaviors during training, thereby improving the detection effect. By capturing short-term dependencies with LSTM, long-term dependencies with Transformer, and adjusting the weighted loss function, the model's ability to identify abnormal behaviors is improved, especially in complex and data-imbalanced network environments.

[0104] The specific steps are as follows:

[0105] 1. Input a high-quality dataset after TimeDiT noise reduction processing.

[0106] 2. First, use the LSTM layer to process the input time series data to capture short-term dependencies.

[0107] 3. Then, through the self-attention mechanism of Transformer, process the long-term dependencies in the data to enhance the model's understanding of the global time series.

[0108] 4. Weight the abnormal data in the loss function so that the model pays more attention to the features of abnormal points and can better distinguish normal and abnormal behaviors through the capabilities of LSTM-Transformer.

[0109] Furthermore, the method also includes: establishing evaluation metrics, evaluating the detection ability of the LSTM-Transformer weighted training model through the evaluation metrics, and adjusting the hyperparameters of the LSTM-Transformer weighted training model based on the evaluation results; among them, the evaluation metrics include: precision, recall, and F1 score.

[0110] Specifically, model verification and tuning: Verify the performance of the model on the test dataset and perform fine-tuning according to the results to ensure that the model has good generalization ability. Evaluate the detection effect of the model through metrics such as precision, recall, and F1 score, adjust the hyperparameters of data generation, weighted loss function, and model structure, and optimize the performance of the model in the real environment.

[0111] 1. Recall rate: Specifically used to measure the model's ability to capture abnormal behaviors.

[0112] 2. F1 score: Balances recall rate and precision to evaluate overall performance.

[0113] 3. AUC: Evaluates the model's classification ability for the minority class (abnormal behaviors).

[0114] A network abnormal behavior detection method based on dynamic data balancing and generation in this embodiment includes:

[0115] Dynamic abnormal data marking module: This embodiment proposes an abnormal data marking module that can flexibly select marking algorithms (such as Isolation Forest, LOF, One-Class SVM, etc.) according to different application scenarios to mark abnormal data in network traffic. This module can select the most suitable abnormal detection algorithm according to specific data characteristics, providing a highly flexible abnormal detection solution. The flexibility of the module and its application in network traffic abnormal marking, especially the automatic marking and screening mechanism for abnormal data based on different algorithms.

[0116] Selective undersampling and intervention in the dynamic analysis layer: In the data balancing process, this embodiment innovatively proposes a selective undersampling technique that retains the diversity in normal data through clustering analysis while deleting redundant normal data. The dynamic analysis layer can dynamically adjust the ratio of normal and abnormal data without destroying the diversity of normal data, enhancing the effectiveness of abnormal behavior detection. The application of the selective undersampling method in data balancing and the dynamic intervention to adjust the dataset distribution, especially the technical mechanism of normal data screening and retention through clustering analysis.

[0117] FlowGAN generates abnormal data: This embodiment introduces FlowGAN (an abnormal data generation technology based on generative adversarial networks) to generate diverse abnormal data samples. FlowGAN generates real abnormal traffic data through the confrontation between the generator and the discriminator, significantly enhancing the diversity of the abnormal dataset and improving the model's recognition ability for complex abnormal behaviors. The technical solution of FlowGAN for generating abnormal data, especially the confrontation mechanism between the GAN-based generator and discriminator and the application of generating diverse abnormal data.

[0118] Noise reduction processing of TimeDiT (Time Series Diffusion Model): In this embodiment, the TimeDiT model is innovatively used to perform noise reduction processing on network traffic data, eliminating random noise in the data and retaining key features in the data. The TimeDiT noise reduction technology can significantly improve the quality of the data set, reduce noise interference, and enhance the detection accuracy of the model. The TimeDiT noise reduction technology for time series network traffic data, especially its application in improving the accuracy of anomaly detection models and reducing the false alarm rate.

[0119] LSTM-Transformer weighted training model: In this embodiment, the LSTM and Transformer models are combined. The LSTM is used to capture short-term dependencies, and the Transformer is used to capture long-term dependencies to model abnormal behaviors in network traffic. Weighted training enhances the model's detection ability on imbalanced data sets by assigning higher weights to abnormal data. The design and application of the LSTM-Transformer combined model, especially the weighted training mechanism in network traffic anomaly detection.

[0120] Complete data processing and detection solution (DADB layer): In this embodiment, the DADB layer (Dynamic Anomaly Data Balancer) realizes the full-process automated processing from data labeling, data balancing to data expansion by integrating mechanisms such as abnormal data labeling, selective undersampling, and FlowGAN to generate abnormal data. This hierarchical processing technology significantly improves the balance and diversity of the data set and enhances the robustness of the anomaly detection system. The full-process dynamic balancing mechanism of the DADB layer, including the integrated scheme of data labeling, selective undersampling, and FlowGAN generation.

[0121] Network anomaly detection system based on multi-module integration: The entire network anomaly detection system in this embodiment integrates multiple innovative modules and can process complex network traffic data. The collaborative work of each module (such as the abnormal data labeling module, FlowGAN, TimeDiT, and LSTM-Transformer weighted training module) provides a complete solution from data labeling to model training. The multi-module integration scheme, especially the collaborative mechanism between each module and the overall innovation in its application scenario, ensures the high efficiency and adaptability of the entire system.

[0122] Optimized Design of Real-time Network Traffic Anomaly Behavior Detection System: In this embodiment, the real-time processing ability of the system is improved through various modules (especially TimeDiT and FlowGAN), enabling efficient processing of abnormal behaviors in a large-scale network traffic environment and adapting to different scenarios such as enterprise networks and cloud service platforms. This embodiment can process the optimization mechanism in real time, especially the network anomaly behavior detection scheme for high-traffic and low-latency environments.

[0123] This embodiment discloses the detection of abnormal traffic in a multi-tenant environment of a cloud service platform:

[0124] Scenario Description: In a multi-tenant environment on a cloud service platform, the network traffic characteristics of different tenants vary, and malicious attack traffic or abnormal tenant behaviors (such as brute force cracking, DDoS attacks, etc.) may be hidden in the traffic.

[0125] Data Source: Network traffic logs from different tenants on the cloud platform, including tenant service requests, data transfer records, etc.

[0126] Implementation Scheme:

[0127] 1. Abnormal Data Marking Module: Use the LOF (Local Outlier Factor) algorithm for local anomaly detection to mark the abnormal behaviors of certain tenants, such as a sudden increase in traffic or frequent service request failures.

[0128] 2. Selective Undersampling in the Dynamic Analysis Layer: Cluster the normal traffic of tenants, retain the diversity in the normal tenant traffic, and at the same time remove redundant normal traffic data.

[0129] 3. FlowGAN Abnormal Data Generation: Generate simulated abnormal tenant behaviors (such as frequent login attempts, brute force cracking, DDoS traffic, etc.) through FlowGAN to enhance the diversity of abnormal samples.

[0130] 4. TimeDiT Noise Reduction Processing: Perform noise reduction on the generated abnormal traffic and tenant normal traffic to retain important tenant behavior characteristics.

[0131] 5. LSTM-Transformer Weighted Training: Train the LSTM-Transformer model through a weighted loss function to identify complex abnormal behaviors in a multi-tenant environment and capture short-term burst behaviors and long-term abnormal patterns among tenants.

[0132] Effect: Improved anomaly detection accuracy in a multi-tenant environment: Through the anomaly tenant behavior samples generated by FlowGAN, the model can effectively identify malicious attack behaviors, such as brute force cracking and DDoS attacks. Enhanced data diversity: The selective undersampling of the dynamic analysis layer preserves the traffic characteristics of different tenants, ensuring that the model can accurately distinguish normal and abnormal traffic in a multi-tenant environment. Fast response ability: When processing the large-scale tenant traffic of the cloud platform, the system can detect and respond to abnormal behaviors in real time to ensure the security of the platform.

[0133] This embodiment discloses the detection of abnormal transaction traffic in financial institutions:

[0134] Scenario description: In the network environment of financial institutions, transaction traffic is often highly monitored. However, due to the frequent transactions and large amount of data, it is difficult to detect abnormal transaction traffic (such as fraudulent transactions, fund transfers, etc.), and the abnormal behavior data is scarce.

[0135] Data source: Transaction logs from financial institutions and traffic data from payment platforms, including a large amount of normal transaction traffic and a very small number of abnormal transaction traffic.

[0136] Implementation plan:

[0137] 1. Abnormal data marking module: Use the One-Class SVM algorithm to mark abnormal transaction traffic and identify transaction traffic that may pose a fraud risk.

[0138] 2. Selective undersampling of the dynamic analysis layer: Perform selective undersampling on normal transaction traffic, retain representative transaction behaviors, and reduce the redundancy of normal transaction data.

[0139] 3. FlowGAN abnormal data generation: Generate simulated abnormal transaction traffic samples, such as fraudulent transactions and abnormal fund flows, through FlowGAN to expand the abnormal data set.

[0140] 4. TimeDiT noise reduction processing: Perform noise reduction processing on transaction traffic, remove random fluctuations in the data, and retain key transaction patterns.

[0141] 5. LSTM-Transformer weighted training: Combine short-term transaction behaviors and long-term fund flow patterns, and train the model through LSTM-Transformer weighted training to improve the ability to detect abnormal transactions.

[0142] Function effects: Improved fraud transaction detection rate: Through the diverse abnormal transaction samples generated by FlowGAN, the model can effectively identify potential fraud behaviors, such as abnormal large - amount fund transfers, etc. Noise filtering for financial transaction data: TimeDiT noise reduction effectively removes the noise interference in transactions and improves the model's detection ability for real abnormal transactions. Efficient processing of large - scale transaction data: When processing a large amount of transaction traffic, the system can operate efficiently and promptly identify abnormal transaction behaviors.

[0143] The embodiments described above are only descriptions of the preferred embodiments of the present invention and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A network abnormal behavior detection method based on dynamic data balancing and generation, characterized in that: include: Obtaining original network traffic data, inputting the original network traffic data into a dynamic abnormal data balancing model, and obtaining an enhanced data set; Acquiring the enhanced data set includes: preprocessing the balanced data set; the preprocessing includes: filtering, truncation / filling and normalization processing; inputting the preprocessed balanced data set into the FlowGAN sub-model to generate new abnormal data, merging the new abnormal data with the balanced data set to obtain the enhanced data set; The FlowGAN sub-model includes: a generator, which is used to analyze the distribution of real data in the preprocessed balanced data set and generate forged data; a discriminator, which is used to judge the data category; the category includes: real data and forged data; a training adversary, which is used to introduce intensity control disturbance in the process of the discriminator judging the data category, calculate the distance between the real data and the forged data, and control the abnormal intensity of the forged data, so as to generate the new abnormal data; In the process of the game between the generator and the discriminator, the intensity control disturbance is introduced, and the formula for generating the new abnormal data is: Among them, D(x) indicates that the discriminator believes that sample x belongs to the real data distribution p data The probability output is represents the judgment of the discriminator on the real sample x, D(G(z)) represents the probability output of the discriminator that the generated sample G(z) belongs to the real data, represents the judgment of the discriminator on the generated sample G(z), Intensity(G(z)) represents the abnormal intensity of the generated sample, represents the difference between the anomaly strength of the generated sample G(z) and the set target strength s, and λ represents the balance between controlling the authenticity of the generated sample and the anomaly strength control; The original network traffic data is marked based on the abnormal data balancing model, the marked data is selectively undersampled, the ratio of normal data to abnormal data is adjusted, a balanced data set is obtained, new abnormal data is generated according to the balanced data set, and the new abnormal data is merged with the balanced data set to obtain an enhanced data set; The enhanced data set is input into the TimeDiT denoising model for denoising, and the denoised target data set is obtained. The enhanced data set is input into the LSTM-Transformer weighted training model for network abnormal behavior detection. The abnormal data in the target data set is weighted in combination with the weighted loss function to obtain the detection result.

2. The network abnormal behavior detection method based on dynamic data balancing and generation according to claim 1 is characterized in that: Marking the original network traffic data includes: gradually separating the data points in the raw network traffic data, and generating anomaly scores according to the isolation depth of the data points; The original network traffic data is marked according to the anomaly score; the mark is used to generate classification labels for abnormal data and normal data.

3. The network abnormal behavior detection method based on dynamic data balancing and generation according to claim 1 is characterized in that: Acquiring the balanced data set includes: Cluster analysis is used to cluster the normal data in the labeled data, and the normal data is divided into multiple data subsets. Representative samples in each data subset are selected for storage. At the same time, distance metric analysis is used to save the normal data related to the abnormal data, and the ratio of normal data to abnormal data is dynamically adjusted to obtain the balanced data set.

4. The network abnormal behavior detection method based on dynamic data balancing and generation according to claim 1 is characterized in that: Acquiring the denoised target data set includes: The enhanced data set is input into the TimeDiT denoising model, noise is gradually added and denoised through a diffusion reverse process to obtain the denoised target data set.

5. The network abnormal behavior detection method based on dynamic data balancing and generation according to claim 1 is characterized in that: Obtaining the detection result includes: Input the denoised target data set into the LSTM layer in the LSTM-Transformer weighted training model to capture short-term dependencies and obtain short-term dependency features; capturing short-term dependencies means: saving useful short-term information and eliminating useless data; The global contextual relationship in the time series is captured and long-term dependency features are obtained through the self-attention mechanism of the Transformer layer in the LSTM-Transformer weighted training model; According to the short-term dependency features and the long-term dependency features, the abnormal data in the target data set is weighted in combination with a weighted loss function to perform network abnormal behavior detection to obtain the detection result.

6. The method for detecting abnormal network behavior based on dynamic data balancing and generation according to claim 5, characterized in that: The weighted loss function includes: FocalLoss weighted loss function.

7. The network abnormal behavior detection method based on dynamic data balancing and generation according to claim 5 is characterized in that: The method further comprises: Establish evaluation indicators, evaluate the detection capability of the LSTM-Transformer weighted training model through the evaluation indicators, and adjust the hyperparameters of the LSTM-Transformer weighted training model based on the evaluation results; wherein the evaluation indicators include: precision, recall and F1 score.

Images