Network abnormal behavior detection method based on dynamic data balance and generation

Abstract

The invention relates to a network abnormal behavior detection method based on dynamic data balance and generation, and the method comprises the steps: obtaining original network flow data, inputting the original network flow data into a dynamic abnormal data balance model, and obtaining an enhanced data set; the method comprises the steps of marking original network flow data based on an abnormal data balance model, performing selective undersampling on marked data, adjusting a ratio of normal data to abnormal data, obtaining a balance data set, generating new abnormal data according to the balance data set, and merging the new abnormal data with the balance data set to obtain an enhanced data set; and inputting the enhanced data set into a TimeDiT noise reduction model for denoising to obtain a target data set after noise reduction, inputting the target data set into an LSTM-Transform weighted training model for network abnormal behavior detection, and weighting abnormal data in the target data set in combination with a weighted loss function to obtain a detection result. According to the invention, the network traffic abnormal behavior detection efficiency and accuracy can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of network security and data analysis, and in particular to a network abnormal behavior detection method based on dynamic data balancing and generation. Background Art

[0002] In the modern information society, the Internet has become an important part of people's daily life and work. However, with the increase in network scale and complexity, network abnormal behaviors (such as network attacks, data leakage, etc.) have become increasingly complex and diverse. Effective network abnormal behavior detection methods are of great significance to ensure network security. In current research, traditional rule-based and statistical detection methods can no longer cope with complex network environments, so it is necessary to use advanced machine learning and deep learning technologies to improve the accuracy and efficiency of detection.

[0003] Network data has the characteristics of complex format, high dimension and large data volume. Network traffic data includes various attributes such as various protocols, source address, destination address, port number, timestamp, etc., and is also accompanied by a large amount of noise and redundant information. These characteristics make it difficult for traditional data processing methods to effectively process and analyze network data while ensuring real-time and accuracy.

[0004] Traditional abnormal behavior detection methods have obvious limitations when dealing with high-dimensional network traffic data, noise interference, data imbalance and other problems. In order to overcome these technical challenges.

[0005] In the field of network abnormal behavior detection, existing technical methods have the following main shortcomings:

[0006] 1. Weak noise processing capability: Network traffic data often contains a lot of noise, such as irrelevant normal traffic data, random fluctuations caused by network delays, etc. Traditional anomaly detection methods (such as those based on statistics or simple filtering mechanisms) perform poorly in processing these noises, which can easily lead to inaccurate classification results. Noise interferes with the model's feature extraction, making it difficult for the model to find effective abnormal behavior signals in the noise.

[0007] Impact of weak noise processing ability: Noise will reduce the robustness of the model, increase the false alarm rate, and make the model unstable in practical applications.

[0008] 2. Inadequate handling of data imbalance: In most network traffic data sets, normal behaviors account for the vast majority, while samples of abnormal behaviors are usually very rare. When faced with such unbalanced data, traditional machine learning models tend to learn the patterns of normal behaviors and ignore a small number of abnormal behaviors, resulting in insufficient ability to identify abnormal behaviors. Even some commonly used data enhancement methods (such as oversampling, SMOTE, etc.) cannot effectively generate complex nonlinear samples.

[0009] Impact of data imbalance problem: Data imbalance leads to a high false negative rate, and the model cannot effectively detect potential network threats.

[0010] 3. Limited ability to capture time series information: Abnormal behaviors in network traffic often have time series dependencies, but many existing anomaly detection methods only focus on static features and ignore time series information. In this case, it is difficult for classification models to identify long-term persistent attack patterns (such as APT attacks) or short-term sudden abnormal behaviors (such as DDoS attacks). Traditional methods such as SVM and decision trees cannot effectively model time series information, resulting in inaccurate detection of abnormal behaviors.

[0011] The impact of limited ability to capture temporal information: Ignoring temporal dependencies makes it difficult for existing models to capture the temporal patterns of network behaviors, especially when dealing with long-term or sudden attack behaviors, the models perform poorly. Summary of the invention

[0012] In order to solve the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a network abnormal behavior detection method based on dynamic data balancing and generation, which improves the detection efficiency and accuracy of network abnormal behavior by solving the problems of data imbalance, scarcity of abnormal data and insufficient diversity in network traffic.

[0013] The present invention aims to:

[0014] 1. Improve the balance of the data set: By using the abnormal data marking module to mark the abnormal data, the dynamic analysis layer to selectively undersample the normal data, and combining FlowGAN to generate more diverse abnormal data, the ratio of normal to abnormal data is dynamically adjusted, thereby effectively solving the problem that the normal data in the network traffic is far more than the abnormal data.

[0015] 2. Expand the diversity of abnormal data: Through FlowGAN, more diverse abnormal samples are generated and the abnormal data set is enriched, so that the detection model can learn more abnormal behavior patterns, especially when abnormal data is scarce, further improving the model's ability to deal with complex abnormal behaviors.

[0016] 3. Reduce noise interference: Through TimeDiT noise reduction processing, the noise in network traffic data is eliminated, the quality of the data is enhanced, and the model is easier to learn key features from the data, reducing false positives and false negatives caused by noise interference.

[0017] 4. Improve the ability to detect abnormal behaviors in complex time series: Through LSTM-Transformer weighted training, the short-term dependency processing capability of LSTM and the global dependency processing capability of Transformer are combined to enhance the model's ability to identify abnormal behaviors in networks under complex time series dependencies. At the same time, the weighted loss function is used to solve the problem of data imbalance and increase the model's attention to abnormal data, thereby improving the overall detection performance.

[0018] In summary, the present invention realizes efficient and accurate detection of abnormal behavior in network traffic, especially in situations of data imbalance, scarcity of abnormal data, data noise interference and complex time series abnormal behavior, and has significant technical advantages.

[0019] To achieve the above object, the present invention provides the following solutions:

[0020] A network abnormal behavior detection method based on dynamic data balancing and generation, comprising:

[0021] Obtaining original network traffic data, inputting the original network traffic data into a dynamic abnormal data balancing model, and obtaining an enhanced data set;

[0022] The original network traffic data is marked based on the abnormal data balancing model, the marked data is selectively undersampled, the ratio of normal data to abnormal data is adjusted, a balanced data set is obtained, new abnormal data is generated according to the balanced data set, and the new abnormal data is merged with the balanced data set to obtain an enhanced data set;

[0023] The enhanced data set is input into the TimeDiT denoising model for denoising, and the denoised target data set is obtained. The enhanced data set is input into the LSTM-Transformer weighted training model for network abnormal behavior detection. The abnormal data in the target data set is weighted in combination with the weighted loss function to obtain the detection result.

[0024] Optionally, marking the original network traffic data includes:

[0025] gradually separating the data points in the raw network traffic data, and generating anomaly scores according to the isolation depth of the data points;

[0026] The original network traffic data is marked according to the anomaly score; the mark is used to generate classification labels for abnormal data and normal data.

[0027] Optionally, obtaining the balanced data set includes:

[0028] Cluster analysis is used to cluster the normal data in the labeled data, and the normal data is divided into multiple data subsets. Representative samples in each data subset are selected for storage. At the same time, distance metric analysis is used to save the normal data related to the abnormal data, and the ratio of normal data to abnormal data is dynamically adjusted to obtain the balanced data set.

[0029] Optionally, acquiring the enhanced data set includes:

[0030] Preprocessing the balanced data set; the preprocessing includes filtering, truncation / filling and normalization;

[0031] The preprocessed balanced data set is input into the FlowGAN sub-model to generate new abnormal data, and the new abnormal data is merged with the balanced data set to obtain an enhanced data set.

[0032] Optionally, the FlowGAN sub-model includes:

[0033] Generator, used to analyze the real data distribution in the preprocessed balanced data set and generate fake data;

[0034] A discriminator, used to determine data categories; the categories include: real data and forged data;

[0035] The training adversary is used to introduce intensity control disturbance in the process of the judger judging the data category, calculate the distance between the real data and the forged data, control the abnormal intensity of the forged data, and thus generate the new abnormal data.

[0036] Optionally, during the game between the generator and the discriminator, an intensity control disturbance is introduced, and the formula for generating the new abnormal data is:

[0037]

[0038] Among them, D(x) indicates that the discriminator believes that sample x belongs to the real data distribution p data The probability output is represents the judgment of the discriminator on the real sample x, D(G(z)) represents the probability output of the discriminator that the generated sample G(z) belongs to the real data, represents the judgment of the discriminator on the generated sample G(z), Intensity(G(z)) represents the abnormal intensity of the generated sample, It represents the difference between the anomaly strength of the generated sample G(z) and the set target strength s, and λ represents the balance between controlling the authenticity of the generated sample and the anomaly strength control.

[0039] Optionally, obtaining the denoised target data set includes:

[0040] The enhanced data set is input into the TimeDiT denoising model, noise is gradually added and denoised through a diffusion reverse process to obtain the denoised target data set.

[0041] Optionally, obtaining the detection result includes:

[0042] Input the denoised target data set into the LSTM layer in the LSTM-Transformer weighted training model to capture short-term dependencies and obtain short-term dependency features; capturing short-term dependencies means: saving useful short-term information and eliminating useless data;

[0043] The global contextual relationship in the time series is captured and long-term dependency features are obtained through the self-attention mechanism of the Transformer layer in the LSTM-Transformer weighted training model;

[0044] According to the short-term dependency features and the long-term dependency features, the abnormal data in the target data set is weighted in combination with a weighted loss function to perform network abnormal behavior detection to obtain the detection result.

[0045] Optionally, the weighted loss function includes: a Focal Loss weighted loss function.

[0046] Optionally, the method further comprises:

[0047] Establishing an evaluation indicator, evaluating the detection capability of the LSTM-Transformer weighted training model by the evaluation indicator, and adjusting the hyperparameters of the LSTM-Transformer weighted training model based on the evaluation result;

[0048] The evaluation indicators include precision, recall and F1 score.

[0049] The beneficial effects of the present invention are:

[0050] 1. Significantly improve the ability to handle data imbalance problems: This invention uses the DADB layer (dynamic abnormal data balancer) to mark abnormal data using an abnormal data marking module, combined with the selective undersampling of the dynamic analysis layer and FlowGAN to generate abnormal data, to achieve a dynamic balance between normal and abnormal data in the data set. Compared with traditional methods, it can more effectively solve the situation where normal data far outnumbers abnormal data, and improve the ability to detect abnormal data.

[0051] 2. Expand the diversity of abnormal data: The present invention can not only generate more abnormal data samples through the introduction of the FlowGAN module, solving the problem of abnormal data scarcity, but also generate diversified abnormal behavior patterns. Compared with the limited abnormal data set that traditional methods rely on, the present invention can enhance the model's detection ability when facing new and complex abnormal behaviors, and significantly improve the generalization performance.

[0052] 3. Reduce false positives and false negatives: This invention uses TimeDiT (time series diffusion model) noise reduction processing to effectively remove noise interference in network traffic and retain key features in the data, so that the model can more accurately identify abnormal behaviors and reduce false positives and false negatives. This technology significantly improves the stability and accuracy of the detection system.

[0053] 4. Enhanced detection capability for complex time series abnormal behaviors: This invention combines LSTM’s ability to process short-term dependencies and Transformer’s ability to capture long-term dependencies through the LSTM-Transformer weighted training module. This module can effectively model and detect complex time series patterns (including short-term and long-term abnormal behaviors) in network traffic. Compared with traditional models, it can better capture long-term potential abnormal behaviors, such as slow attacks and data leakage behaviors, greatly improving the comprehensiveness of detection.

[0054] 5. Optimize the adaptability of the model to unbalanced data sets: This invention uses a weighted loss function (such as FocalLoss) to deal with the problem of unbalanced data sets. By giving higher weights to abnormal data, the model pays more attention to abnormal data during training, thereby improving the detection accuracy of abnormal behavior. This weighting mechanism enables the model to have higher adaptability and robustness when facing unbalanced data sets.

[0055] 6. Efficient data processing and model training: The present invention reduces the redundancy of normal data through selective undersampling technology, significantly reduces the size of the data set, and improves the efficiency of data processing. At the same time, the abnormal data generated by FlowGAN also improves data quality and training efficiency. Compared with traditional network traffic detection methods, the present invention can complete data processing and model training in a shorter time, and adapt to the real-time detection needs in large-scale network environments.

[0056] 7. Flexible adaptation to different network environments: The various modules of the present invention (such as DADB layer, FlowGAN, TimeDiT, LSTM-Transformer weighted training) are highly adjustable and can be flexibly adjusted according to different network environments and detection requirements. Therefore, this solution is not only suitable for typical network traffic anomaly detection, but also can adapt to more complex scenarios (such as encrypted traffic, slow attacks, etc.), with strong adaptability and wide application.

[0057] 8. Higher model generalization ability: The present invention generates diversified abnormal data samples through FlowGAN, so that the model can learn more potential abnormal behavior patterns, thereby enhancing the generalization ability of the model. This enables the model to not only handle known abnormal behaviors, but also show stronger detection capabilities when facing new attacks and unknown threats.

[0058] Summary: Through the combination of various modules, the present invention solves the common problems of data imbalance, scarcity of abnormal data, noise interference and complex timing dependency in network traffic anomaly detection, significantly improves the accuracy and efficiency of detection, reduces false positives and missed positives, and enhances the robustness and generalization ability of the model in practical applications, making the present invention have significant advantages in a variety of network environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0060] Figure 1 A schematic diagram of a network abnormal behavior detection method based on dynamic data balancing and generation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] like Figure 1As shown, the present embodiment is a method for detecting abnormal network behavior based on dynamic data balancing and generation, including: obtaining original network traffic data, inputting the original network traffic data into a dynamic abnormal data balancing model, and obtaining an enhanced data set; marking the original network traffic data based on the abnormal data balancing model, selectively undersampling the marked data, adjusting the ratio of normal data to abnormal data, obtaining a balanced data set, generating new abnormal data based on the balanced data set, and merging the new abnormal data with the balanced data set to obtain an enhanced data set; inputting the enhanced data set into a TimeDiT denoising model for denoising, obtaining a denoised target data set, and inputting the LSTM-Transformer weighted training model for network abnormal behavior detection, weighting the abnormal data in the target data set in combination with a weighted loss function, and obtaining a detection result.

[0064] Specifically, this embodiment proposes a new method based on the DADB layer (Dynamic Anomaly Data Balancer), TimeDiT denoising and LSTM-Transformer weighted training model. This technical solution solves the problems of sample data imbalance, abnormal data scarcity and insufficient diversity through the DADB layer, solves the problem of high sample noise through TimeDiT denoising, and finally combines the LSTM-Transformer weighted training model to output the classification results. It aims to effectively deal with the problems of high-dimensional data, noise, data imbalance, abnormal data scarcity and insufficient diversity and time series dependence in network abnormal behavior detection, and has significant creativity, novelty and practicality.

[0065] For the problems of imbalanced network traffic data, scarcity of abnormal data and insufficient diversity, this embodiment uses the DADB layer to handle them. The DADB layer (dynamic abnormal data balancer) is a key component used in this embodiment to solve the problems of data imbalance, scarcity of abnormal data and insufficient diversity in network traffic data. It consists of three parts: abnormal data labeling, dynamic analysis layer and FlowGAN (Flow generative adversarial network) data generation. Through this series of steps, abnormal data is effectively labeled, adjusted and expanded to improve the accuracy and coverage of abnormal behavior detection. Abnormal data labeling is used to preliminarily mark and filter abnormal data in the data set, help identify potential abnormal points in the data, and output the labeled data set, including labels of normal data and abnormal data, for subsequent dynamic analysis layer processing. The dynamic analysis layer processes the labeled data set through selective undersampling technology, dynamically intervenes to balance the ratio of normal and abnormal data, reduce redundant normal data, and maintain the diversity of normal data. Then it is input into FlowGAN to generate new abnormal data samples to expand the scale and diversity of abnormal data sets and alleviate the problem of scarce abnormal data. The abnormal data generated by FlowGAN is combined with the original abnormal data to form a balanced, high-quality normal and abnormal data set for subsequent model training.

[0066] For the noise interference in network traffic, this embodiment adopts the noise reduction processing based on TimeDiT (Time-aware Diffusion Transformer). TimeDiT combines the diffusion model with Transformer and uses the time-dependent information in the time series to remove noise and redundant features in the data. Compared with traditional noise reduction methods, TimeDiT can better retain the timing information of the data and reduce the impact of noise on subsequent feature extraction and classification. This method has significant novelty in the field of network abnormal behavior detection and can significantly improve the robustness of the model to noisy data.

[0067] For the temporal dependencies in network traffic, this embodiment combines LSTM-Transformer weighted training to perform temporal dependency modeling, uses LSTM to process local temporal dependencies, and uses Transformer's self-attention mechanism to capture long-distance dependencies in network traffic. This combination can not only handle changes in short-term behaviors, but also cope with complex long-term dependent behaviors. Compared with traditional static feature classification methods, it can better capture the temporal patterns of abnormal behaviors, thereby improving the overall performance of the classifier, and because the model incorporates weighted loss functions (such as Focal Loss), it will pay more attention to abnormal data during training, balancing the category bias in unbalanced data sets, thereby improving the ability to detect abnormal behaviors.

[0068] Furthermore, marking the raw network traffic data includes: gradually separating the data points in the raw network traffic data, and generating anomaly scores according to the isolation depth of the data points; marking the raw network traffic data according to the anomaly scores; and marking is used to generate classification labels for abnormal data and normal data.

[0069] Specifically, the DADB layer is a key module of this embodiment, which aims to solve the problems of data imbalance, scarcity of abnormal data and lack of diversity in network traffic data. The DADB layer realizes the dynamic balance and expansion of the data set through abnormal data marking, selective undersampling of the dynamic analysis layer, and abnormal data generation by FlowGAN (Flow generative adversarial network), providing high-quality abnormal data sets for subsequent model training.

[0070] The abnormal data marking module separates data points step by step by randomly dividing the data set, and generates anomaly scores based on the isolation depth of the data points. The easier it is to isolate a data point, the higher the probability that it is abnormal. According to the set anomaly score threshold, the data point is marked as normal or abnormal, which is used to preliminarily mark and filter abnormal data in the data set. It can quickly and efficiently mark potential abnormal behaviors in the data, providing a basis for subsequent processing.

[0071] The specific steps are as follows:

[0072] 1. Input raw network traffic data.

[0073] 2. Randomly divide the dataset, gradually separate the data points, and generate anomaly scores based on the isolation depth of the data points.

[0074] 3. Output a labeled data set, including classification labels for abnormal and normal data.

[0075] Furthermore, obtaining a balanced data set includes: clustering normal data in the labeled data using cluster analysis, dividing the normal data into multiple data subsets, selecting representative samples in each data subset for preservation, and simultaneously using distance metric analysis to preserve normal data related to abnormal data, and dynamically adjusting the ratio of normal data to abnormal data to obtain a balanced data set.

[0076] Specifically, the dynamic analysis layer uses selective undersampling technology to balance the ratio of normal data and abnormal data in the data set and maintain the diversity of the data set. Cluster analysis is used to cluster normal data, divide normal data into multiple subsets, and select representative samples in each subset for retention to ensure the diversity of normal behavior in the data set. At the same time, through distance metric analysis, normal data that is similar to abnormal data is retained to reduce redundant normal data. In addition, intervention is performed during the analysis process to dynamically adjust the ratio of normal data to abnormal data, while maintaining the diversity of normal data, reducing data imbalance, and providing a balanced data set for the FlowGAN module to generate more useful abnormal data.

[0077] The specific steps are as follows:

[0078] 1. Input labeled normal and abnormal datasets.

[0079] 2. Adjust the original data set and retain representative samples through dynamic analysis and clustering.

[0080] 3. Interfere with the ratio of normal samples to abnormal samples, and output normal and abnormal data sets that have been selectively undersampled, containing diverse normal data.

[0081] Furthermore, obtaining the enhanced data set includes: preprocessing the balanced data set, inputting the preprocessed balanced data set into the FlowGAN sub-model, and generating new abnormal data; the preprocessing includes: filtering, truncation / filling and normalization; merging the new abnormal data with the balanced data set to obtain the enhanced data set.

[0082] Specifically, FlowGAN is a data augmentation method based on generative adversarial networks (GAN), which is specifically used to solve the problem of imbalanced network traffic data. The FlowGAN data generation module generates more diverse abnormal data, expands the abnormal data set, and alleviates the problem of scarce abnormal data. It can generate more high-quality abnormal data, expand the diversity of abnormal behaviors in the data set, and enhance the detection ability of the model in complex network environments.

[0083] The specific steps are as follows:

[0084] 1. Input the normal and abnormal data sets after being processed by the dynamic analysis layer.

[0085] 2. Input format standardization: The data is first preprocessed by PCAP file, including filtering, truncation / padding, and normalization.

[0086] 3. FlowGAN model training: generator training, discriminator training and game training.

[0087] 4. Data balancing: Generate new abnormal data and then merge it with the balanced dataset.

[0088] 5. Output an augmented dataset containing the original anomaly data and the anomaly data generated by FlowGAN.

[0089] Furthermore, the FlowGAN sub-model includes: a generator, which is used to analyze the distribution of real data in the preprocessed balanced data set and generate forged data; a discriminator, which is used to judge the data category; the category includes: real data and forged data; a training adversary, which is used to introduce intensity control disturbance in the process of the discriminator judging the data category, calculate the distance between the real data and the forged data, control the abnormal intensity of the forged data, and thus generate the new abnormal data.

[0090] Specifically, the balanced data set is preprocessed, and the preprocessed balanced data set is input into the FlowGAN sub-model, and new abnormal data is generated through game confrontation. The generator generates minority category traffic data similar to real samples by learning the real data distribution. The discriminator is responsible for discriminating whether the input data comes from the real data distribution or is forged data generated by the generator. During the adversarial training process, the generator and the discriminator are continuously optimized through the game to generate more realistic traffic samples. At the same time, different from traditional adversarial training, the intensity control perturbation is innovatively introduced in the game process. By calculating the distance between the generated sample and the normal sample (such as the Mahalanobis distance), the abnormal intensity of the generated sample is controlled by parameters. The judgment criteria of the game are not only constrained by the authenticity of the confrontation between the generator and the discriminator, but also by the customized intensity control perturbation. The abnormal data sample is finally output through the above module. The specific formula is as follows:

[0091]

[0092] Where D(x) is the discriminator's judgment that sample x belongs to the true data distribution p data The probability output is represents the judgment of the discriminator on the real sample x, D(G(z)) is the probability output of the discriminator that the generated sample G(z) belongs to the real data, Indicates the judgment of the discriminator on the generated sample G(z). Intensity(G(z)) is the abnormal intensity of the generated sample, which can be calculated by distance metric (such as Mahalanobis distance) or other methods. It represents the difference between the abnormal intensity of the generated sample G(z) and the set target intensity s. The weight parameter λ controls the balance between the authenticity of the generated sample and the abnormal intensity control. It can be determined through experimental adjustment. The improved game points include:

[0093] 1. The samples generated by the generator are realistic enough that the discriminator cannot easily distinguish between real samples and generated samples (D(G(z))≈0.5).

[0094] 2. The abnormal intensity of the generated samples meets the target requirement (|s-Intensity(G(z))|≤∈);

[0095] Among them, ∈ is the tolerance range, that is, whether the difference between the actual intensity of the generated sample and the specified intensity s is within the tolerance range.

[0096] Furthermore, obtaining the denoised target dataset includes: inputting the enhanced dataset into the TimeDiT denoising model, gradually increasing the noise and denoising through a diffusion reverse process to obtain the denoised target dataset.

[0097] Specifically, the TimeDiT denoising module: uses TimeDiT (Time-aware Diffusion Transformer) to denoise network traffic data through a time diffusion mechanism. The denoising process of TimeDiT utilizes the time-dependent information in network traffic and gradually removes noise to make the key information in the data clearer. This can improve the overall quality of the data set, ensure that subsequent model training can focus on learning key time series features, and reduce false positives and negatives.

[0098] The specific steps are as follows:

[0099] 1. Input the dataset processed by FlowGAN generation and dynamic analysis layer.

[0100] 2. Diffusion process: gradually increase the noise and denoise it through the diffusion reverse process.

[0101] 3. Output a high-quality dataset after TimeDiT denoising, including normal and abnormal data.

[0102] Furthermore, obtaining the detection result includes: inputting the denoised target data set into the LSTM layer in the LSTM-Transformer weighted training model to capture short-term dependencies and obtain short-term dependency features; capturing short-term dependencies means: saving useful short-term information and eliminating useless data; capturing the global contextual relationship in the time series through the self-attention mechanism of the Transformer layer in the LSTM-Transformer weighted training model to obtain long-term dependency features; based on the short-term dependency features and the long-term dependency features, combining the weighted loss function to weight the abnormal data in the target data set to perform network abnormal behavior detection and obtain the detection result; wherein the weighted loss function includes: Focal Loss weighted loss function.

[0103] Specifically, the LSTM-Transformer weighted training module can capture the short-term and long-term dependency features in the data, solve the data imbalance problem through the weighted loss function, and improve the detection ability of abnormal data. LSTM layer: handles short-term dependencies and captures short-term behavior changes in network traffic. LSTM can retain useful short-term information through its memory unit while discarding unimportant data. Transformer layer: handles long-term dependencies through the self-attention mechanism and captures the global contextual relationship in the time series, which is particularly suitable for detecting abnormal behaviors that require a long time to accumulate before they can be observed (such as data leakage). Weighted loss function: Use weighted loss functions such as Focal Loss to give higher weights to abnormal data and balance the classification weights of normal and abnormal data. This method ensures that the model pays more attention to abnormal behaviors during training, thereby improving the detection effect. By capturing short-term dependencies with LSTM, capturing long-term dependencies with Transformer, and adjusting the weighted loss function, the model's ability to identify abnormal behaviors is improved, especially in complex and data-imbalanced network environments.

[0104] The specific steps are as follows:

[0105] 1. Input high-quality dataset after TimeDiT denoising.

[0106] 2. First use the LSTM layer to process the input time series data to capture short-term dependencies.

[0107] 3. Then, through the self-attention mechanism of Transformer, long-term dependencies in the data are processed to improve the model's understanding of the global time series.

[0108] 4. Weight the abnormal data in the loss function so that the model pays more attention to the characteristics of the abnormal points and better distinguishes normal from abnormal behaviors through the ability of LSTM-Transformer.

[0109] Furthermore, the method also includes: establishing evaluation indicators, evaluating the detection ability of the LSTM-Transformer weighted training model through the evaluation indicators, and adjusting the hyperparameters of the LSTM-Transformer weighted training model based on the evaluation results; wherein the evaluation indicators include: precision, recall and F1 score.

[0110] Specifically, model verification and tuning: verify the performance of the model on the test data set, and fine-tune it based on the results to ensure that the model has good generalization ability. Evaluate the detection effect of the model through indicators such as precision, recall, and F1 score, adjust the hyperparameters of data generation, weighted loss function, and model structure, and optimize the performance of the model in the real environment.

[0111] 1. Recall rate: It is especially used to measure the model’s ability to capture abnormal behaviors.

[0112] 2. F1 score: balances recall and precision and evaluates overall performance.

[0113] 3.AUC: evaluates the model’s ability to classify minority classes (abnormal behaviors).

[0114] This embodiment provides a method for detecting abnormal network behavior based on dynamic data balancing and generation, including:

[0115] Dynamic abnormal data marking module: This embodiment proposes an abnormal data marking module, which can flexibly select marking algorithms (such as Isolation Forest, LOF, One-Class SVM, etc.) according to different application scenarios to mark abnormal data in network traffic. This module can select the most suitable anomaly detection algorithm according to specific data characteristics and provide a highly flexible anomaly detection solution. The flexibility of the module and its application in network traffic anomaly marking, especially the automatic marking and screening mechanism of abnormal data based on different algorithms.

[0116] Selective undersampling and intervention of the dynamic analysis layer: In the process of data balancing, this embodiment innovatively proposes a selective undersampling technology, which retains the diversity in normal data through cluster analysis and deletes redundant normal data. The dynamic analysis layer can dynamically adjust the ratio of normal and abnormal data without destroying the diversity of normal data, thereby enhancing the effectiveness of abnormal behavior detection. The application of the selective undersampling method in data balancing and the dynamic intervention to adjust the distribution of data sets, especially the technical mechanism of screening and retaining normal data through cluster analysis.

[0117] FlowGAN generates abnormal data: This embodiment introduces FlowGAN (abnormal data generation technology based on generative adversarial network) to generate diverse abnormal data samples. FlowGAN generates real abnormal traffic data through the adversarial between the generator and the discriminator, which significantly enhances the diversity of abnormal data sets and improves the model's ability to identify complex abnormal behaviors. FlowGAN is a technical solution for generating abnormal data, especially the adversarial mechanism of the generator and the discriminator based on GAN and the application of generating diverse abnormal data.

[0118] TimeDiT (time series diffusion model) noise reduction processing: This embodiment innovatively uses the TimeDiT model to perform noise reduction processing on network traffic data, eliminating random noise in the data and retaining the key features in the data. TimeDiT noise reduction technology can significantly improve the quality of the data set, reduce noise interference, and enhance the detection accuracy of the model. TimeDiT is a noise reduction processing technology for time series network traffic data, especially its application in improving the accuracy of anomaly detection models and reducing false alarm rates.

[0119] LSTM-Transformer weighted training model: This embodiment combines the LSTM and Transformer models, using the ability of LSTM to capture short-term dependencies and Transformer to capture long-term dependencies to model abnormal behaviors in network traffic. Weighted training improves the model's detection capabilities on unbalanced data sets by giving higher weights to abnormal data. The design and application of the LSTM-Transformer combined model, especially the weighted training mechanism in network traffic anomaly detection.

[0120] Complete data processing and detection solution (DADB layer): The DADB layer (Dynamic Anomaly Data Balancer) in this embodiment realizes the full process automation from data labeling, data balancing to data expansion by integrating abnormal data labeling, selective undersampling and FlowGAN to generate abnormal data. This hierarchical processing technology significantly improves the balance and diversity of the data set and enhances the robustness of the anomaly detection system. The full process dynamic balancing mechanism of the DADB layer, including data labeling, selective undersampling and FlowGAN generation integrated solution.

[0121] Network anomaly detection system based on multi-module integration: The entire network anomaly detection system of this embodiment integrates multiple innovative modules and can process complex network traffic data. The collaborative work of various modules (such as abnormal data labeling module, FlowGAN, TimeDiT and LSTM-Transformer weighted training module) provides a complete solution from data labeling to model training. The multi-module integration solution, especially the collaborative mechanism between modules and the overall innovation in their application scenarios, ensures the efficiency and adaptability of the entire system.

[0122] Optimized design of real-time network traffic abnormal behavior detection system: This embodiment improves the real-time processing capability of the system through various modules (especially TimeDiT and FlowGAN), and can efficiently handle abnormal behaviors in large-scale network traffic environments, adapting to different scenarios such as enterprise networks and cloud service platforms. This embodiment can process optimization mechanisms in real time, especially for network abnormal behavior detection solutions in high-traffic, low-latency environments.

[0123] This embodiment discloses abnormal traffic detection in a multi-tenant environment of a cloud service platform:

[0124] Scenario description: In a multi-tenant environment on a cloud service platform, different tenants have different network traffic characteristics, and malicious attack traffic or abnormal tenant behavior (such as brute force cracking, DDoS attacks, etc.) may be hidden in the traffic.

[0125] Data source: Network traffic logs from different tenants on the cloud platform, including tenants’ service requests, data transmission records, etc.

[0126] Implementation plan:

[0127] 1. Abnormal data marking module: Use the LOF (local outlier factor) algorithm to perform local anomaly detection and mark abnormal behaviors of certain tenants, such as sudden increase in traffic or frequent service request failures.

[0128] 2. Selective undersampling of the dynamic analysis layer: cluster the normal traffic of tenants, retain the diversity in normal tenant traffic, and remove redundant normal traffic data.

[0129] 3. FlowGAN abnormal data generation: FlowGAN is used to generate simulated abnormal tenant behaviors (such as frequent login attempts, brute force cracking, DDoS traffic, etc.) to enhance the diversity of abnormal samples.

[0130] 4. TimeDiT noise reduction processing: noise reduction is performed on the generated abnormal traffic and the normal traffic of tenants, retaining important tenant behavior characteristics.

[0131] 5. LSTM-Transformer weighted training: The LSTM-Transformer model is trained through a weighted loss function to identify complex abnormal behaviors in a multi-tenant environment and capture short-term burst behaviors and long-term abnormal patterns between tenants.

[0132] Effect: Improved anomaly detection accuracy in multi-tenant environments: Through the abnormal tenant behavior samples generated by FlowGAN, the model can effectively identify malicious attack behaviors, such as brute force cracking and DDoS attacks. Enhanced data diversity: Selective undersampling of the dynamic analysis layer retains the traffic characteristics of different tenants, ensuring that the model can accurately distinguish between normal and abnormal traffic in a multi-tenant environment. Rapid response capability: When processing large-scale tenant traffic on the cloud platform, the system can detect and respond to abnormal behavior in real time to ensure platform security.

[0133] This embodiment discloses transaction flow anomaly detection of financial institutions:

[0134] Scenario description: In the network environment of financial institutions, transaction traffic is often highly monitored. However, due to the frequent transactions and huge amount of data, it is difficult to detect abnormal transaction traffic (such as fraudulent transactions, fund transfers, etc.), and abnormal behavior data is scarce.

[0135] Data source: transaction logs from financial institutions and payment platform traffic data, including a large amount of normal transaction traffic and very little abnormal transaction traffic.

[0136] Implementation plan:

[0137] 1. Abnormal data labeling module: Use the One-ClassSVM algorithm to label abnormal transaction flows and identify transaction flows that may have fraud risks.

[0138] 2. Selective undersampling of the dynamic analysis layer: Selectively undersample normal transaction flows to retain representative transaction behaviors and reduce the redundancy of normal transaction data.

[0139] 3. FlowGAN abnormal data generation: Generate simulated abnormal transaction flow samples, such as fraudulent transactions and abnormal capital flows, through FlowGAN to expand the abnormal data set.

[0140] 4. TimeDiT noise reduction: De-noise the transaction flow, remove random fluctuations in the data, and retain key transaction patterns.

[0141] 5. LSTM-Transformer weighted training: Combining short-term trading behavior and long-term capital flow patterns, the LSTM-Transformer weighted training model is used to improve the ability to detect abnormal transactions.

[0142] Effects: Improved fraudulent transaction detection rate: Through the diverse abnormal transaction samples generated by FlowGAN, the model can effectively identify potential fraudulent behaviors, such as abnormal large-scale fund transfers. Noise filtering of financial transaction data: TimeDiT noise reduction effectively removes noise interference in transactions and improves the model's ability to detect real abnormal transactions. Efficient processing of large-scale transaction data: When processing large amounts of transaction traffic, the system can maintain efficient operation and identify abnormal transaction behaviors in a timely manner.

[0143] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A network abnormal behavior detection method based on dynamic data balancing and generation, characterized in that: include: Obtaining original network traffic data, inputting the original network traffic data into a dynamic abnormal data balancing model, and obtaining an enhanced data set; The original network traffic data is marked based on the abnormal data balancing model, the marked data is selectively undersampled, the ratio of normal data to abnormal data is adjusted, a balanced data set is obtained, new abnormal data is generated according to the balanced data set, and the new abnormal data is merged with the balanced data set to obtain an enhanced data set; The enhanced data set is input into the TimeDiT denoising model for denoising, and the denoised target data set is obtained. The enhanced data set is input into the LSTM-Transformer weighted training model for network abnormal behavior detection. The abnormal data in the target data set is weighted in combination with the weighted loss function to obtain the detection result.

2. The network abnormal behavior detection method based on dynamic data balancing and generation according to claim 1 is characterized in that: Marking the original network traffic data includes: gradually separating the data points in the raw network traffic data, and generating anomaly scores according to the isolation depth of the data points; The original network traffic data is marked according to the anomaly score; the mark is used to generate classification labels for abnormal data and normal data.

3. The network abnormal behavior detection method based on dynamic data balancing and generation according to claim 1 is characterized in that: Acquiring the balanced data set includes: Cluster analysis is used to cluster the normal data in the labeled data, and the normal data is divided into multiple data subsets. Representative samples in each data subset are selected for storage. At the same time, distance metric analysis is used to save the normal data related to the abnormal data, and the ratio of normal data to abnormal data is dynamically adjusted to obtain the balanced data set.

4. The network abnormal behavior detection method based on dynamic data balancing and generation according to claim 1 is characterized in that: Acquiring the enhanced data set includes: Preprocessing the balanced data set; the preprocessing includes filtering, truncation / filling and normalization; The preprocessed balanced data set is input into the FlowGAN sub-model to generate new abnormal data, and the new abnormal data is merged with the balanced data set to obtain an enhanced data set.

5. The network abnormal behavior detection method based on dynamic data balancing and generation according to claim 4 is characterized in that: The FlowGAN sub-model includes: Generator, used to analyze the real data distribution in the preprocessed balanced data set and generate fake data; A discriminator, used to determine data categories; the categories include: real data and forged data; The training adversary is used to introduce intensity control disturbance in the process of the judger judging the data category, calculate the distance between the real data and the forged data, control the abnormal intensity of the forged data, and thus generate the new abnormal data.

6. The method for detecting abnormal network behavior based on dynamic data balancing and generation according to claim 4, characterized in that: In the process of the game between the generator and the discriminator, the intensity control disturbance is introduced, and the formula for generating the new abnormal data is: Among them, D(x) indicates that the discriminator believes that sample x belongs to the real data distribution p data The probability output is represents the judgment of the discriminator on the real sample x, D(G(z)) represents the probability output of the discriminator that the generated sample G(z) belongs to the real data, represents the judgment of the discriminator on the generated sample G(z), Intensity(G(z)) represents the abnormal intensity of the generated sample, λ· It represents the difference between the anomaly strength of the generated sample G(z) and the set target strength s, and λ represents the balance between controlling the authenticity of the generated sample and the anomaly strength control.

7. The network abnormal behavior detection method based on dynamic data balancing and generation according to claim 1 is characterized in that: Acquiring the denoised target data set includes: The enhanced data set is input into the TimeDiT denoising model, noise is gradually added and denoised through a diffusion reverse process to obtain the denoised target data set.

8. The network abnormal behavior detection method based on dynamic data balancing and generation according to claim 1 is characterized in that: Obtaining the detection result includes: Input the denoised target data set into the LSTM layer in the LSTM-Transformer weighted training model to capture short-term dependencies and obtain short-term dependency features; capturing short-term dependencies means: saving useful short-term information and eliminating useless data; The global contextual relationship in the time series is captured and long-term dependency features are obtained through the self-attention mechanism of the Transformer layer in the LSTM-Transformer weighted training model; According to the short-term dependency features and the long-term dependency features, the abnormal data in the target data set is weighted in combination with a weighted loss function to perform network abnormal behavior detection to obtain the detection result.

9. The method for detecting abnormal network behavior based on dynamic data balancing and generation according to claim 8, characterized in that: The weighted loss function includes: FocalLoss weighted loss function.

10. The method for detecting abnormal network behavior based on dynamic data balancing and generation according to claim 8, characterized in that: The method further comprises: Establish evaluation indicators, evaluate the detection capability of the LSTM-Transformer weighted training model through the evaluation indicators, and adjust the hyperparameters of the LSTM-Transformer weighted training model based on the evaluation results; wherein the evaluation indicators include: precision, recall and F1 score.

Images