DNS anomaly detection method and system based on generative adversarial network
Through the DNS anomaly detection method based on a generative adversarial network, using the combined training of generator and discriminator, generate simulated normal traffic samples and distinguish abnormal traffic, which solves the problem that the existing technology cannot adapt to new attacks and high false alarm rates, and realizes the problem of More accurate abnormal detection and lower false alarm rate.
Patent Information
- Application Number
- CN202411070274.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-12-03
AI Technical Summary
The existing DNS anomaly detection methods cannot effectively adapt to new attacks, and it is difficult to accurately distinguish between normal and abnormal behavior, which is prone to false positives, especially in real-time monitoring of large-scale networks.
Using a generative adversarial network-based method, collects multiple DNS logs, parses and preprocesses data, jointly trains the generator and discriminator, generates simulated normal traffic samples, and uses the discriminator to distinguish between real and abnormal traffic, and defines abnormal detection indicators. , reduce the probability of false positives.
Generative adversarial networks can reduce dependence on labeled data through unsupervised learning, adapt to new attacks, improve the ability to distinguish between normal and abnormal behaviors, and reduce the false positive rate.
Smart Images

Figure CN119071025A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of DNS anomaly detection, and in particular to a DNS anomaly detection method and system based on a generative adversarial network. Background Art
[0002] The current DNS anomaly detection methods mainly include rule-based detection methods, statistics-based detection methods, machine learning-based detection methods and behavior analysis-based detection methods.
[0003] The rule-based detection method analyzes and detects DNS traffic through pre-defined rules, such as detecting the length of the domain name, the character set of the domain name, the frequency of the domain name, etc. The defect of this method is that it cannot adapt to new types of DNS abnormal attacks, and the rule base needs to be constantly updated; the statistical-based detection method detects abnormal behavior through statistical analysis of DNS traffic, such as the request frequency of domain names, the distribution of IP addresses, etc. The defect of this method is that it cannot accurately distinguish between normal traffic and abnormal traffic, and it is easy to generate false positives; the machine learning-based detection method mainly uses machine learning algorithms to model and analyze DNS traffic to identify abnormal behavior. The defect of this method is that a large amount of labeled data is required for training, and the selection of algorithms and the adjustment of parameters are high; the detection method based on behavioral analysis analyzes the behavioral patterns of DNS traffic, such as the order of domain name resolution, the time interval of requests, etc., to detect abnormal behavior. The defect of this method is that it cannot adapt to complex DNS abnormal attacks, and it is difficult to monitor large-scale networks in real time. Summary of the invention
[0004] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.
[0005] Therefore, the purpose of the present invention is to provide a DNS anomaly detection method and system based on a generative adversarial network, so that the generative adversarial network has more potential to adapt to new attacks and can better distinguish between normal and abnormal behaviors, thereby reducing the probability of false positives.
[0006] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:
[0007] A DNS anomaly detection method based on a generative adversarial network, comprising:
[0008] S1. Collect DNS logs from multiple sources;
[0009] S2. Parse the original DNS query response log, convert it into structured data, extract key feature data, remove invalid records, filter out protocol errors or other obviously abnormal data, deduplicate duplicate queries, establish statistical feature data reflecting traffic change trends, convert non-numerical features such as domain names into one-hot encoding or word embedding, and then normalize the features;
[0010] S3, jointly train the generator G and the discriminator N, so that the generator G can simulate normal traffic. After the training converges, use the generator to generate a large number of simulated normal traffic samples;
[0011] S4. Define anomaly detection indicators, input new DNS traffic into the trained model, and determine whether it is abnormal based on the defined anomaly detection indicators;
[0012] S5. Divide the data set into training set, validation set and test set. Evaluate the model's ability in anomaly detection based on the validation set and adjust parameters such as thresholds. Draw a confusion matrix to display the classification effect. Clarify the proportion of true positives, true negatives, false positives and false negatives. Optimize the threshold of the discriminator output according to the preset evaluation criteria. Pursue the best precision, recall rate or F1 score. Then further screen and transform the DNS traffic features input into the GAN model to improve the model's ability to distinguish abnormal traffic.
[0013] As a preferred solution of the DNS anomaly detection method based on a generative adversarial network described in the present invention, in step S1, the DNS logs from multiple sources include real-time or offline data from network devices, log files, intrusion detection systems, and network monitoring tools and traffic analyzers.
[0014] As a preferred solution of the DNS anomaly detection method based on a generative adversarial network described in the present invention, real-time data is collected using a Kafka distributed stream processing platform; offline data is regularly acquired using a flume or dataX synchronization tool.
[0015] As a preferred solution of the DNS anomaly detection method based on a generative adversarial network described in the present invention, in step S3, the generator G and the discriminator N are trained as follows:
[0016] Set training parameters;
[0017] Get real DNS query records;
[0018] The generator generates pseudo samples, and the discriminator is updated by minimizing the binary cross entropy loss. The generator is updated by maximizing the discriminator's discrimination probability of the generated samples. The model performance is evaluated at the end of each epoch and the model checkpoint is saved.
[0019] As a preferred solution of the DNS anomaly detection method based on a generative adversarial network described in the present invention, in step S4, the anomaly detection index is defined as follows: the Euclidean distance, Mahalanobis distance or cosine similarity between the actual traffic and the simulated traffic is calculated, and those exceeding the threshold are marked as anomalies. The probability distribution learned by the generator is used to estimate the density of the new traffic, and those below the threshold are regarded as anomalies.
[0020] A DNS anomaly detection system based on generative adversarial network, comprising:
[0021] The data collection module is configured to: collect DNS logs from multiple sources;
[0022] The data preprocessing module is configured to: parse the original DNS query response log, convert it into structured data, extract key feature data, remove invalid records, filter out protocol errors or other obviously abnormal data, deduplicate duplicate queries, establish statistical feature data reflecting traffic change trends, convert non-numerical features such as domain names into one-hot encoding or word embedding, and then normalize the features;
[0023] The GAN joint training module is configured to: jointly train the generator G and the discriminator N so that the generator G can simulate normal traffic. After the training converges, the generator is used to generate a large number of simulated normal traffic samples;
[0024] The anomaly scoring module is configured to: define anomaly detection indicators, input new DNS traffic into the trained model, and determine whether it is abnormal based on the defined anomaly detection indicators;
[0025] The model performance verification and optimization module is configured to: divide the data set into training set, validation set and test set, evaluate the model's ability in anomaly detection based on the validation set and adjust parameters such as thresholds, draw a confusion matrix to display the classification effect, clarify the proportion of true positives, true negatives, false positives and false negatives, optimize the threshold of the discriminator output according to the preset evaluation criteria, pursue the best accuracy, recall rate or F1 score, and then further screen and transform the DNS traffic features input into the GAN model to improve the model's ability to distinguish abnormal traffic.
[0026] Compared with the prior art, the present invention has the following beneficial effects:
[0027] 1. Generative adversarial networks can learn the distribution characteristics of data through unsupervised learning, so they do not require a large amount of labeled data for training.
[0028] 2. Generative adversarial networks can generate fake DNS traffic similar to normal data by learning a large amount of normal DNS traffic data, and then use the discriminator to distinguish between real DNS traffic and fake DNS traffic, thereby achieving anomaly detection. This ability makes generative adversarial networks more adaptable to new attacks.
[0029] 3. Generative adversarial networks can better distinguish normal and abnormal behaviors by learning the distribution characteristics of data, thereby reducing the probability of false positives. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below in combination with the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:
[0031] Figure 1 A flowchart of a DNS anomaly detection method based on a generative adversarial network according to the present invention;
[0032] Figure 2 This is a principle block diagram of a DNS anomaly detection system based on a generative adversarial network in the present invention. DETAILED DESCRIPTION
[0033] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0034] Secondly, the present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention in detail, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.
[0035] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0036] The present invention provides a DNS anomaly detection method and system based on a generative adversarial network, so that the generative adversarial network has more potential to adapt to new attacks and can better distinguish between normal and abnormal behaviors, thereby reducing the probability of false alarms.
[0037] like Figure 1 As shown in the figure, the DNS anomaly detection method based on generative adversarial network has the following steps:
[0038] (1) Data collection: DNS logs are collected from various sources, such as network devices, log files, intrusion detection systems, network monitoring tools, and traffic analyzers, as well as real-time or offline data. The Kafka distributed stream processing platform is used for real-time data collection to achieve real-time transmission and preprocessing of DNS logs. Offline data is regularly acquired using synchronization tools such as Flume or DataX and imported into the preprocessing module.
[0039] (2) Data preprocessing: including:
[0040] Data collection and formatting: parse the original DNS query response log, convert it into structured data, and extract key features, including but not limited to source IP, destination IP, query domain name, response type, and timestamp information;
[0041] Initial identification and cleaning of anomalies: Eliminate invalid records, filter out protocol errors or other obviously abnormal data, and deduplicate repeated queries.
[0042] Feature engineering: Construct statistical features that reflect traffic change trends, such as query frequency, rate changes, etc., and convert non-numerical features such as domain names into one-hot encoding or word embedding forms.
[0043] Standardization and normalization: Apply methods such as Z-score standardization or mmin-max scaling to ensure that different features are comparable.
[0044] (3) Generative adversarial network architecture. Specifically, in the DNS anomaly detection scenario, the generative adversarial network consists of two parts: the generator (G) and the discriminator (D). These two parts of the network structure undertake different tasks respectively, and jointly improve the learning of DNS behavior patterns and anomaly detection capabilities through adversarial training. The generator G:
[0045] Input: The generator receives as input random noise vectors sampled from some prior distribution (e.g., Gaussian or uniform distribution). These noise vectors represent random seeds in the latent space and are used to generate pseudo samples that simulate normal DNS behavior.
[0046] Structure: The generator usually adopts a deep neural network structure, such as a deep convolutional neural network (CNN), a recurrent neural network (RNN) or its variants, such as a long short-term memory network (LSTM) or a gated recurrent unit (GRU). According to the characteristics of DNS data, a suitable network structure can be selected to capture time series features, pattern repetitiveness or complex network topology relationships.
[0047] Output: The output of the generator is a pseudo sample with the same structure as the real DNS query record. These samples should be as close as possible to the distribution characteristics of real DNS query traffic, including query timestamp, source IP address, destination domain name, returned IP address, query type and other information.
[0048] Discriminator D:
[0049] Input: The discriminator receives two sets of inputs: one is the real DNS query record, and the other is the pseudo sample generated by the generator. These inputs are feature extracted and encoded to form a vector representation that the discriminator can process.
[0050] Structure: The discriminator also uses a deep neural network structure, such as multi-layer perceptron (MLP), CNN, RNN and its variants. In DNS anomaly detection, the discriminator may need to consider time series context information, so it may choose to use a model with time dependency, such as LSTM or Bi-LSTM.
[0051] Output: The discriminator outputs a probability value, indicating the probability that the input sample belongs to normal DNS behavior. Ideally, for real DNS query records, the output probability should be close to 1, while for pseudo samples generated by the generator, the output probability should be close to 0. During the training process, the goal of the discriminator is to maximize the ability to correctly distinguish between real samples and generated samples.
[0052] (4) Joint training and application
[0053] The generator and discriminator are jointly trained so that the generator can simulate normal traffic and the discriminator can accurately distinguish between real traffic and simulated traffic. After the training converges, the generator is used to generate a large number of simulated normal traffic samples.
[0054]
[0055]
[0056]
[0057] In the above pseudo code, iterate_over_batches(batch_size) means getting DNS query record data in batches, get_real_dns_records(batch) returns the real DNS query records of the current batch, sample_from_prior_distribution(batch_size) generates a specified number of noise vectors, binary_cross_entropy() calculates the binary cross entropy loss, ones_like() and zeros_like() generate all-1 or all-0 tensors with the same shape as the input, and apply_gradients() performs backpropagation and parameter updates.
[0058] Through the above joint training process, the generator G continuously tries to generate more realistic DNS query records to deceive the discriminator D, while the discriminator D strives to improve its ability to distinguish between real records and generated records. As the training progresses, the pseudo samples generated by G are closer and closer to the real data distribution, while D's ability to identify normal DNS behavior patterns becomes more and more accurate.
[0059] (5) Abnormality score
[0060] Defining anomaly detection metrics
[0061] Statistical testing: Compare the statistical characteristics of real traffic and simulated traffic, and calculate relevant metrics such as mean and variance.
[0062] Distance measurement: Calculate the Euclidean distance, Mahalanobis distance or cosine similarity between the actual traffic and the simulated traffic, and mark those exceeding the threshold as abnormal.
[0063] Density estimation: Use the probability distribution learned by the generator to estimate the density of new traffic, and those below the threshold are considered abnormal.
[0064] Evaluation of traffic to be tested
[0065] Real-time detection: Input new DNS traffic into the trained model and determine whether it is abnormal based on the above abnormal indicators.
[0066] Offline analysis: Batch analyze historical DNS traffic data to find traffic that does not conform to normal patterns.
[0067] Scoring Methodology
[0068] Based on the discriminator output:
[0069] During the training process, the discriminator D learns to distinguish between normal DNS query records and pseudo samples generated by the generator G. When the training is completed, the discriminator outputs a probability value between 0 and 1 for new input samples, indicating the possibility that the sample belongs to normal behavior.
[0070] For the DNS query records to be detected, we can directly use the output of the discriminator as the anomaly score. The lower the score, the more the sample deviates from the normal behavior pattern and the more likely it is an anomaly.
[0071] Set a global or dynamically adjusted threshold to mark DNS query records with anomaly scores higher than the threshold as abnormal. The selection of the threshold can be based on statistical analysis of historical data (such as setting it to the mean of abnormal samples + several times the standard deviation), or determine the optimal threshold through cross-validation, ROC curve analysis, etc.
[0072] The following is a pseudocode example of a simple threshold anomaly detection based on the discriminator output:
[0073]
[0074] (6) Model performance verification and optimization
[0075] Evaluation indicator setting
[0076] Use precision, recall, and F1 score to measure model performance, with special attention paid to the performance on imbalanced datasets.
[0077] Performance Verification
[0078] Cross-validation: Divide the dataset into training set, validation set, and test set. Based on the validation set, evaluate the model's ability to detect anomalies and adjust parameters such as thresholds.
[0079] Confusion Matrix: Draw a confusion matrix to show the classification effect and clarify the ratio of true positive examples, true negative examples, false positive examples, and false negative examples.
[0080] Performance Optimization
[0081] Threshold adjustment: Optimize the threshold of the discriminator output according to the preset evaluation criteria, pursuing the best precision, recall rate or F1 score.
[0082] Feature selection and engineering: Further filter and transform the DNS traffic features input into the GAN model to improve the model's ability to distinguish abnormal traffic.
[0083] like Figure 2 As shown, a DNS anomaly detection system based on a generative adversarial network includes:
[0084] The data collection module is configured to: collect DNS logs from multiple sources;
[0085] The data preprocessing module is configured to: parse the original DNS query response log, convert it into structured data, extract key feature data, remove invalid records, filter out protocol errors or other obviously abnormal data, deduplicate duplicate queries, establish statistical feature data reflecting traffic change trends, convert non-numerical features such as domain names into one-hot encoding or word embedding, and then normalize the features;
[0086] The GAN joint training module is configured to: jointly train the generator G and the discriminator N so that the generator G can simulate normal traffic. After the training converges, the generator is used to generate a large number of simulated normal traffic samples;
[0087] The anomaly scoring module is configured to: define anomaly detection indicators, input new DNS traffic into the trained model, and determine whether it is abnormal based on the defined anomaly detection indicators;
[0088] The model performance verification and optimization module is configured to: divide the data set into training set, validation set and test set, evaluate the model's ability in anomaly detection based on the validation set and adjust parameters such as thresholds, draw a confusion matrix to display the classification effect, clarify the proportion of true positives, true negatives, false positives and false negatives, optimize the threshold of the discriminator output according to the preset evaluation criteria, pursue the best accuracy, recall rate or F1 score, and then further screen and transform the DNS traffic features input into the GAN model to improve the model's ability to distinguish abnormal traffic.
[0089] Although the present invention has been described above with reference to the embodiments, various modifications may be made thereto and parts thereof may be replaced by equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the embodiments disclosed in the present invention may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.
Claims
1. A DNS anomaly detection method based on generative adversarial network, characterized in that: include: S1. Collect DNS logs from multiple sources; S2. Parse the original DNS query response log, convert it into structured data, extract key feature data, remove invalid records, filter out protocol errors or other obviously abnormal data, deduplicate duplicate queries, establish statistical feature data reflecting traffic change trends, convert non-numerical features such as domain names into one-hot encoding or word embedding, and then normalize the features; S3, jointly train the generator G and the discriminator N, so that the generator G can simulate normal traffic. After the training converges, use the generator to generate a large number of simulated normal traffic samples; S4. Define anomaly detection indicators, input new DNS traffic into the trained model, and determine whether it is abnormal based on the defined anomaly detection indicators; S5. Divide the data set into training set, validation set and test set. Evaluate the model's ability in anomaly detection based on the validation set and adjust parameters such as thresholds. Draw a confusion matrix to display the classification effect. Clarify the proportion of true positives, true negatives, false positives and false negatives. Optimize the threshold of the discriminator output according to the preset evaluation criteria. Pursue the best precision, recall rate or F1 score. Then further screen and transform the DNS traffic features input into the GAN model to improve the model's ability to distinguish abnormal traffic.
2. According to claim 1, a DNS anomaly detection method based on a generative adversarial network is characterized in that: In step S1, the DNS logs from various sources include network devices, log files, intrusion detection systems, and real-time or offline data from network monitoring tools and traffic analyzers.
3. According to claim 1, a DNS anomaly detection method based on a generative adversarial network is characterized in that: Real-time data is collected using the Kafka distributed stream processing platform; offline data is regularly acquired using the Flume or DataX synchronization tools.
4. According to claim 1, a DNS anomaly detection method based on a generative adversarial network is characterized in that: In step S3, the generator G and the discriminator N are trained as follows: Set training parameters; Get real DNS query records; The generator generates pseudo samples, and the discriminator is updated by minimizing the binary cross entropy loss. The generator is updated by maximizing the discriminator's discrimination probability of the generated samples. The model performance is evaluated at the end of each epoch and the model checkpoint is saved.
5. According to claim 1, a DNS anomaly detection method based on a generative adversarial network is characterized in that: In step S4, the anomaly detection index is defined as follows: the Euclidean distance, Mahalanobis distance or cosine similarity between the actual flow and the simulated flow is calculated, those exceeding the threshold are marked as anomalies, and the probability distribution learned by the generator is used to estimate the density of the new flow, and those below the threshold are considered anomalies.
6. A system for implementing the DNS anomaly detection method based on a generative adversarial network as described in any one of claims 1 to 5, characterized in that: include: The data collection module is configured to: collect DNS logs from multiple sources; The data preprocessing module is configured to: parse the original DNS query response log, convert it into structured data, extract key feature data, remove invalid records, filter out protocol errors or other obviously abnormal data, deduplicate duplicate queries, establish statistical feature data reflecting traffic change trends, convert non-numerical features such as domain names into one-hot encoding or word embedding, and then normalize the features; The GAN joint training module is configured to: jointly train the generator G and the discriminator N so that the generator G can simulate normal traffic. After the training converges, the generator is used to generate a large number of simulated normal traffic samples; The anomaly scoring module is configured to: define anomaly detection indicators, input new DNS traffic into the trained model, and determine whether it is abnormal based on the defined anomaly detection indicators; The model performance verification and optimization module is configured to: divide the data set into training set, validation set and test set, evaluate the model's ability in anomaly detection based on the validation set and adjust parameters such as thresholds, draw a confusion matrix to display the classification effect, clarify the proportion of true positives, true negatives, false positives and false negatives, optimize the threshold of the discriminator output according to the preset evaluation criteria, pursue the best accuracy, recall rate or F1 score, and then further screen and transform the DNS traffic features input into the GAN model to improve the model's ability to distinguish abnormal traffic.
