A DGA detection method based on character and behavior double features

Abstract

A DGA detection method based on character and behavior double features, comprising the following steps: 1) collecting NXDomain domain names that fail to be parsed in original traffic data, performing data preprocessing and filtering, and dividing the data into a training set, a validation set and a test set for subsequent feature extraction model; 2) first using the training set and the validation set data to train the self-encoder based on bidirectional LSTM, and then inputting the test set data after the training is completed, and taking the output of the encoder as the character feature vector extracted from the domain name; 3) converting all the obtained data based on the domain name and IP request relationship into a domain-host request bipartite graph, taking the sparse adjacency matrix and weight thereof as model input, and extracting the representation vector of each node as the behavior feature vector of the domain name; 4) adopting hierarchical clustering to cluster and group the behavior feature vectors; the application can efficiently find the DGA domain name based on Domain-flux in actual network traffic, and solve the problem of high false alarm rate for pinyin domain names.

Description

Technical Field

[0001] This invention belongs to the field of computer network security and deep learning technology, specifically relating to a DGA detection method based on dual features of characters and behavior. Background Technology

[0002] Currently, the main threats to cybersecurity come from malware, which largely operates through domain names. This includes botnets that distribute malicious programs, phishing and spam domains that disrupt users' lives, Trojan horses that manipulate victim hosts, and advanced persistent threats (APTs) that steal state secrets. In botnets, attackers use domain name generation algorithms to generate a large number of DGA domains, initiate connections to control and command servers, and implant malware carrying viruses into the botnet hosts, enabling them to perform malicious activities such as ransomware attacks and distributed denial-of-service attacks. Therefore, given the increasingly severe cyber environment, researching domain name security assessment technologies and malicious domain name detection methods is of great significance for combating cybercrime and protecting the world's cyber environment.

[0003] Numerous studies, both domestically and internationally, have been conducted on the detection of malicious domain names, achieving significant progress. Single-domain-based detection methods effectively extract domain name features from textual information and have shown good detection results in many experiments. However, due to their focus on a single feature, they are only applicable to randomly generated DGA domains like those from Domain-flux, while failing to detect many dictionary-based malicious domains and generating false positives for domestic pinyin domains. Furthermore, these methods do not consider the behavioral characteristics of domain names, ignoring the relationship between domain names and their requesting hosts in actual traffic, leading to high false positive rates in real-world networks and poor applicability. Detection methods based on DNS traffic extension features focus on their behavioral relationships within the network. However, because the extracted features rely on feature engineering and use machine learning models as a framework, they primarily employ linguistic and statistical features of domain characters, such as n-grams and character distances, neglecting hidden features within the text. This makes them vulnerable to attackers, and frequent modifications to the domain name generation seed and algorithm can render the detection method ineffective, failing to capture the latest active malicious domains in a timely manner.

[0004] Therefore, researching detection methods that can efficiently and accurately detect DGA domain names has significant research value and application value. Summary of the Invention

[0005] In order to overcome the problems existing in the prior art, the purpose of this invention is to provide a DGA detection method based on dual features of characters and behavior, which can efficiently find DGA domains based on Domain-flux in actual network traffic and solve the problem of high false positive rate for Pinyin domains in the past.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A DGA detection method based on dual character and behavior features includes the following steps;

[0008] 1) Data preprocessing and filtering:

[0009] From the raw traffic data collected from the Domain Name System (DNS) of a certain Internet Service Provider (ISP), NXDomain domains that failed to resolve were filtered out. Data preprocessing and filtering were performed, and the data was divided into training set, validation set and test set for subsequent feature extraction model.

[0010] 2) Character feature extraction:

[0011] First, use the training set and validation set data from step 1) to train the autoencoder based on bidirectional LSTM. After training, input the test set data from step 1) and use the encoder's output as the character feature vector extracted from the domain name.

[0012] 3) Behavioral feature extraction:

[0013] All the data obtained in step 1) is converted into a domain-host request bipartite graph based on the domain name and IP request relationship. The domain name is used as the node of the graph, the behavior relationship of IP requesting domain name is used as the edge of the graph, and the number of common requesting IPs between each pair of domain names is used as the weight of the edge. The sparse adjacency matrix and weight are used as the model input, and the representation vector of each node is extracted by the node2vec algorithm as the extracted domain name behavior feature vector.

[0014] 4) Clustering:

[0015] Hierarchical clustering is used to cluster and group the behavioral feature vectors. First, the behavioral features obtained in step 3) are used to find the group domains with highly overlapping behavioral patterns. Then, the character features obtained in step 2) are used to find the domains with similar structures in these group domains. Finally, a reasonable threshold is set through experiments to filter out suspected malicious domains.

[0016] Furthermore, the specific steps of data preprocessing and filtering are as follows: First, remove illegal top-level domains and domains that do not conform to the naming rules; second, filter domain request data suspected of being from Network Address Translation (NAT), DNS servers, cloud services, etc.; finally, remove suspected misspelled domains with too few requesting users; after filtering, use database tools and group join methods to merge the discrete multi-table data into data with the second-level domain as the primary key, containing the set of IPs requesting the second-level domain and the number of IPs.

[0017] Furthermore, in step 1), before transmitting data to the character feature extraction model, the domain name string is first converted according to the following rules: the dictionary is set to all legal characters that make up the domain name, including the numbers 0-9, lowercase letters a and b, the English period (.), and the hyphen (-). The dictionary size is 38. Each character in the second-level field is converted to its corresponding index according to its position in the dictionary. Finally, zero padding is performed at the beginning to ensure that the length of all sequences is 35, which is convenient for model processing. After the above conversion, the original variable-length domain name string becomes a fixed 35-dimensional token sequence.

[0018] Furthermore, in step 2), the autoencoder consists of an input layer, an embedding layer, a two-layer encoder, and a two-layer decoder. The input layer receives the transformed token sequence vector from step 1), and the embedding layer converts the vector from the input layer into a fixed-size 35×64-dimensional embedding vector. The encoder consists of two different bidirectional LSTM layers. The first layer outputs a 35×128-dimensional vector, and the second layer outputs a 64-dimensional vector, which serves as the domain name character features extracted by the model. This latent vector needs to be converted to 35×64 dimensions by the RepeatVector layer before entering the decoder. The decoder is responsible for reconstructing the initial data from the vector received from the encoder. Its composition is similar to that of the encoder, also containing two bidirectional LSTM layers. The first layer outputs a 35×64-dimensional vector, and the second layer outputs a 35×128-dimensional vector. Finally, after passing through a Dense layer and using the softmax activation function, it outputs a 35×38-dimensional vector, where each bit represents the probability of a different character. The loss function used by the model is sparse cross-entropy, which is optimized by the RMSprop optimizer.

[0019] Furthermore, in step 3), the domain-host request bipartite graph is converted into a network graph based on request relationships according to the following rules: all domain names are treated as nodes in the network graph, the relationship between IP requests to domain names is treated as edges in the network graph, and the number of common requesting IPs between any two domain names is treated as the edge weight. To obtain the sparse adjacency matrix of this network graph, each pair of nodes with edges in the graph is traversed, and the edge weights are normalized to obtain N*3 data, which is then fed into the node2vec algorithm model, where N is the number of edges in the network graph, the first column is the starting node of the traversal, the second column is the ending node of the traversal, and the third column is the edge weight.

[0020] Furthermore, in step 4), feature clustering is performed twice in sequence. First, clustering is performed based on the domain-host behavior feature vector to find clusters with high overlap of request host sequences. Then, clustering is performed on each cluster with similar behavior based on the character features of the domain name to obtain the final clustering result, that is, a set of domain names with high overlap of behavior and highly similar character composition, which serves as a suspected DGA domain name set.

[0021] The beneficial effects of this invention are:

[0022] The DGA detection model proposed in this invention uses an autoencoder and DNS data for unsupervised learning training, eliminating the need for labeled samples. By analyzing domain name data in actual network traffic, it extracts the latent features of characters in text composition, simplifying feature engineering and making it easy to use. By introducing the node2vec graph algorithm, complex network behavior relationships are converted into embedding vectors that the model can process, solving the behavior loss problem of previous single-character feature-based models and identifying malicious members of different versions of the same malicious family. Through secondary clustering, domain names with similar behavioral characteristics are first clustered, avoiding the false positive problem of previous models for pinyin domain names and prioritizing the capture of hosts in suspected botnets. Then, by clustering character features, DGAs generated by different algorithms within the same DGA family can be obtained, which is beneficial for timely detection of new versions of existing DGA families.

[0023] This invention solves the problems of previous methods that missed detection of many dictionary-based malicious domain names and false detection of domestic pinyin domain names. It can also efficiently find Domain-flux-based DGA domain names in actual network traffic and has broad application prospects. Attached Figure Description

[0024] Figure 1 This is a flowchart of the method of the present invention;

[0025] Figure 2 This is a schematic diagram of the model framework of the present invention;

[0026] Figure 3 This is a schematic diagram of the domain name string conversion process in the character feature extraction module;

[0027] Figure 4 This is a schematic diagram of the structure and parameters of an autoencoder based on bidirectional LSTM;

[0028] Figure 5 This is a schematic diagram of the request behavior network construction process of the behavior feature extraction module;

[0029] Figure 6(a) is a schematic diagram of the cumulative distribution of IP request domain names;

[0030] Figure 6(b) is a schematic diagram of the cumulative distribution of domain name request hosts;

[0031] Figure 7(a) is a schematic diagram showing the cumulative distribution of the number of characters in reconstruction errors;

[0032] Figure 7(b) is a schematic diagram of the domain cosine similarity and Jaccard coefficient curve;

[0033] Figure 8 This is a schematic diagram of the DGA family distribution;

[0034] Figure 9(a) is a schematic diagram of known DGAs discovered by the model;

[0035] Figure 9(b) is a schematic diagram of the unknown DGA discovered by the model. Detailed Implementation

[0036] The present invention will now be described in further detail with reference to the accompanying drawings.

[0037] This invention provides a DGA detection method based on dual character and behavioral features, involving theoretical knowledge in network security, malicious domain name detection, deep learning, and graph theory. This invention can be effectively applied to detect Domain-flux-based DGA domains in real-world networks, solving the problem of high false positive rates for Pinyin domain names. Its operation process is as follows: Figure 1 As shown.

[0038] Specifically, the detection model based on character and behavioral features in this embodiment of the invention employs an autoencoder structure, a node2vec algorithm, and a clustering algorithm, ultimately outputting the suspected DGA domain names predicted by the model. The specific process of the model is as follows: Figure 2 As shown.

[0039] Preferably, the model in this embodiment of the invention uses the DNS traffic of the actual network throughout the day as input, without the need for labeled samples, which saves data costs compared with previous technologies and is more suitable for actual network environments.

[0040] Preferably, the model in this embodiment of the invention uses an autoencoder as the main architecture for extracting character features. It does not require the extraction of statistical features of the domain name string, which is more convenient for experimentation compared with previous technologies. It simplifies the domain name character feature extraction process and can directly obtain the domain name embedding vector for downstream tasks.

[0041] Specifically, embodiments of the present invention include the following steps:

[0042] 1) Data preprocessing and filtering:

[0043] The NXDomain domain was filtered three times. First, illegal top-level domains and domains that did not conform to the naming rules were removed. Second, domain request data suspected of being NAT, DNS server, cloud service, etc. were filtered to remove IPs whose number of failed host requests exceeded the threshold N = 200. The threshold N was determined by the cumulative distribution map of the host requesting the domain, as shown in Figure 6(a). Finally, domains whose number of requesting users exceeded the threshold M = 200 were filtered out. The threshold M was determined by the cumulative distribution map of the IP requesting the domain, as shown in Figure 6(b).

[0044] The specific statistics of the original and preprocessed datasets are shown in Table 1. After the first filtering, 16,540,148 records containing invalid domain names were removed, leaving 49,756,558 invalid domain names that conformed to the naming rules. After the second filtering, 6,592,690 domain name request records related to NAT devices and DNS servers were removed, leaving 619,833 domain names after deduplication. After the third filtering, domain names with an excessive number of requesting users were removed, leaving 27,534 domain names.

[0045] Table 1. Experimental raw dataset and preprocessing statistics

[0046]

[0047] 2) Character feature extraction:

[0048] Before passing data to the character feature extraction model, the domain name string is first converted. According to domain name naming conventions, the dictionary is set to contain all legal characters that make up the domain name, namely the numbers 0-9, lowercase letters a and b, the period (.), and the hyphen (-), with a size of 38. Each character in the second-level domain is converted to its corresponding index (0-38) based on its position in the dictionary, and zero-padding is applied at the beginning to ensure consistent sequence length. The specific conversion process of the domain name string is as follows... Figure 3As shown, for the domain name "www.baidu.com", the second-level domain part "baidu" is first taken, and converted into the sequence "[2 1 9 4 21]" according to the dictionary. Finally, the sequence is zero-padded at the beginning to become a sequence of length 35, which is used as the data to be passed to the input layer of the autoencoder.

[0049] The domain name character feature extraction model consists of a bidirectional LSTM-based autoencoder, including an input layer, an embedding layer, a two-layer encoder, and a two-layer decoder. Specific parameter settings and procedures are as follows: Figure 4 As shown, the encoder consists of two different bidirectional LSTM layers. The first layer outputs a 35×128-dimensional vector, and the second layer outputs a 64-dimensional vector, which serves as the domain name character features extracted by the model. Before entering the decoder, the latent vectors need to be transformed into 35×64-dimensional vectors through a RepeatVector layer. The decoder is responsible for reconstructing the initial data from the vectors received from the encoder. Its composition is similar to that of the encoder, also containing two bidirectional LSTM layers. The first layer outputs a 35×64-dimensional vector, and the second layer outputs a 35×128-dimensional vector. Finally, after passing through a Dense layer, a softmax activation function is used to output a 35×38-dimensional vector, where each digit represents the probability of a letter. The loss function used by the autoencoder is sparse cross-entropy, which is optimized using the RMSprop optimizer.

[0050] 3) Behavioral feature extraction:

[0051] Define an undirected graph of domain name relationships G = (V, E), and denote the set of domain names as D = {d1, d2, d3, ..., d...}. n The corresponding set of nodes is V = {v1, v2, v3, ..., v}. n}, node v i Corresponding domain name d i Q i For domain d i The set of requesting IPs. To construct the request behavior network graph, the following rule is established: for any two nodes v in the graph... i v j If it exists Then v i v j There is an edge between them, with edge weight e. ij =|Q i ∩Q j After completing the graph construction, the edge weights are normalized.

[0052] The transformation process from a domain-host request bipartite graph to a request-relationship-based network graph is as follows: Figure 5As shown. After obtaining the network graph, each pair of nodes with an edge is traversed, and the number of elements in the intersection of the requested IPs is used as the edge weight. Finally, to obtain the value of each node v in the graph... i The embedding vectors are linearly normalized (Min-Max Scaling) for edge weights, and the sparse adjacency matrix of graph G is used as input to the node2vec algorithm model, which is then trained using Spark.

[0053] 4) Clustering:

[0054] After character feature extraction and behavioral feature extraction, the data yields two feature vectors. During clustering, the system uses these two feature vectors to identify suspected DGA families active in the network. First, clustering is performed based on the domain-host behavioral feature vectors to identify clusters with high overlap in request host sequences. Then, each cluster with similar behavior is further clustered based on the domain's character features, resulting in the final clustering result: a set of domains with high behavioral overlap and highly similar character structures, considered as the suspected DGA domain set. If a strategy of clustering character features first and then behavioral features is adopted, benign Pinyin domains might be falsely flagged as malicious during the first clustering, and different versions of DGA families would be impossible to separate. Therefore, behavioral feature clustering is prioritized.

[0055] To test the accuracy and superiority of the present invention, the following experiments were conducted to further illustrate its effectiveness.

[0056] 1) Domain Name Character Feature Validity Test Experiment

[0057] In the experiment, the autoencoder based on bidirectional LSTM, employing sparse classification cross-entropy, achieved reconstruction accuracy of 94.72% and 95.97% on the training and test sets, respectively, with reconstruction errors of 0.1501 and 0.1139. Because the domain name conversion sequence contains zero-padding elements, shorter domain names with a fixed length will contain more zero characters, potentially leading to inflated reconstruction accuracy calculations. For example, for the domain name "360.com," its string contains only 3 characters representing the second-level domain; the remaining 32 characters are zero-padding additions. When all characters are correctly reconstructed, the accuracy is 100%; with only one incorrect character, the accuracy is 97.14%. If zero-padding is ignored, the accuracy should be 66.67%, failing to accurately represent the autoencoder's reconstruction performance. To avoid the impact of zero-padding on reconstruction accuracy and error, the following domain name character feature validity test was redesigned.

[0058] First, we analyze the domain name lengths, and their distribution is as follows: Figure 3-8As shown, most domain names are between 12 and 23 characters long, with very few being less than 10 characters or more than 35 characters.

[0059] As shown in Figure 7(a), in the character reconstruction effect test, the situation before and after reconstruction of each domain name was compared, and the number of characters with reconstruction errors was counted to verify the performance of the autoencoder. The cumulative distribution is as follows: Figure 3 As shown in Figure -9, 93.01% of the domain names were completely restored correctly, 5.38% of the domain names had only one character reconstruction error, and the remaining 1.61% of the domain names contained two or more character errors. Since only 13 domain names were longer than 25 characters, accounting for 0.07% of the total number of domain names, the model did not learn the characteristics of long-character domain names well, resulting in poor reconstruction performance.

[0060] From the perspective of the overall reconstruction effect of the domain name, the autoencoder based on bidirectional LSTM can accurately extract the character features of the domain name.

[0061] 2) Validity testing of user behavior features

[0062] To verify the effectiveness of the behavioral features, a domain name d0 was randomly selected, and its Jaccard coefficient and cosine similarity were compared with those of domain names in the same cluster and other clusters in the clustering results. The results are shown in Figure 7(b). For ease of observation, the calculated cosine similarity was normalized when plotting the graph. Among the 50 selected domain names, the first 10 domain names are members of the same cluster as d0 (the first being d0 itself), and the last 40 are domain names in other clusters. As can be seen from the figure, domain names within the same cluster have high cosine similarity and Jaccard coefficients, while domain names in different clusters have low correlation, indicating that the node2vec algorithm can accurately extract the behavioral features of domain names.

[0063] Through blacklist matching, 569 known malicious domains were found out out of 27,534 domains, accounting for 2.07% of the total number of domains. Their distribution is as follows: Figure 8 As shown in the figure. In the behavioral feature clustering, clusters with fewer than 20 domain names were filtered out, leaving 34 clusters. In the subsequent character feature clustering, the same parameters were used to obtain 86 clusters. After removing clusters containing fewer than 20 domain names, a final 13 clusters were obtained, totaling 6957 suspected malicious domain names.

[0064] After clustering, the model identified three known DGAs, as shown in Figure 9(a). Further analysis of the clustering results revealed that all qsnatch family members were correctly clustered into the same cluster, while virut and conficker family members were divided into two separate clusters based on their versions. Domains from the suppobox and nymaim families were filtered out due to insufficient generation, and no related family members were detected. The model also discovered eight previously undetected new DGAs, as shown in Figure 9(b).

[0065] Furthermore, the model proposed in this invention can also classify different versions of DGA within the same family into different clusters, which is beneficial for the timely detection of new versions of known malicious families. For example, in the experiment, three malicious domains of the Virut family, "aisino.com", "zegelo.com" and "uuuwin.com", all conform to the family's regular expression "[az]{6}\.com$", but the first two have 0 overlapping IPs and a union of 96, so they are classified into two different clusters; while the latter two have 24 overlapping IPs and a union of 49, with a Jaccard coefficient of 0.49, so they are classified into the same cluster. In the experiment, the model proposed in this invention successfully distinguished malicious members of the two versions of the Virut family and the Conficker family.

Claims

1. A DGA detection method based on dual character and behavior features, characterized in that, Includes the following steps; 1) Data preprocessing and filtering: From the raw traffic data collected from the domain name system of a certain network service provider, NXDomain domains that failed to resolve were selected, and data preprocessing and filtering were performed. The data was divided into training set, validation set and test set for subsequent feature extraction model. After filtering, the discrete multi-table data is merged into data with a second-level domain as the primary key, containing the set of IPs requesting that second-level domain and the number of IPs. 2) Character feature extraction: First, use the training set and validation set data from step 1) to train the autoencoder based on bidirectional LSTM. After training, input the test set data from step 1) and use the encoder's output as the character feature vector extracted from the domain name. 3) Behavioral feature extraction: All the data obtained in step 1) is converted into a domain-host request bipartite graph based on the domain name and IP request relationship. The domain name is used as the node of the graph, the behavior relationship of IP requesting domain name is used as the edge of the graph, and the number of common requesting IPs between each pair of domain names is used as the weight of the edge. The sparse adjacency matrix and weight are used as the model input, and the representation vector of each node is extracted by the node2vec algorithm as the extracted domain name behavior feature vector. 4) Clustering: Two feature clustering operations are performed sequentially. First, clustering is performed based on the domain-host behavior feature vector to find clusters with high overlap in request host sequences. Then, clustering is performed separately for each cluster with similar behavior based on the character features of the domain name to obtain the final clustering result, which is a set of domain names with high overlap in behavior and highly similar character composition, as a suspected DGA domain name set.

2. The DGA detection method based on dual character and behavior features according to claim 1, characterized in that, The specific steps of data preprocessing and filtering are as follows: First, remove illegal top-level domains and domains that do not conform to the naming rules; second, filter domain request data that are suspected of being from network address translation, DNS servers, and cloud services; and finally, remove suspected misspelled domains with too few requesting users.

3. The DGA detection method based on dual character and behavior features according to claim 1, characterized in that, In step 1), before transmitting data to the character feature extraction model, the domain name string is first converted according to the following rules: the dictionary is set to all legal characters that make up the domain name, including numbers 0-9, lowercase letters a and b, English periods (.) and hyphens (-), and the dictionary size is 38. Each character in the second-level field is converted to its corresponding index according to its position in the dictionary. Finally, zero padding is performed at the beginning to ensure that the length of all sequences is 35, which is convenient for model processing. After the above conversion, the original variable-length domain name string becomes a fixed 35-dimensional token sequence.

4. The DGA detection method based on dual character and behavior features according to claim 1, characterized in that, In step 2), the autoencoder consists of an input layer, an embedding layer, a two-layer encoder, and a two-layer decoder. The input layer receives the token sequence vector transformed in step 1), and the embedding layer converts the vector from the input layer into a fixed-size 35×64-dimensional embedding vector. The encoder consists of two different bidirectional LSTM layers. The first layer outputs a 35×128-dimensional vector, and the second layer outputs a 64-dimensional latent vector, which serves as the domain name character features extracted by the model. Before entering the decoder, this latent vector needs to be transformed into a 35×64-dimensional size through a RepeatVector layer. The decoder is responsible for reconstructing the initial data from the vector received from the encoder. Its composition is similar to that of the encoder, also containing two bidirectional LSTM layers. The first layer outputs a 35×64-dimensional vector, and the second layer outputs a 35×128-dimensional vector. Finally, after passing through a Dense layer and using the softmax activation function, it outputs a 35×38-dimensional vector, where each bit represents the probability of a different character. The loss function used by the model is sparse cross-entropy, which is optimized by the RMSprop optimizer.

5. The DGA detection method based on dual character and behavior features according to claim 1, characterized in that, In step 3), the domain-host request bipartite graph is converted into a network graph based on request relationships according to the following rules: all domain names are nodes in the network graph, the relationship between IP requests to domain names is an edge in the network graph, and the number of common requesting IPs between any two domain names is the edge weight. In order to obtain the sparse adjacency matrix of the network graph, each pair of nodes with an edge is traversed, and the edge weights are normalized to obtain N*3 data as input to the node2vec algorithm model, where N is the number of edges in the network graph, the first column is the starting node of the traversal, the second column is the ending node of the traversal, and the third column is the edge weight.

Images