DGA detection model training method, detection method and device based on subword segmentation

Abstract

The present invention discloses a DGA detection model training method, detection method and device based on subword segmentation, including obtaining sample traffic data, and extracting the main domain name and top-level domain name corresponding to each piece of sample traffic data to form a first domain name file; segmenting each domain name in the first domain name file to obtain a fragment file containing each domain name input sequence fragment; constructing a vector mapping dictionary based on the frequency of occurrence of whole words and / or subwords in the fragment file; encoding the input sequence fragment of each domain name using the vector mapping dictionary to obtain a corresponding encoding vector; adding training labels to each encoding vector to train the DGA detection model as training data. The encoding vector constructed by the method disclosed in the present invention takes into account the composition structure and inter-word order of the domain name, and the domain name is classified by the generated sentence encoding vector, so that low-randomness DGA families and dictionary-based DGA families can be accurately detected.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a DGA detection model training method, a detection method and a device based on subword segmentation. Background Art

[0002] Botnets or malware use the DGA algorithm to generate a large number of candidate domain names on the terminal. When an attack is needed, a small number of domain names are selected for registration, so that communication can be established with their C&C and then the attack can be launched. At the same time, the attacker can use technologies such as IP Flux to quickly change the IP of the registered domain name. This method can effectively evade traditional blacklist-based detection technology. When botnets use DGA technology, the update speed of the blacklist is far slower than the generation speed of DGA domain names. The DGA detection method based on machine learning determines whether the domain name is generated by the DGA algorithm, thereby blocking the connection between the DGA domain name and its C&C, and curbing the occurrence of network attacks from the root.

[0003] At present, DGA detection schemes based on machine learning usually adopt classification algorithms, which first segment the domain name by single characters or extract Ngrams, then train the word vectors of characters or Ngrams in combination with classification tasks, and use the word vector sequence of the domain name as the input of recurrent neural network (LSTM or GRU) or deep convolutional neural network for classification. Since the existing schemes only use single characters (unigram) or character-level NGrams to generate word vectors, they cannot detect low-random DGA families and families of DGA domain names generated based on dictionaries. Summary of the invention

[0004] The embodiment of the present invention provides a DGA detection model training method, a detection method and a device based on subword segmentation, which considers the composition structure and word order of the domain name, classifies the domain name through the generated sentence vector, and accurately detects low-random DGA families and dictionary-based DGA families.

[0005] In a first aspect, an embodiment of the present invention provides a DGA detection model training method based on subword segmentation, comprising:

[0006] Obtain sample traffic data, and extract the primary domain name and top-level domain name corresponding to each piece of sample traffic data to form a first domain name file;

[0007] Segmenting each domain name in the first domain name file based on a pre-built word segmentation dictionary to obtain a segment file containing each domain name input sequence segment, wherein the input sequence segment of each domain name includes a whole word and / or a subword and a corresponding N-Gram of word granularity;

[0008] Building a vector mapping dictionary based on the occurrence frequency of whole words and / or subwords and word-granular N-Gram in the segment file;

[0009] Encoding the input sequence segments of each domain name using the vector mapping dictionary to obtain corresponding encoding vectors;

[0010] Add training labels to each encoded vector to serve as training data for training the DGA detection model.

[0011] In some embodiments, the word segmentation dictionary is constructed in the following manner:

[0012] Based on the encyclopedia corpus, the word segmentation dictionary is constructed using a preset segmentation algorithm, wherein the word segmentation dictionary includes whole words and subwords.

[0013] In some embodiments, segmenting each domain name in the first domain name file based on a pre-built word segmentation dictionary to obtain a segment file containing each domain name input sequence segment includes:

[0014] Based on the word segmentation dictionary, according to the forward maximum matching principle, each domain name in the first domain name file is divided into sequence segments of whole words and subwords;

[0015] An N-Gram of the word granularity of the sequence segment is added to the sequence segment of each domain name to form an input sequence segment of each domain name.

[0016] In some embodiments, constructing a vector mapping dictionary based on the occurrence frequency of whole words and / or subwords and word-granular N-Gram in the segment file includes:

[0017] The K whole words and / or subwords that appear most frequently in the segment file and M word granularities N-Gram form a vector mapping dictionary.

[0018] In some embodiments, encoding the input sequence segments of each domain name using the vector mapping dictionary to obtain the corresponding encoding vector includes:

[0019] Based on the index of the vector mapping dictionary and the input sequence fragment of each domain name, each domain name is encoded into a coding vector of a specified length, wherein if a word of the vector mapping dictionary appears in the domain name, the index of the word in the vector mapping dictionary is a preset value, and the corresponding coding vector has a corresponding indication value at a position corresponding to the preset value.

[0020] In some embodiments, the sample traffic data obtained includes normal DNS traffic samples and malicious samples generated by running the DGA algorithm;

[0021] Adding training labels to each encoding vector includes adding training labels based on encoding vectors corresponding to normal DNS traffic samples and encoding vectors corresponding to malicious samples.

[0022] In some embodiments, the DGA detection model includes 1 input layer, 1 hidden layer and 1 output layer, and the input layer includes nodes corresponding to the number of the specified lengths.

[0023] In a second aspect, an embodiment of the present invention further provides a method for detecting a DGA domain name, comprising:

[0024] Get the traffic to be tested and extract the domain name to be tested;

[0025] Based on a pre-built word segmentation dictionary, the domain name to be detected is segmented into input sequence segments of the domain name;

[0026] Inputting the input sequence fragment of the domain name into the DGA detection model trained by the method described in each embodiment of the present disclosure to perform detection;

[0027] Based on the probability output by the DGA detection model, it is determined whether the domain name is a domain name generated by the DGA algorithm.

[0028] In a third aspect, an embodiment of the present invention further provides a device for detecting a DGA domain name, comprising a processor and a memory, wherein the memory stores one or more programs, and the one or more programs can be executed by one or more processors to implement the aforementioned method for detecting a DGA domain name.

[0029] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method for detecting a DGA domain name described in each embodiment of the present disclosure.

[0030] The embodiment of the present invention segments each domain name in the first domain name file based on a pre-constructed word segmentation dictionary to obtain a segment file containing each domain name input sequence segment; constructs a vector mapping dictionary based on the appearance frequency of whole words and / or subwords and word-granular N-Gram in the segment file, and the encoding vector constructed in this way takes into account the composition structure and word order of the domain name. The domain name is classified by the generated sentence encoding vector, and low-randomness DGA families and dictionary-based DGA families can be accurately detected.

[0031] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0033] Figure 1 It is a basic flow chart of the DGA detection model training method disclosed in the present invention;

[0034] Figure 2 This is a segmentation training example of the DGA detection model disclosed in the present invention. DETAILED DESCRIPTION

[0035] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0036] The first embodiment of the present invention provides a DGA detection model training method based on subword segmentation, such as Figure 1 As shown, the following steps are included:

[0037] In step S101, sample traffic data is obtained, and the primary domain name and top-level domain name corresponding to each sample traffic data are extracted to form a first domain name file. Specifically, normal DNS sample traffic data can be collected on a local DNS server through a traffic collection device, and the primary domain name and top-level domain name corresponding to the fully qualified domain name (FQDN) of the DNS Queries Name field of each DNS traffic can be extracted and parsed, and each piece of information is saved as a record (normal sample) in a first domain name file of a specific format. For malicious samples, malicious sample traffic using the DGA algorithm can be run in a sandbox, and the primary domain name and top-level domain name corresponding to the fully qualified domain name (FQDN) of the DNS Queries Name field of each DNS traffic can be extracted and parsed, and each piece of information is saved as a record (malicious sample) in a file of a specific format.

[0038] In step S102, each domain name in the first domain name file is segmented based on a pre-constructed word segmentation dictionary to obtain a segmentation file containing input sequence segments of each domain name, wherein the input sequence segments of each domain name include whole words and / or sub-words and corresponding word granularity N-Gram. In some embodiments, the word segmentation dictionary can be constructed in the following manner: based on encyclopedia corpus, the word segmentation dictionary is constructed using a preset separation algorithm, wherein the word segmentation dictionary includes whole words and sub-words. For example, a whole word dictionary can be constructed by counting the top-N (N is 30,000) most frequently used words in the Wikipedia corpus; secondly, a sub-word dictionary is constructed through the Wikipedia corpus using algorithms such as WordPiece and BPE (Byte Pair Encoding), wherein the sub-word begins with "##", such as "##able". Finally, the whole word dictionary and the sub-word dictionary together form the word segmentation dictionary V for domain name segmentation. s . Through the word segmentation dictionary V s Each domain name in the first domain name file may be segmented, and a segment file containing a segment of an input sequence of each domain name may be obtained.

[0039] In step S103, a vector mapping dictionary is constructed based on the occurrence frequencies of whole words and / or subwords and N-Grams of word granularity in the segment file. The vector mapping dictionary constructed based on the occurrence frequencies of whole words and / or subwords and N-Grams of word granularity in the segment file takes into account the composition structure and word order of the domain name, so that low-randomness DGA families and dictionary-based DGA families can be accurately detected after training.

[0040] In step S104, the input sequence segments of each domain name are encoded using the vector mapping dictionary to obtain corresponding encoding vectors.

[0041] In step S105, training labels are added to each encoding vector to serve as training data for training the DGA detection model.

[0042] The embodiment of the present invention segments each domain name in the first domain name file based on a pre-constructed word segmentation dictionary to obtain a segment file containing each domain name input sequence segment; constructs a vector mapping dictionary based on the appearance frequency of whole words and / or subwords and word-granular N-Gram in the segment file, and the encoding vector constructed in this way takes into account the composition structure and word order of the domain name. The domain name is classified by the generated sentence encoding vector, and low-randomness DGA families and dictionary-based DGA families can be accurately detected.

[0043] In some embodiments, segmenting each domain name in the first domain name file based on a pre-built word segmentation dictionary to obtain a segment file containing each domain name input sequence segment includes:

[0044] Based on the word segmentation dictionary and in accordance with the forward maximum matching principle, each domain name in the first domain name file is segmented into sequence segments of whole words and subwords.

[0045] An N-Gram of the word granularity of the sequence segment is added to the sequence segment of each domain name to form an input sequence segment of each domain name.

[0046] Specifically, we can use the word segmentation dictionary V s , segment the domain names in the first domain name file. The domain name is segmented into whole word and subword sequence fragments through the forward maximum matching principle, and the word granularity N-Gram of the sequence fragment is added to form the input sequence fragment of the domain name. For example, after the DGA domain name "whetherchance.ru" generated by the suppobox family is segmented by the dictionary, its input sequence fragment is {whether, ##chance, ru, whether##chance, ##chanceru}. Segmentation is performed on each domain name in the first domain name file, and each generated input sequence fragment is saved in the fragment file.

[0047] In some embodiments, constructing a vector mapping dictionary based on the frequency of occurrence of whole words and / or subwords and N-Grams of word granularity in the fragment file includes: constructing a vector mapping dictionary with the K whole words and / or subwords that appear most frequently and M N-Grams of word granularity in the fragment file.

[0048] Specifically, the aforementioned segment file can be loaded, and the top-K whole words or subwords with the highest frequency in the training set and the top-M word-sized N-Grams can be counted to form the dictionary V for the model input vector mapping. m .

[0049] In some embodiments, encoding the input sequence fragments of each domain name using the vector mapping dictionary to obtain the corresponding encoding vector includes: encoding each domain name into an encoding vector of a specified length based on the index of the vector mapping dictionary and the input sequence fragments of each domain name, wherein if a word of the vector mapping dictionary appears in the domain name, then the index of the word in the vector mapping dictionary is a preset value, and the corresponding encoding vector has a corresponding indication value at a position corresponding to the preset value.

[0050] Specifically, we can use the vector mapping dictionary V m The index and input sequence fragment of each domain name encode each domain name into a dictionary with dimension length | V m | vector. If the vector mapping dictionary V m The word in appears in the domain name, and the word is in the vector mapping dictionary V mThe index in is 100, then the value of the encoding vector at position 100 is set to 1, and the values ​​of the remaining positions are 0. Finally, the vector mapping dictionary V m Save to device disk.

[0051] In some embodiments, the sample traffic data obtained includes normal DNS traffic samples and malicious samples generated by running the DGA algorithm; adding training labels to each encoding vector includes adding training labels based on the encoding vector corresponding to the normal DNS traffic sample and the encoding vector corresponding to the malicious sample. Specifically, labels can be added based on the aforementioned normal samples and malicious samples, for example, the vector label of the normal domain name is set to 0, the vector label of the DGA domain name is set to 1, and then the labeled vector data is used to train the DGA detection model.

[0052] The neural network used in the DGA detection model in this disclosure is a multi-layer network that trains weights for nonlinear differentiable functions. Its greatest feature is that it can achieve highly nonlinear mapping from input space to output space with only sample data without establishing a mathematical model of the system. Assume that the eigenvector of the model input is x∈R n , y∈{0,1}, g is the activation function, the neural network model can be simply expressed as:

[0053] h θ (x) = g(θ T x)

[0054] The neural network updates its parameters through back propagation and approximates the solution to the parameter θ by continuously iteratively minimizing the loss function of the model.

[0055] The DGA detection model can be implemented using LSTM or GRU or a deep convolutional neural network. Compared with a deep neural network, the network layer is shallower and fewer parameters need to be learned, which can effectively reduce the computational complexity. In some embodiments, the DGA detection model includes 1 input layer, 1 hidden layer and 1 output layer, and the input layer contains nodes corresponding to the number of specified lengths. Specifically, the DGA detection model of the present disclosure includes 1 input layer, 1 hidden layer and 1 output layer, respectively, and the input layer has a dictionary size |V m | neurons, the hidden layer has 20 neurons, and the output layer has 1 neuron. Figure 2 As shown in the figure, for a domain name (whetherchance.ru as an example), a vector of each sequence segment of the domain name, that is, the encoding vector of each subword, is generated. The DGA detection model sums up the word vectors output by the hidden layer as the sentence vector expression of the domain name, and then the sentence vector is classified through an output layer. The model is tuned by adjusting the hyperparameters, and finally the optimal model is persisted and saved to the device disk.

[0056] The method disclosed in the present invention regards the domain name as a collection of subwords and whole words, and considers the order of input subwords or whole word fragments. By classifying sentence vectors, the detection effect of the DGA detection model on low-random DGA families and dictionary-based DGA families is effectively improved. At the same time, the sentence vectors generated by the method disclosed in the present invention based on subwords, whole words and word order can effectively represent the characteristics of the domain name, and even if a simple classifier is used to classify the sentence vector, a good effect can be achieved.

[0057] In a second aspect, an embodiment of the present invention further provides a method for detecting a DGA domain name, comprising:

[0058] Get the traffic to be tested and extract the domain name to be tested;

[0059] Based on a pre-built word segmentation dictionary, the domain name to be detected is segmented into input sequence segments of the domain name;

[0060] Inputting the input sequence fragment of the domain name into the DGA detection model trained by the method described in each embodiment of the present disclosure to perform detection;

[0061] Based on the probability output by the DGA detection model, it is determined whether the domain name is a domain name generated by the DGA algorithm.

[0062] For example, for a domain name to be predicted, the input sequence segments of the domain name are generated after DNS traffic collection and processing and domain name segmentation. The vector mapping dictionary V is loaded from the device disk. m , encode the input sequence fragment of the domain name to be predicted into a vector, then load the trained DGA detection model from the device disk, predict the domain name, and the DGA detection model outputs the probability p that the domain name is a DGA domain name. For the manually set threshold k (default 0.5), if p>k, it can be determined that the domain name is a low-randomness domain name generated by the DGA algorithm, otherwise it is a normal domain name.

[0063] In a third aspect, an embodiment of the present invention further provides a device for detecting a DGA domain name, comprising a processor and a memory, wherein the memory stores one or more programs, and the one or more programs can be executed by one or more processors to implement the aforementioned method for detecting a DGA domain name.

[0064] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method for detecting a DGA domain name described in each embodiment of the present disclosure.

[0065] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0066] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0067] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for a terminal (which can be a mobile phone, computer, server or network equipment, etc.) to execute the methods described in each embodiment of the present invention.

[0068] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A DGA detection model training method based on subword segmentation, characterized in that: include: Obtain sample traffic data, and extract the primary domain name and top-level domain name corresponding to each piece of sample traffic data to form a first domain name file; Segmenting each domain name in the first domain name file based on a pre-built word segmentation dictionary to obtain a segment file containing each domain name input sequence segment, wherein the input sequence segment of each domain name includes a whole word and / or a subword and a corresponding N-Gram of word granularity; Building a vector mapping dictionary based on the occurrence frequency of whole words and / or subwords and word-granular N-Gram in the segment file; Encoding the input sequence segments of each domain name using the vector mapping dictionary to obtain corresponding encoding vectors; Add training labels to each encoding vector to serve as training data for training the DGA detection model; Encoding the input sequence segments of each domain name using the vector mapping dictionary to obtain the corresponding encoding vector includes: Based on the index of the vector mapping dictionary and the input sequence fragment of each domain name, each domain name is encoded into a coding vector of a specified length, wherein if a word of the vector mapping dictionary appears in the domain name, the index of the word in the vector mapping dictionary is a preset value, and the corresponding coding vector has a corresponding indication value at a position corresponding to the preset value.

2. The DGA detection model training method based on subword segmentation as claimed in claim 1, characterized in that: The word segmentation dictionary is constructed in the following way: Based on the encyclopedia corpus, the word segmentation dictionary is constructed using a preset segmentation algorithm, wherein the word segmentation dictionary includes whole words and subwords.

3. The DGA detection model training method based on subword segmentation as claimed in claim 2, characterized in that: Segmenting each domain name in the first domain name file based on a pre-built word segmentation dictionary to obtain a segment file containing each domain name input sequence segment includes: Based on the word segmentation dictionary, according to the forward maximum matching principle, each domain name in the first domain name file is divided into sequence segments of whole words and subwords; An N-Gram of the word granularity of the sequence segment is added to the sequence segment of each domain name to form an input sequence segment of each domain name.

4. The DGA detection model training method based on subword segmentation according to claim 1, characterized in that: Building a vector mapping dictionary based on the frequency of occurrence of whole words and / or subwords and N-Gram of word granularity in the segment file includes: The K whole words and / or subwords that appear most frequently in the segment file and N-Grams of M word granularities form a vector mapping dictionary.

5. The DGA detection model training method based on subword segmentation according to claim 1, characterized in that: The sample traffic data obtained includes normal DNS traffic samples and malicious samples generated by running the DGA algorithm; Adding training labels to each encoding vector includes adding training labels based on encoding vectors corresponding to normal DNS traffic samples and encoding vectors corresponding to malicious samples.

6. The DGA detection model training method based on subword segmentation according to claim 1, characterized in that: The DGA detection model includes 1 input layer, 1 hidden layer and 1 output layer, and the input layer contains nodes corresponding to the number of the specified lengths.

7. A method for detecting a DGA domain name, characterized in that: include: Get the traffic to be tested and extract the domain name to be tested; Based on a pre-built word segmentation dictionary, the domain name to be detected is segmented into input sequence segments of the domain name; Inputting the input sequence fragment of the domain name into the DGA detection model trained by the method according to any one of claims 1 to 6 to perform detection; Based on the probability output by the DGA detection model, it is determined whether the domain name is a domain name generated by the DGA algorithm.

8. A device for detecting a DGA domain name, characterized in that: The system comprises a processor and a memory, wherein the memory stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method for detecting a DGA domain name according to claim 7.

9. A computer-readable storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method for detecting a DGA domain name according to claim 7.

Images