Rule and deep learning dual-drive-based malicious URL detection method

Abstract

The invention discloses a malicious URL (Uniform Resource Locator) detection method based on rule and deep learning dual drive, which comprises the following steps: acquiring a to-be-detected URL and preprocessing, converting the URL into a preset form and eliminating a placeholder in the URL to obtain a preprocessed URL; according to the rule and deep learning dual-drive-based malicious URL detection method, firstly, a URL is decoded through preprocessing, placeholders are eliminated to improve the definition of a URL structure, features of the URL are extracted from words, characters and feature tags, the extracted features are input into a URL detector to be detected, and the detection accuracy is improved; a mode of combining a convolution module and a Transform module is adopted in a URL detector, and local and global information can be utilized at the same time, so that the accuracy of classification detection is improved; according to the rule and deep learning dual-drive-based malicious URL detection method, URL samples in a training data set are balanced in a training process, so that the balance between benign URL samples and malicious URL samples is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of malicious URL detection, and specifically relates to a malicious URL detection method based on dual drive of rules and deep learning. Background Art

[0002] URL is a uniform resource locator, which is a concise representation of the location and access method of resources on the Internet. Among them, malicious URL refers to a URL carefully constructed by an attacker for malicious purposes. Attackers use actively constructed malicious URLs to illegally access the target website, such as SQL injection, cross-site scripting attacks, and cross-site request forgery attacks; or passively deceive victims into clicking on malicious URLs, such as phishing links, to steal user information and install and execute backdoor Trojan programs. Therefore, the detection of malicious URLs has become an important research direction in the field of network security.

[0003] Traditional malicious URL detection methods are mainly based on manually designed features and rules, which require continuous updates and maintenance by experts. They face delays and limitations in identifying new threats because they rely on known URL structures and manual updates, making it difficult to deal with new malicious URLs, resulting in limited detection results. In recent years, with the continuous development of deep learning technology, malicious URL detection methods based on automatic feature extraction have gradually become a research hotspot. Deep learning models commonly used for malicious URL detection include CNN (convolutional neural network), RNN (recurrent neural network), Transformer, etc. Although deep learning methods have made some progress in detecting malicious URLs, there are still some limitations: (1) The performance of deep learning models depends largely on the scale and quality of training data. Data sets of malicious URLs are often limited in size and the labeling work is cumbersome, which limits the training effect and generalization ability of the model. (2) In malicious URL detection, data samples are often unbalanced, with malicious samples far less than benign samples, which may cause the model to tend to predict the majority class, thereby affecting the accuracy of detection. Summary of the invention

[0004] The purpose of the present invention is to solve the problems raised in the background technology and propose a malicious URL detection method based on dual drive of rules and deep learning.

[0005] To achieve the above object, the technical solution adopted by the present invention is: The present invention proposes a malicious URL detection method based on dual-drive of rules and deep learning, including: Obtain the URL to be detected and perform preprocessing, so that the URL is converted into a preset format and the placeholder in the URL is eliminated to obtain the preprocessed URL; The preprocessed URL is represented at word level, character level and feature tag level respectively; The represented words, characters and feature labels are input into the first model to obtain three different word vector matrices respectively, and the three word vector matrices are fused to obtain a new word vector matrix; The new word vector matrix is ​​input into the trained URL detector, and the output of the URL detector is passed through the softmax layer to obtain the detection result of whether the URL to be detected is malicious; The URL detector includes a convolution module and a Transformer module. The new word vector matrix is ​​input into the convolution module and the Transformer module respectively to obtain the output of each module. The two outputs are fused to obtain the output of the URL detector.

[0006] Preferably, the URL to be detected is preprocessed so that the URL is converted into a preset format and the placeholder in the URL is eliminated to obtain the preprocessed URL, including: First decode the URL to be detected: Using a preset first regular expression to identify whether a corresponding type of encoding exists in the URL to be detected; After the encoding is recognized, the corresponding type of decoding function is used to decode the URL, and the URL is converted into a preset format; Then eliminate the placeholders from the decoded URL: The decoded URL is identified by using a preset second regular expression, and the identified placeholders are replaced with spaces; For multiple consecutive spaces, only one is retained to obtain the preprocessed URL.

[0007] Preferably, the encoding types include at least four types: base64 encoding, unicode encoding, url encoding and hex encoding, and the first regular expressions corresponding to the four types are in the form of [A-Za-z0-9+ / =]+, \\u[0-9A-Fa-f], %[0-9a-fA-F] and [0-9A-Fa-f] respectively; The second regular expression is of the form .

[0008] Preferably, the URL includes at least four components: protocol, host name resolved by domain name, file path and parameters.

[0009] Preferably, the preprocessed URL is represented at the word level, the character level and the feature tag level, respectively, including: Word-level representation: Split the preprocessed URL into words according to the spaces in it; Character-level representation: Each word in the word-level representation is divided into individual letters as characters; Feature tag level representation: first, a rule base is pre-set, and the rule base contains preset malicious keywords and corresponding prefixes set according to the malicious types of the malicious keywords. The malicious keywords belong to the host name or parameters of the domain name resolution. The preprocessed URL is matched with the malicious keywords in the rule base, and the successfully matched malicious keywords and prefixes are extracted as feature tags for the preprocessed URL.

[0010] Preferably, the first model is a Word2Vec model, and a new word vector matrix is ​​obtained by weighted summing up three different word vector matrices output by the Word2Vec model.

[0011] Preferably, in the convolution module, the new word vector matrix passes through a preset number of convolution layers in parallel, the output of each convolution layer passes through a pooling layer to obtain a corresponding preset number of first feature vectors, the preset number of first feature vectors are concatenated to obtain a second feature vector, and the second feature vector is passed through a linear layer to obtain the output of the convolution module.

[0012] Preferably, in the Transformer module, each row vector in the new word vector matrix is ​​position-encoded to obtain a position-encoded vector, and then each position-encoded vector is added to the corresponding row vector in the new word vector matrix to obtain a third eigenvector, and then the third eigenvector is sequentially input into a preset number of Transformer encoding layers to obtain a fourth eigenvector, and the Transformer encoding layer includes a self-attention mechanism layer and a feedforward neural network layer connected in sequence, and then the fourth eigenvector is passed through a linear layer to obtain the output of the Transformer module.

[0013] Preferably, the output of the convolution module and the output of the Transformer module are concatenated through a fully connected layer to obtain the output of the URL detector, and finally the output of the URL detector is input into the softmax layer to obtain a detection result of whether the URL to be detected is malicious.

[0014] Preferably, the malicious URL detection method based on dual-drive of rules and deep learning also includes balancing the URL samples in the training data set before training: The training data set includes benign URL samples and malicious URL samples, and the number of benign URL samples is greater than the number of malicious URL samples; For benign URL samples, measure the length difference between any two benign URL samples , and then based on the length of the longest benign URL sample among the two benign URL samples Calculate the similarity between the two benign URL samples with the gap , and the calculation formula is as follows: ; When the similarity is greater than a preset threshold, one sample is discarded from the two benign URL samples; For malicious URL samples, characters are randomly inserted, deleted, and replaced to generate new malicious URL samples.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This malicious URL detection method based on rule and deep learning dual drive first decodes the URL and eliminates placeholders through preprocessing to improve the clarity of the URL structure, extracts the features of the URL from words, characters and feature tags, and inputs the extracted features into the URL detector for detection, thereby improving the accuracy of detection; the combination of convolutional modules and Transformer modules in the URL detector can simultaneously utilize local and global information, thereby improving the accuracy of classification detection; 2. This malicious URL detection method based on dual-drive of rules and deep learning balances the URL samples in the training data set during the training process, so as to achieve a balance between benign URL samples and malicious URL samples, thus solving the problem that data samples in the prior art are often unbalanced, with malicious samples far less than benign samples, which affects the accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of a malicious URL detection method based on rule-based and deep learning dual-driven methods of the present invention; Figure 2 This is a module block diagram of the malicious URL detection method based on dual drive of rules and deep learning in the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] like Figure 1-Figure 2 As shown, a malicious URL detection method based on rule and deep learning dual drive is provided, including: Step 1: Obtain the URL to be detected and perform preprocessing, so that the URL is converted into a preset format and the placeholders in the URL are eliminated to obtain the preprocessed URL.

[0019] It should be noted that URL is the Uniform Resource Locator of WWW, that is, the network address, which is used to locate and access specific resources on the Internet. Through URL, users can accurately locate and access various resources on the Internet, such as web pages, pictures, videos, files, etc. For example, the URL to be tested is: http: / / 192.168.52.134 / 06 / vul / sqli / sqli_str.php?name=test%27+union+select+1%2Cdatabase%28%29%23&submit=%E6%9F%A5%E8%AF%A2.

[0020] Preprocess the URL to be detected, convert the URL into a preset form and eliminate the placeholders in the URL to obtain the preprocessed URL (the attacker uses various encoding methods to encode the URL form that is easy for security analysts to understand into a difficult-to-understand form. At the same time, because each encoding method has its own form, we first identify its possible encoding method, and then call the corresponding decoding function of this possible encoding method to complete the URL decoding). The specific process is as follows: First decode the URL to be detected: Using a preset first regular expression to identify whether a corresponding type of encoding exists for the URL to be detected (the encoding types include at least four types: base64 encoding, unicode encoding, url encoding, and hex encoding. The first regular expressions corresponding to the four types are [A-Za-z0-9+ / =]+, \\u[0-9A-Fa-f], %[0-9a-fA-F], and [0-9A-Fa-f], respectively); After the encoding is identified, the corresponding type of decoding function is used to decode the URL, and the URL is converted into a preset format (the preset format is a UTF-8 format that is easy to read. When the corresponding type of encoding is identified through each first regular expression, the corresponding type of encoding is returned, and then various decoding functions that come with Python are called according to the returned results to decode it into UTF-8 format); Then eliminate the placeholders from the decoded URL: The decoded URL is identified by the preset second regular expression, and the identified placeholders are replaced with spaces (the second regular expression is in the form of ); For multiple consecutive spaces, only one is retained to obtain the preprocessed URL.

[0021] Step 2: Represent the preprocessed URL at the word level, character level, and feature tag level.

[0022] First of all, it should be noted that a URL consists of at least four components: protocol, host name (or IP address) resolved by domain name, file path, and parameters; The preprocessed URL is represented at the word level, character level and feature tag level respectively. The specific process is as follows: Word-level representation: Split the preprocessed URL into words according to the spaces in it; Character-level representation: Each word in the word-level representation is divided into individual letters as characters; Feature tag level representation: First, a rule base is pre-set, and the rule base contains preset malicious keywords and corresponding prefixes set according to the malicious types of malicious keywords (wherein the rule base is composed of keywords in existing malicious attack methods, and corresponding prefixes are set according to the malicious types of malicious keywords, such as SQL injection, XSS attack, and phishing attack, and the prefixes corresponding to the three attack types are set as sql, xss, and domain respectively), and the malicious keywords belong to the host name or parameters of the domain name resolution (because the malicious URL is divided into two types, one is that the attacker constructs malicious parameters to the normal host, so that it returns abnormal results to achieve the purpose of destruction; the other is that the attacker builds a malicious host by himself and launches attacks such as phishing attacks; therefore, whether a URL is malicious is usually determined based on the following two aspects, such as what is the host name of the domain name resolution of this URL, and what are the parameters it carries, so the malicious keywords in the rule base are the host name or parameters of the domain name resolution), the preprocessed URL is matched with the malicious keywords in the rule base, and the successfully matched malicious keywords and prefixes are extracted as the feature tags of the preprocessed URL, where the representation of the feature tag is "prefix_malicious keyword", such as the matched malicious keywords are "select", " <script>”、"example.com”，且该三个恶意关键词在规则库中的相应前缀依次为"sql”、"xss”、"domain”，则对应的特征标签依次为"sql_select”、"xss_<script>”、"domain _example.com”。

[0023] 步骤3、将表示后的单词、字符和特征标签均输入至第一模型，分别得到三个不同的词向量矩阵，并将该三个词向量矩阵进行融合得到新词向量矩阵；第一模型为Word2Vec模型，将表示后的单词、字符和特征标签均输入至Word2Vec模型得到三个不同的词向量矩阵（各词向量矩阵的尺寸表示形式为LW，L表示矩阵长度，W表示矩阵维度），且将Word2Vec模型输出的三个不同的词向量矩阵进行加权求和得到新词向量矩阵（其中该加权求和中的权重为训练好的URL检测器对应的权重参数，在训练URL检测器过程中，首先随时初始化该权重的值，然后随着URL检测器的训练不断更新，直到URL检测器训练结束）。

[0024] 步骤4、将新词向量矩阵输入至训练好的URL检测器中，将URL检测器的输出经过softmax层得到待检测的URL是否为恶意的检测结果；其中，URL检测器包括卷积模块和Transformer模块，将新词向量矩阵分别输入至卷积模块和Transformer模块得到各自模块的输出，将得到的两个输出进行融合得到URL检测器的输出。

[0025] 在卷积模块中，新词向量矩阵并行经过预设数量的卷积层（如本实施例中采用三个卷积层，且其卷积核大小尺寸不同，如分别为2W、3W、4W），各卷积层的输出经过池化层（因此本实施例中池化层的数量也为三个，各池化层对输入的特征向量进行最大值选取得到第一特征向量），得到对应预设数量的第一特征向量（即本实施例中得到三个第一特征向量），将预设数量的第一特征向量进行拼接得到第二特征向量，再将第二特征向量经过线性层得到卷积模块的输出。

[0026] 在Transformer模块中，对新词向量矩阵中的每个行向量进行位置编码得到位置编码向量，再将各位置编码向量与新词向量矩阵中对应的行向量进行相加得到第三特征向量，再将第三特征向量依次输入至预设数量的Transformer编码层中得到第四特征向量（如本实施例中Transformer编码层的数量为三个），且Transformer编码层包括由数据输入至输出方向依次连接的自注意力机制层和前馈神经网络层，再将第四特征向量经过线性层得到Transformer模块的输出。

[0027] 其中位置编码通过使用正弦和余弦函数的不同频率来构建，以确保每个位置的编码都是唯一的，具体公式如下：；；其中，表示位置编号，表示偶数维度，表示奇数维度。

[0028] 在URL检测器中，通过卷积模块使用多个大小不同的卷积核在URL连续文本序列中捕获多个连续单词或字符之间的特征，因此卷积模块用来提取URL中局部的关键特征，而Transformer模块通过注意力机制层使得模型能够在序列数据中捕获长距离依赖关系，结合这两者，可以同时利用局部和全局信息，从而提高分类的准确性。

[0029] 将卷积模块的输出和Transformer模块的输出经过一个全连接层实现拼接，得到URL检测器的输出，最后将URL检测器的输出输入至softmax层，得到待检测的URL是否为恶意的检测结果（softmax层将输入的特征向量中每个元素转换为概率，将概率最高的类别作为最终的检测结果，且类别为恶意URL或良性URL）。

[0030] 本基于规则与深度学习双驱动的恶意URL检测方法还包括在训练之前，对训练数据集中的URL样本进行平衡处理（平衡处理之后，将平衡之后的训练数据集对本基于规则与深度学习双驱动的恶意URL检测方法按照步骤1-步骤4进行训练）：训练数据集包括良性URL样本和恶意URL样本，且良性URL样本数量大于恶意URL样本数量；对于良性URL样本，衡量任意两个良性URL样本之间的长度差距（本实施例中采用莱文斯坦距离（Levenshtein距离）来计算衡量），然后根据该两个良性URL样本中长度最大的良性URL样本长度与差距计算该两个良性URL样本之间的相似性，且计算公式如下：；且当相似性大于预设的阈值时，则从该两个良性URL样本中丢弃一个样本，从而实现对训练数据集中良性URL样本的欠采样；对于恶意URL样本，随机地对恶意URL样本进行插入、删除和替换字符（即单词中的字母）操作来生成新的恶意URL样本，从而实现对训练数据集中恶意URL样本的过采样；通过过采样与欠采样实现训练数据集中良性URL样本与恶意URL样本之间的平衡。

[0031] 本基于规则与深度学习双驱动的恶意URL检测方法首先通过预处理对URL进行解码并消除占位符来提升URL结构的清晰度，并从单词、字符和特征标签来提取URL的特征，并将提取的特征输入至URL检测器中进行检测，提高了检测的准确性；在URL检测器中采用卷积模块和Transformer模块相结合的方式，可以同时利用局部和全局信息，从而提高分类检测的准确性；本基于规则与深度学习双驱动的恶意URL检测方法在训练过程中对训练数据集中的URL样本进行平衡，使得良性URL样本与恶意URL样本之间的平衡，解决了现有技术中数据样本往往不均衡，恶意样本远少于良性样本，而导致影响检测的准确性问题。

[0032] 以上所述实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利的保护范围应以所附权利要求为准。< / script>

Claims

1. A malicious URL detection method based on rule and deep learning dual drive, characterized by: The malicious URL detection method based on dual drive of rules and deep learning includes: Obtain the URL to be detected and perform preprocessing, so that the URL is converted into a preset format and the placeholder in the URL is eliminated to obtain the preprocessed URL; The preprocessed URL is represented at word level, character level and feature tag level respectively; The represented words, characters and feature labels are input into the first model to obtain three different word vector matrices respectively, and the three word vector matrices are fused to obtain a new word vector matrix; The new word vector matrix is ​​input into the trained URL detector, and the output of the URL detector is passed through the softmax layer to obtain the detection result of whether the URL to be detected is malicious; The URL detector includes a convolution module and a Transformer module. The new word vector matrix is ​​input into the convolution module and the Transformer module respectively to obtain the output of each module. The two outputs are fused to obtain the output of the URL detector.

2. The malicious URL detection method based on rule and deep learning dual drive as claimed in claim 1 is characterized by: Preprocessing the URL to be detected so that the URL is converted into a preset format and the placeholder in the URL is eliminated to obtain the preprocessed URL, including: First decode the URL to be detected: Using a preset first regular expression to identify whether a corresponding type of encoding exists in the URL to be detected; After the encoding is recognized, the corresponding type of decoding function is used to decode the URL, and the URL is converted into a preset format; Then eliminate the placeholders from the decoded URL: The decoded URL is identified by using a preset second regular expression, and the identified placeholders are replaced with spaces; For multiple consecutive spaces, only one is retained to obtain the preprocessed URL.

3. The malicious URL detection method based on rule and deep learning dual drive as claimed in claim 2 is characterized by: The encoding types include at least base64 encoding, unicode encoding, url encoding and hex encoding, and the first regular expressions corresponding to the four types are [A-Za-z0-9+ / =]+, \\u[0-9A-Fa-f], %[0-9a-fA-F] and [0-9A-Fa-f] respectively; The second regular expression is of the form .

4. The malicious URL detection method based on rule and deep learning dual drive as claimed in claim 2 is characterized by: The URL includes at least four components: protocol, host name resolved by domain name, file path and parameters.

5. The malicious URL detection method based on rule and deep learning dual drive as claimed in claim 4 is characterized by: The preprocessed URL is represented at the word level, the character level and the feature tag level, respectively, including: Word-level representation: Split the preprocessed URL into words according to the spaces in it; Character-level representation: Each word in the word-level representation is divided into individual letters as characters; Feature tag level representation: first, a rule base is pre-set, and the rule base contains preset malicious keywords and corresponding prefixes set according to the malicious types of the malicious keywords. The malicious keywords belong to the host name or parameters of the domain name resolution. The preprocessed URL is matched with the malicious keywords in the rule base, and the successfully matched malicious keywords and prefixes are extracted as feature tags for the preprocessed URL.

6. The malicious URL detection method based on rule and deep learning dual drive as claimed in claim 1, characterized in that: The first model is a Word2Vec model, and a new word vector matrix is ​​obtained by weighted summing up three different word vector matrices output by the Word2Vec model.

7. The malicious URL detection method based on rule and deep learning dual drive as claimed in claim 1 is characterized by: In the convolution module, the new word vector matrix passes through a preset number of convolution layers in parallel, and the output of each convolution layer passes through a pooling layer to obtain a corresponding preset number of first feature vectors. The preset number of first feature vectors are concatenated to obtain a second feature vector, and then the second feature vector is passed through a linear layer to obtain the output of the convolution module.

8. The malicious URL detection method based on rule and deep learning dual drive as claimed in claim 1 is characterized by: In the Transformer module, each row vector in the new word vector matrix is ​​position-encoded to obtain a position-encoded vector, and each position-encoded vector is added to the corresponding row vector in the new word vector matrix to obtain a third eigenvector, and the third eigenvector is sequentially input into a preset number of Transformer encoding layers to obtain a fourth eigenvector, and the Transformer encoding layer includes a self-attention mechanism layer and a feedforward neural network layer connected in sequence, and the fourth eigenvector is passed through a linear layer to obtain the output of the Transformer module.

9. The malicious URL detection method based on rule and deep learning dual drive as claimed in claim 1, characterized in that: The output of the convolution module and the output of the Transformer module are concatenated through a fully connected layer to obtain the output of the URL detector. Finally, the output of the URL detector is input into the softmax layer to obtain the detection result of whether the URL to be detected is malicious.

10. The malicious URL detection method based on rule and deep learning dual drive as claimed in claim 1, characterized in that: This malicious URL detection method based on dual-drive of rules and deep learning also includes balancing the URL samples in the training data set before training: The training data set includes benign URL samples and malicious URL samples, and the number of benign URL samples is greater than the number of malicious URL samples; For benign URL samples, measure the length difference between any two benign URL samples , and then based on the length of the longest benign URL sample among the two benign URL samples Calculate the similarity between the two benign URL samples with the gap , and the calculation formula is as follows: ； When the similarity is greater than a preset threshold, one sample is discarded from the two benign URL samples; For malicious URL samples, characters are randomly inserted, deleted, and replaced to generate new malicious URL samples.

Images