A malicious URL detection system and method based on automatic feature extraction

[0041] Hereinafter, a number of preferred embodiments of the present invention will be introduced with reference to the accompanying drawings in the specification to make the technical content clearer and easier to understand. The present invention can be embodied by many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0042] figure 1 It shows a schematic structural diagram of a malicious URL detection system based on automatic feature extraction in an embodiment of the present invention. This embodiment provides a malicious URL detection system based on automatic feature extraction. The system is composed of a preprocessing module, a parallel learning module, and a detection classification module. For the input URL, the system will determine whether it is a malicious URL and give Its category. In the present invention, the preprocessing module converts different types of data sources such as character strings, webpage text, and webpage images into three digital matrices that carry URL structural features, text features, and image features. In view of the different characteristics of these three digital matrices, in the parallel learning module of the present invention, three different deep learning networks of n-gram convolutional network, TextCNN, and image convolutional network are used to learn the features respectively. In the detection classification module of the present invention, the learning results of the three parts in the parallel learning module are comprehensively used to obtain the final detection result, which is returned to the client. This method of completely automatically extracting features by a computer and integrating the three features to draw conclusions is one of the core innovations of the present invention. The processing and learning process of URL structural features, webpage text features and webpage image features in the present invention are as follows:

[0043] URL structural features: The traditional URL structural feature extraction relies on human experience, and the present invention is inspired by the processing method of word2vec converting text into word vectors and calculating its association, and discards the way of manually extracting URL structural features. figure 2 The conversion process of the character string of the malicious URL detection system based on automatic feature extraction to the multi-dimensional vector in the embodiment of the present invention is shown: a character in the URL string corresponds to a multi-dimensional vector, so that a URL string is converted into Digital matrix. Similar characters are closer in the multidimensional space, and vice versa. In the embodiment of the present invention, the experimental results show that symbols are considered by the system as similar characters, lowercase letters are considered as similar characters, and uppercase letters are also considered as similar characters. The next step after the character string is transformed into a multidimensional vector is to use neural networks to learn features. image 3 Shows the convolution process of a fan-shaped window on a multi-dimensional vector. In the embodiment of the present invention, convolution windows of sizes 3, 4, and 5 are used to convolve character vectors respectively. The convolutional network first automatically summarizes the pattern features from a large number of URL character matrix inputs that have been marked. Then when there is a new URL input, the neural network can perform pattern matching on it through convolution. The pattern matching here can be understood by the following example. If the neural network finds a capital letter followed by a number or a control character, it will automatically compare it with the pattern feature set to see if it matches an existing pattern. The result of pattern matching is the learning result of URL structural features.

[0044] Webpage text features: In the traditional sense, convolutional neural networks are used for image processing and also show good performance. Intuitively, the top-down scanning characteristics of the convolutional neural network from left to right are indeed very similar to the way we process images. However, this does not mean that it cannot be used for text processing. The basic algorithm of the text convolutional neural network (TextCNN) is the same as the above-mentioned convolutional neural network. The difference is that in natural language processing, we need to select a feature extraction window with the same width as the input matrix, and the height of the window is optional , Its typical value is 2-5. In actual operation, we have selected three windows with widths of 3, 4, and 5, and the number of each window is set to 128, so that more comprehensive features can be extracted, which helps to improve the accuracy of the final result. In general, our extraction of text features can be divided into two parts: word2vec word vector conversion part and TextCNN word vector processing part. After entering a web page body segment, word2vec converts each word in the text into a word vector, so that for the entire text, we get a digitized matrix. Taking this digitized matrix as the input of TextCNN, we can get a probability matrix about the text, which contains the classification features of the text. The implementation framework of the entire text extraction process can be determined by Figure 4 Said.

[0045] Webpage image features: The image feature data source of this project is the webpage image information of the webpage corresponding to the malicious URL. After preprocessing such as clipping and filtering, the webpage is adapted to the input requirements of the deep image convolutional neural network. Then use deep image convolutional neural network to learn image features.

[0046] In order to make full use of the information provided by the URL, reduce artificial errors, and make the selected three features (URL structure feature, web page text feature, web page image feature) more closely related, add after the single-layer training model The first layer is a Softmax model that fully connects the three models. In this way, the information association between the three is maximized, and the utilization of various information is maximized. At the same time, due to less manual intervention, the error of feature extraction can be further reduced. The learning result of the fully connected layer is the final system's judgment result of the URL. We divide URLs into 7 categories, normal URLs into one category, and malicious URLs into 6 categories of systems. Finally, the system will give a classification report for the entered URL, the specific classification such as Figure 5 Shown.

[0047] The preferred embodiments of the present invention are described in detail above. It should be understood that ordinary technologies in the field can make many modifications and changes according to the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by those skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should fall within the protection scope determined by the claims.