A malicious URL detection method based on sparse autoencoder

Abstract

The application discloses a kind of based on sparse auto-encoder's malicious URL detection method, this method with sparse auto-encoder as the core model to complete the detection of malicious URL, including: collecting normal URL to construct data set, as training data input;URL is preprocessed and parameter mapping;URL is converted respectively by TF-IDF and Doc2Vec method, two vectors obtained are connected as the complete feature vector of URL;A sparse auto-encoder is constructed, and the feature vector is used as input to train;The URL to be tested is input into the trained sparse auto-encoder, the spatial distance of its input and the output obtained is calculated, and whether it is a malicious URL is judged in combination with a given threshold value.The method effectively reduces the data cost required in the process of constructing the detection model using unsupervised algorithm, and also has the advantages of accurate feature extraction and efficient model training, and achieves an average accuracy of 95% in detecting new URL samples, achieving simultaneous improvement in efficiency and accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of network and information security, and particularly relates to a malicious URL detection method based on a sparse autoencoder BACKGROUND

[0002] With the development of the Internet and web technology, the function and scope of application are continuously expanding, while there are also a large number of security problems, illegal users take various means to send disguised requests to web servers, to achieve the purpose of stealing information, detecting vulnerabilities, etc., which may cause huge losses to individuals, enterprises and government-related units. Since a large number of malicious access and attacks can be discovered through analysis and detection of URLs, the study of malicious URLs is an important topic in the field of network security.

[0003] Various methods are constantly being proposed with the development of technology, and the current detection of malicious URLs mainly includes two methods based on template matching and machine learning. The method based on template matching is the most commonly used method for malicious URL detection in practical engineering practice, which can complete the identification and filtering of malicious URLs by maintaining a blacklist or rule library, and is simple, efficient and accurate in application, but this method is a method that relies heavily on existing knowledge, and the biggest problem is that it has a lag in detection capability, and cannot effectively detect new malicious types, so it is rarely considered and used in new method exploration. Another type of detection method based on machine learning is the current main research direction, various methods including deep learning are being tried in this field, but one problem is that most methods need to use supervised algorithms to complete the detection of malicious URL samples, so these methods also have high requirements for training data, and the quality of the data set will directly affect the final detection capability. In the process of constructing the training data set, there is a natural imbalance in the number of normal URL samples and malicious URL samples, which also makes the construction of malicious URL samples require a high cost.

[0004] In view of the above problems, it is necessary to provide an unsupervised malicious URL detection method, which can ensure high accuracy of the detection result while minimizing the data cost of training the detection model. SUMMARY

[0005] In view of the above deficiencies in the prior art, the present application provides a malicious URL detection method based on a sparse autoencoder.

[0006] The malicious URL detection method based on a sparse autoencoder provided by the present application comprises the following steps:

[0007] S1. Collect comprehensive normal URLs as samples through web applications to build a training dataset;

[0008] S2. Preprocess the URL and map some parameter values ​​in the URL according to requirements;

[0009] S3. Extract features from the URL, convert them into computable feature vectors, and normalize the feature vectors.

[0010] S4. Construct a sparse autoencoder (SAE) and train the sparse autoencoder.

[0011] S5. Test the detection effect of the model: Convert a URL into a corresponding feature vector, input it into the trained sparse autoencoder, and obtain the reconstructed output vector. Determine whether it is a malicious URL by calculating the spatial distance between the input and output.

[0012] Furthermore, the URL preprocessing in step S2 includes several operations such as escaping, lowercase character conversion, and null character replacement. In addition, it is necessary to map some parameter values ​​in the URL according to actual needs. The mapping method is as follows:

[0013] For numeric characters in the parameter value, map them to the character '0';

[0014] For English characters in the parameter value, map them to the character 'a';

[0015] For Chinese characters in the parameter value, map them to the character 'b';

[0016] For other symbol characters in the parameter value, they are mapped to the character '#'.

[0017] Furthermore, step S3 includes the following sub-steps:

[0018] S3.1. Process the URL using the TF-IDF method to convert the corresponding URL into vector v1;

[0019] S3.2. Process the URL using the Doc2Vec method to convert the corresponding URL into a vector v2;

[0020] S3.3 Connect vectors v1 and v2 to obtain the feature vector v used to represent the URL;

[0021] S3.4, through formula Normalize the feature vector v to obtain the feature vector v′, where max(v) represents the maximum value in the vector v and min(v) represents the minimum value in the vector v.

[0022] Furthermore, step S3.1 includes the following sub-steps:

[0023] S3.1.1, adopt bigram method to segment URL string, get corresponding URL word array;

[0024] S3.1.2, calculate the frequency of word w in URL word array d through formula

[0025] S3.1.3, calculate the inverse document frequency of word w through formula , wherein N represents the number of URL samples in the training data set, and N(w) represents the number of URL samples containing word w.

[0026] S3.1.4, calculate the TF-IDF value of word w through formula TF-IDF(w) = TF(d, w) x IDF(w);

[0027] S3.1.5, repeat steps S3.1.2 to S3.1.4 to complete the calculation of all words, and obtain the vector representation of the URL based on the TF-IDF values of the respective words.

[0028] Further, the step S3.2 includes the following sub-steps:

[0029] S3.2.1, divide the URL into single characters, and convert the continuous URL string into a single character discontinuous combination, taking space as a separator;

[0030] S3.2.2, adopt Doc2Vec method to train PV-DM model, in the training process, first map each character and sentence to vector space, and then combine sentence vector and its context 2k character vectors when processing each character vector, and optimize the model through maximum formula , to obtain good character vector expression after training convergence, wherein w t represents the vector corresponding to character t, and p(w t |w t-k ,...,w t+k ) is a probability function.

[0031] S3.2.3, calculate the mean of the sum of each character vector in the URL as the vector representation of the URL.

[0032] Further, the sparse autoencoder in step S4 is composed of three layers of network, including input layer, hidden layer and output layer, the mapping from input layer to hidden layer is the encoding process, the mapping from hidden layer to output layer is the decoding process, and the activation function of the network is Sigmod function.

[0033] ​In the training process of the sparse autoencoder, the feature vector of the URL is taken as the original data into the input layer, and the output of the hidden layer is obtained through the encoder, and the output of the hidden layer is further taken as the input of the decoder, and finally the reconstructed data of the output is obtained, the error between the original data and the output data is calculated through the loss function, and the loss function is minimized as the optimization objective, the back propagation is carried out, and the parameter values of the network are updated.

[0034] In the training process, the loss function is defined as Wherein, the KL divergence function is Wherein The average activation of the hidden layer node j is described, m represents the number of samples, s represents the number of hidden layer nodes, and β is a parameter for controlling the sparsity weight. With the loss function minimization as the optimization objective, the parameters in the network are updated in iterations through the optimization algorithm, and the trained network is obtained when the loss function converges or the training round ends.

[0035] Further, in step S5, the trained model is used for detecting the URL, the converted feature vector of the URL is input into the sparse autoencoder, the reconstructed output is obtained, and finally the spatial distance between the input and the output is calculated through the distance function If greater than a given threshold, the URL is considered as a malicious URL, otherwise it is considered as a normal URL.

[0036] In summary, the present application has the following beneficial effects:

[0037] The malicious URL detection method based on the sparse autoencoder is key to the sparse autoencoder which can sample and reconstruct the input data in an unsupervised manner, and only normal URL samples need to be provided in the model training process, without the need for marking operation on malicious URL data, so that the cost of data set construction and model training is greatly reduced, and the average accuracy of detection using the method reaches 95%, achieving a good detection level of high accuracy, low false positive rate and low false negative rate. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The flowchart of the malicious URL detection method proposed by the present application is shown in the figure;

[0039] Figure 2 The schematic diagram of the parameter mapping of the URL in the present application is shown in the figure;

[0040] Figure 3 The schematic diagram of the PV-DM model of Doc2Vec in the present application is shown in the figure;

[0041] Figure 4 The structure schematic diagram of the detection model established by the present application is shown in the figure. DETAILED DESCRIPTION

[0042] In order to make the objects, technical solutions and advantages of the present application clearer, further descriptions will be made to the embodiments of the present application with reference to the drawings.

[0043] As shown in the drawings, the present application provides a malicious URL detection method based on sparse auto-encoder, which comprises the following steps: Figure 1

[0044] S1, collecting comprehensive normal URLs as samples through web application to construct training data set, and in this example, the public CSIC2010 data set is used for implementation, and only the normal URL samples are used to train the detection model;

[0045] S2, preprocessing the URL, and mapping part of the parameter values in the URL according to the requirements;

[0046] S3, extracting features of the URL, converting it into a calculable feature vector, and normalizing the feature vector;

[0047] S4, constructing sparse auto-encoder and training the sparse auto-encoder;

[0048] S5, testing the model detection effect: converting a URL into a corresponding feature vector, inputting the trained sparse auto-encoder, obtaining the reconstructed output vector, and judging whether it is a malicious URL by calculating the spatial distance between the input and the output.

[0049] The URL preprocessing in step S2 includes escaping, character lowercase conversion and null character replacement operations, and in addition, part of the parameter values in the URL need to be mapped according to actual requirements, Figure 2 The sample is mapped in the following way:

[0050] For the numerical characters in the parameter value, it is mapped to the character '0';

[0051] For the English characters in the parameter value, it is mapped to the character 'a';

[0052] For the Chinese characters in the parameter value, it is mapped to the character 'b';

[0053] For other symbol characters in the parameter value, it is mapped to the character '#'.

[0054] Step S3 comprises the following steps:

[0055] S3.1, processing the URL by TF-IDF method, converting the corresponding URL into vector v1;

[0056] ​S3.2, processing the URL by the Doc2Vec method, converting the corresponding URL into a vector v2;

[0057] S3.3, connecting the vectors v1 and v2 to obtain a feature vector v for expressing the URL;

[0058] S3.4, performing a normalization operation on the feature vector v to obtain a feature vector v', wherein max(v) represents the maximum value in the vector v, and min(v) represents the minimum value in the vector v.

[0059] Step S3.1 includes the following sub-steps:

[0060] S3.1.1, adopting the bigram method to segment the URL string to obtain an array of corresponding URL words;

[0061] S3.1.2, calculating the frequency of the word w in the URL word array d by the formula

[0062] S3.1.3, calculating the inverse document frequency of the word w by the formula , wherein N represents the number of URL samples in the training data set, and N(w) represents the number of URL samples containing the word w;

[0063] S3.1.4, calculating the TF-IDF value of the word w by the formula TF-IDF(w) = TF(d, w) x IDF(w);

[0064] S3.1.5, repeating steps S3.1.2 to S3.1.4 to complete the calculation of all words, and based on the TF-IDF values of the respective words, obtaining the vector representation of the URL, and in this example, the vector length is 746.

[0065] Step S3.2 includes the following sub-steps:

[0066] S3.2.1, dividing the URL by single characters, converting the continuous URL string into a single character discontinuous combination, and taking spaces as the separator;

[0067] S3.2.2, adopting the Doc2Vec method to train the PV-DM model, as shown in Figure 3 , in the training process, first map each character and sentence to the vector space, and then combine the sentence vector and its context 2k character vectors when processing each character vector, and optimize the model by maximizing the formula , after training convergence, a good character vector representation is obtained, and in the formula, w t represents the vector corresponding to the character t, and p(w t |w​​t-k ,...,w t+k ) is a probability function;

[0068] S3.2.3 Calculate the mean of the sum of the character vectors in the URL, and use it as the vector representation of the URL. In this example, the vector length is 200.

[0069] The overall structure of the detection model and the structure of the sparse autoencoder are as follows: Figure 3 As shown, the sparse autoencoder in step S4 consists of a three-layer network, including an input layer, a hidden layer, and an output layer. The input layer maps to the hidden layer for encoding, and the hidden layer maps to the output layer for decoding. In this example, both the input and output layers have 946 nodes, the hidden layer has 128 nodes, and the activation function is the Sigmoid function.

[0070] During the training of the sparse autoencoder, the feature vector of the URL is used as the raw data to enter the input layer. After passing through the encoder, the output of the hidden layer is obtained. The output of the hidden layer is further used as the input of the decoder, and finally the reconstructed output data is obtained. The error between the raw data and the output data is calculated by the loss function. The optimization objective is to minimize the loss function and perform backpropagation to update the parameter values ​​of the network.

[0071] During training, the loss function is defined as Wherein, the KL divergence function is in The average activation of hidden layer node j is described, where m represents the number of samples, s represents the number of hidden layer nodes, and β is a parameter controlling the sparsity weights. The optimization objective is to minimize the loss function. In this example, the optimization is performed using the Adam algorithm, with a learning rate of 0.01, a target loss function of 0.01, a sparsity constant of 0.002, a batch size of 100, and 200 training epochs. The network parameters are updated iteratively, and the trained network is obtained when the loss function converges or the training epochs end.

[0072] Step S5 uses the trained model to detect URLs. The feature vectors of the URLs are input into a sparse autoencoder to obtain the reconstructed output, which is then processed by a distance function. Calculate the spatial distance between the input and output. If it is greater than a given threshold, the URL is considered a malicious URL; otherwise, it is considered a normal URL.

[0073] The above description is only one specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for malicious URL detection based on sparse autoencoder, characterized in that, The method comprises the following steps: S1, collecting a comprehensive normal URL as a sample through a web application to construct a training data set; S2, URL preprocessing, and mapping processing of part of the parameter values in the URL according to requirements; S3, feature extraction of the URL sample, conversion into a calculable feature vector, and normalization processing of the feature vector; S4, constructing a sparse autoencoder SAE and training the sparse autoencoder; S5, testing the model detection effect, inputting the feature vector converted from a URL into the trained sparse autoencoder to obtain a reconstructed output vector, and judging whether it is a malicious URL by calculating the spatial distance between the input and the output. 2.The sparse autoencoder-based malicious URL detection method of claim 1, wherein, The URL preprocessing process of step S2 includes escaping, character lowercase conversion, and null character replacement operations, and according to requirements, part of the parameter values in the URL are mapped, and the mapping method is as follows: For the numerical characters in the parameter values, map them to the character '0'; For English characters in the parameter values, map them to the character 'a'; For Chinese characters in the parameter values, map them to the character 'b'; For other symbol characters in the parameter values, map them to the character '#'. 3.The sparse autoencoder-based malicious URL detection method of claim 1, wherein, The URL feature extraction process of step S3 comprises the following steps: S3.1, processing the URL by the TF-IDF method to convert the corresponding URL into a vector v1; S3.2, processing the URL by the Doc2Vec method to convert the corresponding URL into a vector v2; S3.3, connecting the vectors v1 and v2 to obtain a feature vector v for expressing the URL. S3.4, by formula A normalization operation is performed on the feature vector v to obtain a feature vector v', where max(v) represents the maximum value in the vector v and min(v) represents the minimum value in the vector v. 4.The sparse autoencoder-based malicious URL detection method of claim 1, wherein, The sparse autoencoder network constructed in step S4 is composed of three layers of network, including an input layer, a hidden layer and an output layer, the mapping from the input layer to the hidden layer is an encoding process, the mapping from the hidden layer to the output layer is a decoding process, and the activation function of the network is a Sigmod function. 5.The sparse autoencoder-based malicious URL detection method of claim 1, wherein, In the process of training the sparse autoencoder in step S4, the feature vector of the URL is taken as the original data into the input layer, the output of the hidden layer is obtained through the encoder, the output of the hidden layer is further taken as the input of the decoder, and finally the reconstructed data of the output is obtained. The error between the original data and the output data is calculated through the loss function, the loss function is minimized as the optimization objective, the back propagation is carried out, and the parameter values of the network are updated; During training, the loss function is defined as where the KL divergence function is where The average activation of hidden layer node j is described, m represents the number of samples, s represents the number of hidden layer nodes, and β is a parameter for controlling the sparsity weight; taking the minimization of the loss function as the optimization objective, the parameters in the network are updated in iterations through the optimization algorithm, and the trained network is obtained when the loss function converges or the training round ends. 6.The sparse autoencoder-based malicious URL detection method of claim 3, wherein, Step S3.1 comprises the following steps: S3.1.1, adopting the bigram method to segment the URL string to obtain a corresponding URL word array; S3.1.2, by formula Calculate the frequency of the word w in the URL word array d; S3.1.3, by formula Calculate the inverse document frequency of the word w, where N represents the number of URL samples in the training data set, and N(w) represents the number of URL samples containing the word w; S3.1.4, calculating the TF-IDF value of the word w through the formula TF-IDF(w)=TF(d,w)×IDF(w); S3.1.5, repeating steps S3.1.2 to S3.1.4 to complete the calculation of all words, and obtaining the vector representation of the URL based on the TF-IDF values of the words. 7.The sparse autoencoder-based malicious URL detection method of claim 3, wherein, Step S3.2 comprises the following steps: S3.2.1, segmenting the URL by single characters, converting the continuous URL string into a single character discontinuous combination, and taking space as a delimiter; S3.2.2, adopt the Doc2Vec method, train the PV-DM model therein, in the training process, first map each character and sentence to a vector space, and then combine the sentence vector and its context 2k character vectors when processing each character vector, and maximize the formula to optimize the model, and after training convergence, a good character vector expression is obtained, wherein w t represents the vector corresponding to the character t, p(w t |w t-k ,...,w t+k ) is a probability function; S3.2.

3. Calculate the mean of the sum of the individual character vectors in the URL as the vector representation of the URL.

Images