Webpage mining attack detection method and system based on deep learning and multi-dimensional feature recognition

Abstract

The invention discloses a webpage mining attack detection method and system based on deep learning and multi-dimensional feature recognition, and the method comprises the steps: carrying out the preprocessing of collected flow at an offline stage, generating a gray-scale map, and respectively training two webpage mining flow detection models based on VGG-16; after the extracted JavaScript is subjected to anti-confusion and character replacement processing, the JavaScript is respectively used for training two webpage mining JavaScript detection models based on BiLSTM (Bidirectional Long Short Term Memory); and in the online stage, classification is carried out through the trained webpage mining flow detection model and the webpage mining JavaScript script detection model, and classification results are subjected to multi-dimensional feature fusion, so that webpage mining attack detection is realized. According to the method, mining behaviors are detected from two aspects of mining network flow and mining JavaScript. According to the method, the original flow data and the JavaScript are directly used as input, end-to-end webpage mining attack detection is achieved, manual feature design is not needed, and therefore the generalization ability and robustness of the detection method are improved.

Description

Technical Field

[0001] The present invention relates to a technology in the field of information security, specifically a web page mining attack detection method and system based on deep learning and multi-dimensional feature recognition. Background Art

[0002] Web mining was originally intended to provide alternative income to legitimate website owners or non-profit organizations, thus replacing traditional web advertising. Due to its lightweight and high concealment, web mining was quickly exploited by attackers and gradually replaced traditional software / Trojan mining. Attackers used website vulnerabilities to implant mining scripts to hijack users' computing resources for their own profit. However, since the host computing resources called by web mining have a low threshold, and the communication between web mining and mining pools is mixed with normal browser communication traffic, traditional software / Trojan mining detection technology cannot effectively detect web mining attacks. In order to prevent web mining attacks, researchers have proposed a variety of detection methods, such as detection methods based on static and dynamic features of websites. Although these methods perform well on known data sets, they still have some problems in real scenarios, such as insufficient recall and precision, inability to cope with code obfuscation, hiding and other technologies, feature extraction bias caused by human factors, and poor generalization ability. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a webpage mining attack detection method and system based on deep learning and multi-dimensional feature recognition, which detects mining behavior from two aspects: mining network traffic and mining JavaScript scripts. By directly taking the original traffic data and JavaScript scripts as input, end-to-end webpage mining attack detection is achieved without the need for manual feature design, thereby improving the generalization ability and robustness of the detection method.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a webpage mining attack detection method based on deep learning and multi-dimensional feature recognition. In an offline stage, after pre-processing the collected traffic and generating a grayscale image, two VGG-16-based webpage mining traffic detection models are respectively used for training; after de-obfuscation and character replacement processing are performed on the extracted JavaScript scripts, two BiLSTM-based webpage mining JavaScript script detection models are respectively used for training; in an online stage, the trained webpage mining traffic detection model and the webpage mining JavaScript script detection model are classified, and the classification results are subjected to multi-dimensional feature fusion to realize webpage mining attack detection.

[0006] The webpage mining JavaScript script detection model based on BiLSTM includes: a download unit, an anti-obfuscation unit, a character replacement processing unit, a first detection model unit and a second detection model unit, wherein: the download unit downloads all JavaScript files from the webpage according to the asynchronous downloader, and stores the extracted code in the corresponding JS file. The anti-obfuscation unit uses the Esprima tool to parse the obfuscated JavaScript script into an abstract syntax tree (AST), traverses the nodes of the AST through a reconstruction algorithm, retains nodes such as variables, arrays, and functions, thereby realizing the restoration of the JavaScript script. The character replacement processing unit replaces the URL in the JavaScript script with "url", the number with "0", the string of the function parameter with "param", and deletes and segmentes special characters, and obtains the word vector matrix of the JavaScript script through the Word2Vec model. The first detection model unit uses the training set of normal JavaScript scripts, non-mining malicious JavaScript scripts, and mining malicious JavaScript scripts to distinguish normal scripts from malicious scripts through the BiLSTM model. The second detection model unit uses the training set of non-mining malicious JavaScript scripts and mining malicious JavaScript scripts to distinguish between non-mining malicious scripts and mining malicious scripts through the BiLSTM model.

[0007] The present invention relates to a webpage mining attack detection system for implementing the above-mentioned method, comprising: an access simulation unit, a data preprocessing unit, a grayscale image generating unit and a flow detection unit, wherein: the access simulation unit uses a selenium tool to simulate a Chrome browser to access the link in headless mode according to a webpage link to be detected input by a user, and captures the communication flow of the webpage in a closed docker environment and saves it as a pcap file; the data preprocessing unit extracts session data from the collected flow, randomizes MAC addresses and IP addresses, removes invalid data packets and generates a grayscale image as input; the grayscale image generating unit of a first webpage mining flow detection model M1 based on VGG-16 performs binary classification on the generated grayscale image to distinguish malicious and non-malicious flow; the flow detection unit of a second webpage mining flow detection model M2 based on VGG-16 performs binary classification on malicious flow, further distinguishes mining malicious flow and non-mining malicious flow, thereby obtaining a three-classification result of the flow. Technical Effects

[0008] Compared with the prior art, the present invention does not require artificially designed features, and enhances the generalization and robustness of the web mining traffic detection model; reduces the false positives and negatives caused by JavaScript code obfuscation, and enhances the detection capability of mining JavaScript scripts; the web mining attack detection system can detect more mining websites while maintaining a low false positive and negative rate, and has a high precision and recall rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a system diagram of the present invention;

[0010] Figure 2 This is a schematic diagram of the architecture of the web mining traffic detection model based on VGG-16;

[0011] Figure 3 This is a schematic diagram of the BiLSTM-based web mining JavaScript script detection model architecture. DETAILED DESCRIPTION

[0012] like Figure 1 As shown, this embodiment relates to a web mining attack detection method based on deep learning and multi-dimensional feature recognition, including:

[0013] Step 1) Receive the web page link to be detected input by the user, use the selenium tool to simulate the Chrome browser to access the URL in headless mode in a closed docker environment with limited CPU usage, stay on the web page for 45 seconds, use tshark to capture the communication traffic of the web page, save the communication traffic as a pcap file, and use an asynchronous downloader to download all js files carried in the web page, and store the codes extracted from the script tag of the html file in js files respectively.

[0014] Step 2) pre-process the collected traffic and generate a grayscale image, and perform deobfuscation and character replacement processing on the extracted JavaScript script.

[0015] The preprocessing refers to: extracting the traffic data in the pcap package according to the session, for the TCP session, the start and end of the session are marked by the SYN and FIN fields respectively, and for the UDP session, a timeout mechanism is used for processing, and each session is stored in its own pcap package respectively. Then, in order to avoid the host characteristics from becoming the basis for judging malicious traffic, the MAC address and IP address are randomized, and then invalid data packets such as ARP data packets, DNS data packets, and retransmission data packets in the session are removed. Finally, after removing the header information, the first 1024 bytes of each session are intercepted, and the length less than 1024 bytes are padded with 0; each byte of the sliced ​​session data is converted into an 8-bit grayscale pixel, and the sliced ​​session data is converted into a 32×32 grayscale image to generate a data set.

[0016] The deobfuscation method is as follows: using the Esprima tool to parse the obfuscated JavaScript script into an abstract syntax tree (AST), and according to the code obfuscation method, using the abstract syntax tree reconstruction algorithm described in (Ma Hongliang, Wang Wei, Han Zhen. Research on Detection and Deobfuscation Methods of Obfuscated Malicious JavaScript Code [J]. Journal of Computers, 2017, 40(7): 1699-1713.), traversing all nodes of the abstract syntax tree, if the node is a variable, array, function, etc., then retaining the node, and restoring the JavaScript script by traversing the tree.

[0017] The character replacement process refers to: replacing all URLs in JavaScript scripts with "url", replacing all numbers in JavaScript with "0", replacing all strings used as function parameters in JavaScript with "param", deleting special characters such as tabs and control characters, using the word segmentation tool NLTK to segment the JavaScript script data after text generalization, using the Word2Vec model provided by the gensim library to calculate the vector representation of each word in the JavaScript script, and obtaining the word vector matrix of the JavaScript script.

[0018] Step 3) Input the generated grayscale image into the first trained VGG-16-based web mining traffic detection model M1 for binary classification, and then continue to input the data classified as malicious traffic into the second trained VGG-16-based web mining traffic detection model M2 for binary classification to obtain a three-classification result of the traffic; input the preprocessed JavaScript script into the first BiLSTM-based web mining JavaScript script detection model M3 for binary classification, and then continue to input the data classified as malicious JavaScript scripts into the second BiLSTM-based web mining JavaScript script detection model M4 for binary classification to obtain a three-classification result of the JavaScript script.

[0019] The first VGG-16-based web mining traffic detection model refers to: using a training set of three types of traffic data, namely normal traffic, non-mining malicious traffic, and mining malicious traffic, to train a VGG-16-based model for distinguishing normal traffic from malicious traffic. Specifically, during the training process, both non-mining malicious traffic and mining malicious traffic are regarded as malicious traffic, and the specific parameters are set as follows: the optimization method uses SGD, the learning rate is 0.0001, the batch size is 32, the loss function of the classification result is calculated for each training round, and the network parameters are updated, and the training is stopped after 50 training rounds.

[0020] The second VGG-16-based web mining traffic detection model refers to: using a training set of three types of traffic data, namely normal traffic, non-mining malicious traffic, and mining malicious traffic, a VGG-16-based model for distinguishing between mining malicious traffic and non-mining malicious traffic is trained, specifically: the optimization method uses SGD, the learning rate is 0.0001, the batchsize is 32, the loss function of the classification result is calculated for each training round, and the network parameters are updated, and the training is stopped after 50 training rounds.

[0021] The training set of the three types of traffic data, namely, normal traffic, non-mining malicious traffic, and mining malicious traffic, is obtained in the following way: the traffic data of normal traffic and non-mining malicious traffic are obtained from the public data set USTC-TFC2016, and the traffic data of mining malicious traffic is obtained from the public data set SmaSheD, and the mining pool IP address and port number are set on the Monero mining webpage, and the network traffic of the mining webpage server communicating with the mining pool IP is tracked using a network traffic capture tool. Each collection lasts 5 minutes, and a total of 64 mining traffic data are collected to generate a data set containing the three types of traffic data, namely, normal traffic, non-mining malicious traffic, and mining malicious traffic, which is stored in the form of a pcap package.

[0022] The first BiLSTM-based web mining JavaScript script detection model refers to: using a training set of three types of JavaScript scripts, namely normal JavaScript scripts, non-mining malicious JavaScript scripts, and mining malicious JavaScript scripts, to train a BiLSTM-based model for distinguishing normal JavaScript scripts from malicious JavaScript scripts. Specifically, both non-mining malicious JavaScript scripts and mining malicious JavaScript scripts are regarded as malicious JavaScript scripts, and the specific parameters are set as follows: the optimization method uses Adam, the learning rate is 0.001, the batch size is 128, the loss function of the classification result is calculated for each training round, and the network parameters are updated, and the training is stopped after 50 training rounds.

[0023] The second BiLSTM-based web mining JavaScript script detection model refers to: using a training set of non-mining malicious JavaScript scripts and mining malicious JavaScript scripts, training a BiLSTM-based model for distinguishing between non-mining malicious JavaScript scripts and mining malicious JavaScript scripts, specifically: the optimization method uses Adam, the learning rate is 0.001, the batch size is 128, the loss function of the classification result is calculated for each training round, and the network parameters are updated, and the training is stopped after 50 training rounds.

[0024] The training set of three types of JavaScript scripts, namely normal JavaScript scripts, non-mining malicious JavaScript scripts, and mining malicious JavaScript scripts, is obtained in the following way: non-mining malicious JavaScript scripts are obtained from the public data set JavaScript Malware Collection of GitHub, the Top 200 websites recorded by Alexa site are regarded as normal websites, JavaScript scripts in the source code of these web pages are obtained by crawlers as normal JavaScript scripts, and mining malicious JavaScript scripts are obtained from web pages in the recent blacklist list of Coinblocklist, thereby generating a data set containing three types of JavaScript scripts, namely normal JavaScript scripts, non-mining malicious JavaScript scripts, and mining malicious JavaScript scripts.

[0025] Step 4) The three-classification results of the traffic and the three-classification results of the JavaScript script are fused through multi-dimensional features to obtain the detection results of the web link, specifically including: when the web mining traffic detection model determines it as malicious mining traffic, the website is determined to be an ongoing web mining attack; when the web mining script detection model determines it as a malicious mining script, but the web mining traffic detection model does not determine it as malicious mining traffic, the website is determined to be a potential web mining attack with high confidence; when the web mining traffic and script detection models are both determined to be normal, the website is determined to be a normal website; in other cases, the website is determined to be a potential web mining attack with low confidence.

[0026] After specific practical experiments, in the Ubuntu 18.04 64-bit system, with Python 3.9 and pytorch1.13.1 as the specific environment settings for writing programs, the web mining traffic detection model, web mining script detection model, and web mining attack detection system involved in this embodiment have the following effects:

[0027] The data set used by the web mining traffic detection model contains 313,454 data items, including 138,431 normal traffic items and 175,023 malicious traffic items. Among the malicious traffic items, there are 83,459 mining malicious traffic items and 91,564 non-mining malicious traffic items. The classification accuracy of models M1 and M2, as well as the precision, recall, and F1-score of each type of traffic are described in the following table:

[0028] The dataset used by the basic web mining detection model contains 6540 data items, including 3651 normal scripts and 2889 malicious scripts. Among the malicious scripts, 769 are mining malicious scripts and 2120 are non-mining malicious scripts. The classification accuracy of models M3 and M4, as well as the precision, recall, and F1-score of each type of script are shown in the following table:

[0029] There are 60 websites in the dataset used for the performance evaluation of the web mining attack detection system, including 20 normal websites, 20 websites with unobfuscated mining scripts and normal running of the scripts, 10 websites with obfuscated mining scripts and normal running of the scripts, and 10 websites with mining scripts but not running. The precision rate of the web mining attack detection system is 100%, the recall rate is 98.91%, and the F1-score is 99.45%.

[0030] Compared with the prior art, the web mining traffic detection model of the present invention directly uses the original traffic as data input to achieve end-to-end web mining traffic detection without the need for artificially designed features, thereby enhancing the generalization ability and robustness of the model; after deobfuscating the obfuscated JavaScript script, the web mining script detection model uses the Doc2Vec model to extract the code feature vector of the script, and uses BiLSTM to classify the JavaScript code to find out the mining malicious JavaScript script, thereby enhancing the detection capability of the web mining malicious script and reducing the false positives and negatives caused by JavaScript code obfuscation; the web mining attack detection system combines the advantages of the web mining traffic detection model and the web mining script detection model, and can detect more web mining attacks while maintaining low false positives and negatives, showing high precision and recall rates, and is suitable for actual application scenarios.

[0031] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A web mining attack detection method based on deep learning and multi-dimensional feature recognition, characterized in that: In the offline stage, after preprocessing the collected traffic and generating a grayscale image, it is used to train two web mining traffic detection models based on VGG-16. After deobfuscation and character replacement processing, the extracted JavaScript scripts are used to train two web mining JavaScript script detection models based on BiLSTM. In the online stage, the trained web mining traffic detection model and web mining JavaScript script detection model are used for classification, and the classification results are subjected to multi-dimensional feature fusion to realize web mining attack detection.

2. The webpage mining attack detection method based on deep learning and multi-dimensional feature recognition according to claim 1 is characterized in that: The BiLSTM-based webpage mining JavaScript script detection model comprises: a download unit, an anti-obfuscation unit, a character replacement processing unit, a first detection model unit and a second detection model unit, wherein: the download unit downloads all JavaScript files from a webpage according to an asynchronous downloader, and stores the extracted code into a corresponding JS file; the anti-obfuscation unit uses the Esprima tool to parse the obfuscated JavaScript script into an abstract syntax tree, traverses the nodes of the AST through a reconstruction algorithm, retains nodes such as variables, arrays, and functions, thereby realizing the restoration of the JavaScript script; the character replacement processing unit performs character replacement and special character deletion and word segmentation in the JavaScript script, and obtains a word vector matrix of the JavaScript script through a Word2Vec model; the first detection model unit uses a training set of normal JavaScript scripts, non-mining malicious JavaScript scripts and mining malicious JavaScript scripts to distinguish normal scripts from malicious scripts through a BiLSTM model; the second detection model unit uses a training set of non-mining malicious JavaScript scripts and mining malicious JavaScript scripts to distinguish non-mining malicious scripts from mining malicious scripts through a BiLSTM model.

3. A web mining attack detection system implementing the method described in claim 1 or 2, characterized in that: include: A simulated access unit, a data preprocessing unit, a grayscale image generating unit and a traffic detection unit, wherein: the simulated access unit uses the selenium tool to simulate the Chrome browser to access the link in headless mode according to the web page link to be detected input by the user, and captures the communication traffic of the web page in a closed docker environment and saves it as a pcap file; the data preprocessing unit extracts session data from the collected traffic, randomizes MAC addresses and IP addresses, removes invalid data packets and generates a grayscale image as input; the grayscale image generating unit of the first web page mining traffic detection model M1 based on VGG-16 distinguishes malicious and non-malicious traffic by performing binary classification on the generated grayscale image; the traffic detection unit of the second web page mining traffic detection model M2 based on VGG-16 performs binary classification on malicious traffic, further distinguishes mining malicious traffic and non-mining malicious traffic, thereby obtaining a three-classification result of the traffic.

Images