A Web attack detection method based on NLP

Abstract

The present invention discloses a Web attack detection method based on NLP, which uses a scanning command to collect basic information of a target system, imitates human behavior, records each click interaction of the current entire page HTML code, passes the HTML request to an AI module for classification and recognition, classifies it through a fastText fast text classification algorithm, marks it in batches through rules, numerically processes the text, realizes text vectorization, performs word segmentation, cleaning and vectorization on a part of special characters, models the result, obtains a one-hot encoded classification output through a deep learning model, maps the one-hot encoded label to a corresponding two-category label, and selects and verifies a POC plug-in according to the information contained through a verification module; an automated penetration testing system designed through an NLP artificial intelligence algorithm replaces traditional manpower for attack inspection, and helps solve the problems of insufficient personnel and uneven operation levels.

Description

Technical Field

[0001] The present invention belongs to the technical field of wide area networks, and in particular relates to a Web attack detection method based on NLP. Background Art

[0002] Web attack detection is a basic function in penetration testing. Penetration testing refers to the process in which penetration test engineers simulate the information detection, vulnerability assessment and attack methods used by hackers to conduct in-depth detection of the security of the target and discover the vulnerable aspects of the system. There is currently no clear standard definition for penetration testing, and its theory and technology are still in constant change and development. Penetration testing requires testers to have comprehensive capabilities, and the penetration testing tools used are mostly auxiliary tools. There is no very mature comprehensive penetration platform. The scale of Internet-based business continues to expand and become more and more complex, and the cost of manual penetration testing is also increasing, and the difficulty is significantly increased. Penetration testing tools convert some of the heavy, repetitive work and basic work such as vulnerability scanning in such work into software or equipment, so they can provide continuous, automated, and intelligent penetration testing. The demand is becoming increasingly urgent, but due to the huge human factor in the penetration process, the automated penetration platform has not made a breakthrough in the past. However, with the rapid development of artificial intelligence, automated penetration testing has gradually become possible.

[0003] At present, there are many vulnerability scanning tools in China, and their functions are concentrated on the vulnerability scanning of the target. Whether the vulnerability is exploitable is still analyzed and compared by professional security personnel manually. The workload is extremely heavy, and a lot of manpower is required for vulnerability analysis and false positive verification. At present, the automated penetration testing tools on the market are based on the combination of semi-automated scanning tools and POC tools, which cannot meet the needs of automated penetration testing very well. The foreign network security industry developed earlier, and there are a large number of relatively mature automated penetration testing tools, such as Canva, MetaSploit, Core-Impact, etc., which are also semi-automated. As auxiliary penetration testing tools, they can usually only play a role in a certain direction, lacking the consideration of the relevance and logical vulnerabilities of the overall target system, and this method usually adopts the scanning detection method based on the feature library, which will generate a large amount of useless data for the target system and may cause network storms. For this reason, we propose a Web attack detection method based on NLP to realize automated detection in this field. Summary of the invention

[0004] The purpose of the present invention is to provide a Web attack detection method based on NLP to solve the problems raised in the above background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: a Web attack detection method based on NLP, comprising the following steps:

[0006] A. Use scanning commands to collect basic information about the target system, and transmit the scanning result information to the AI ​​module for vulnerability judgment and decision-making. According to the scanning results, such as the scanned port, it will determine whether it is a Web service by sending a request. If it is determined to be a Web service, it will send instructions to the crawler module to crawl page content and capture requests;

[0007] B. Use web crawler technology to imitate the mouse and click recording behavior of humans browsing pages, record each click interaction of the current entire page HTML code, and pass the HTML request to the AI ​​module for classification and recognition;

[0008] C. The AI ​​module receives the HTTP request data passed by the crawler, performs NLP-based data processing, classifies it using the fastText fast text classification algorithm, and then batch labels it using rules, digitizes the text, implements text vectorization, performs word segmentation and vectorization on some special characters, and then models and learns the data to start deep learning;

[0009] D. Through the deep learning model, the information is output according to the one-hot encoding classification, and the corresponding one-hot encoding label is mapped to the corresponding binary classification label;

[0010] E. For the data passed through the AI ​​module, the vulnerability verification module will select the POC plug-in according to the information contained. The AI ​​module will output the corresponding POC file name and dynamically execute the corresponding POC file code through the importlib library of Python. After successful execution, if the vulnerability is confirmed, the module will send the running result of the POC code to the signal module. The signal module will confirm that this vulnerability attack exists and store the information in the corresponding database. Further, after the above steps, the results obtained by AI model processing are stored, and the signal processing mechanism of Django is used to interact with the database through signals. The signal system allows one or more senders to send notifications or signals to a group of receivers. The signal accepts two parameters, the task ID and the result content. The signal will receive feedback on the completion of the POC code detection. Once the signal is received, the logic code corresponding to the signal will be executed. Further, before performing step A, the user enters the detection target information on the client, and the user's input is sent to the server through an HTTP request. After the server verifies the data, it stores the scan information in the database and sends it to the scanning engine through the message queue to complete the scan.

[0011] Furthermore, in step B, specifically, after the crawler module requests the target for the first time, it will obtain all visible 〈a〉 tags of the target, parse the href attributes therein and archive them into a queue waiting for the next crawl. After the extraction is completed, it will look for the form of the current page, and automatically identify the type and content required in the form by simulating the submission of the form. After filling in the form, a simulated submission action will be performed to submit the corresponding form to the remote server, and the corresponding HTTP request will be captured and sent to the AI ​​module. After operating all forms, all buttons on the page will be identified and clicked. After clicking the button, the HTTP request triggered by clicking the button will be captured and sent to the AI ​​module. The second request will get the previously parsed URL from the queue and submit the form and click the button again until the data in the queue is empty.

[0012] Furthermore, in step C, batch labeling is performed, and the labeling rules are as follows: the request containing an ID and being characters is labeled as 1; the presence of parameter values ​​and Chinese characters is labeled as 1; and the short or random parameters with numbers are labeled as 0.

[0013] Furthermore, the Keras class Tokenizer is used for serialized text operations during numerical processing, including the establishment of the bag-of-words model TF-IDF. When analyzing the request parameters of hacker attacks, it can be found that dimension and semantics are not factors that have a close impact on the success of the attack. Word2vec can also be used here for text vectorization.

[0014] Furthermore, modeling is used to process data, including three models, ANN model, TEXTCNN model, and DQN model, to enhance the detection accuracy of attack detection behavior.

[0015] Furthermore, in the ANN model, the positive integers are first converted into dense vectors of fixed size through the word embedding layer, and the global pooling is used to shorten the length of the entire fixed-size dense vector and take the average value to reduce the dimension of the next layer of the network, thereby suppressing the deviation of the estimated mean caused by parameter errors and modeling output through the ANN model.

[0016] Furthermore, the same operations as the ANN model are adopted in the TEXTCNN model.

[0017] Furthermore, in the DQN model, the state information obtained from each interaction with the environment is vectorized, an action is obtained based on the state information, and then the action and state are passed to the neural network to obtain the Q value.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] The automatic detection of Web attacks can be achieved through artificial intelligence algorithms, which can be used to automate penetration testing systems, replace traditional manpower for Web security testing, help solve the problems of insufficient security technicians and uneven operation levels, and reduce the data pollution caused by traditional detection methods for target system testing. This application focuses on the search for application vulnerabilities such as SQL injection points, and realizes a complete penetration testing platform. The accuracy of Web attack detection depends on multiple factors. Comprehensive and in-depth retrieval of target sites can ensure the integrity of crawler data, and the integrity of crawler data affects the quality of modeling data. The accuracy of AI module classification and filtering and the rigor of POC code are also crucial to the accuracy of attack detection. In this study, the page request parameters and page themes were classified by NLP algorithm, and the detection results were output through modeling, which solved the problems of long scanning time and more invalid parameter attack tests in traditional inspection methods, and reduced the output of traffic and the possibility of network storms. The accuracy of the entire model lies in whether the training set is accurate and the order of magnitude. In the future, the focus will be on automatic modeling of data collection and annotation, so as to form a high-quality training set for rapid iteration requirements, thereby having a more accurate impact on the output results. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0021] Figure 1 It is a system framework diagram of the present invention;

[0022] Figure 2 The crawler workflow diagram of the present invention;

[0023] Figure 3 is a Loss curve diagram of an embodiment of the present invention;

[0024] Figure 4 It is an Epochs curve diagram of an embodiment of the present invention; DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0026] Reference Figure 1 - Figure 4 The present invention proposes a technical solution: a Web attack detection method based on NLP, comprising the following steps:

[0027] S1. The user enters the target information on the client, and the user's input is sent to the server through an HTTP request. The server verifies the data and stores the scan information in the database, and sends it to the scanning engine through a message queue to complete the scan;

[0028] S2. Use NMAP to collect basic information about the target system, including all open ports, port services, operating systems and versions, MAC addresses, and open Web service ports. The scan result information is output in XML format and transmitted to the AI ​​module for vulnerability judgment and decision-making. The scanned port will determine whether it is a Web service by sending a request. If it is determined to be a Web service, it will be sent to the crawler module for page content crawling and request capture;

[0029] S3. The crawler module uses Webkit technology to imitate human browsing behavior, record each click interaction of the HTML code of the entire page, and pass the HTML request to the AI ​​module for classification and identification. When it is determined to be a Web service, it will be sent to the crawler module for page content crawling and request capture. Specifically, after the crawler module requests the target for the first time, it will obtain all visible 〈a〉 tags of the target, parse the href attributes in them and archive them in the queue for the next crawl. After the extraction is completed, it will look for the form of the current page, and automatically identify the type and content required in the form by simulating the submission of the form. After filling in the form, simulate the submission action to submit the corresponding form to the remote server, capture the corresponding HTTP request and send it to the AI ​​module. After operating all forms, all buttons on the page will be identified and clicked. After clicking the button, the HTTP request triggered by clicking the button will be captured and sent to the AI ​​module. The second request will get the previously parsed URL from the queue and submit the form and click the button again until the data in the queue is empty.

[0030] S4. Receive HTTP request data from the crawler and perform NLP-based processing. Classify it using the fastText fast text classification algorithm, then label it in batches using rules, digitize the text, implement text vectorization, segment and clean some special characters, and model the vectorized data to complete deep learning. During digitization, use Keras's Tokenizer class to perform serialized text operations, including building a bag-of-words model TF-IDF. Word2vec can also be used for text vectorization.

[0031] Classification processing uses the fastText fast text classification algorithm, which does not require pre-trained word vectors and can speed up training and testing while maintaining high accuracy. For hacker attacks, purely static websites or systems are generally inaccessible, and a feature of dynamic websites is that there is database interaction during the access process. Therefore, the classification here is based on whether there is database interaction. It uses a binary classification, where the possible existence of database interaction behavior is 1, and the absence of database interaction behavior is 0;

[0032] Labeling: batch labeling of classified data according to rules. The rules here include: if the request contains an ID and is in characters, it is labeled as 1; if there is a parameter value and Chinese characters, it is labeled as 1; if it is short or random and contains numbers, it is labeled as 0. Labeling can be done manually, which can better control the access and quality of data output;

[0033] The vectorized data is modeled and learned. This system establishes three models: ANN model, TEXTCNN model, and DQN model. In the ANN model, the word embedding layer is used to convert positive integers into dense vectors of fixed size, for example: [[4],

[20] ]->[[0.25, 0.1], [0.6, -0.2]]. Then, GlobalAveragePooling1D is used to shorten the length of the dense vector of fixed size, take the average value, reduce the dimension of the next layer of the network, and suppress the deviation of the estimated mean caused by parameter errors. The modeling output is performed through ANN. The same operation as the ANN model is used in the TEXTCNN model. In the DQN model, the state information obtained from each interaction with the environment is vectorized, an action is obtained based on the state information, and then the action and state are passed to the neural network to obtain the Q value.

[0034] S5. The one-hot encoded classification output is obtained through the deep learning model, and the corresponding one-hot encoded label is mapped to the corresponding binary classification label. For example, [[0, 1]] represents the possible existence of database interaction behavior; [[1, 0]] represents the absence of interaction with the database. In the actual test, we predict the common attack types in vulnerability detection, including SQL injection, file inclusion, cross-site scripting, etc., as follows:

[0035] ['SQL_INJECT' 'FILE_INCLUDE' 'SQL_INJECT, XSS_INJECT' XSS_INJECT' 'PHPINFO'], construct the matrix through one-hot encoding.

[0036] S6. The vulnerability verification module will select the POC plug-in based on the information contained in the data passed by the AI ​​module. The AI ​​module will output the corresponding POC file name and dynamically execute the corresponding POC file code through the python importlib library. After successful execution, if it is confirmed that there is a vulnerability, the module will send the running result of the POC code to the signal module.

[0037] S7. The signal module stores the results. The signal module interacts with the database through signals. The signal system allows one or more senders to send notifications or signals to a group of receivers.

[0038] By dividing the dataset into three parts, training dataset, validation dataset and test dataset, we can diagnose whether the model is trained properly based on the loss of the training set and the loss of the validation set when training the model, so that we can optimize the algorithm parameters through the visualized Loss curve, and finally train a better model, that is, a network model with strong generalization ability. In this figure, we can see that after about 3 epochs, the Loss curve of the algorithm begins to converge, and a good-fit model is obtained.

[0039] The accuracy curve is a scalar form of whether the true value and the model prediction value are consistent. The accuracy is output by each test on the validation set to show the performance of the algorithm at the current stage. The accuracy shows the performance climbing degree of the algorithm, which can be used to observe whether the neural network is learning in the right direction and stop training at the right time. In this figure, by observing the convergence curve of the objective function value and the validation set accuracy, the algorithm has obtained sufficient performance after 3 epochs.

[0040] The key to automating Web attack detection is the discovery and verification of vulnerabilities. Existing discovery tools are based on features such as scan feedback, which can easily attract the attention of the target and also pollute data and network traffic. This paper implements a complete penetration testing platform that includes Web attack detection. Whether the crawler data used is comprehensive depends on whether the target site is fully retrieved. The accuracy of system vulnerability detection lies in the accuracy of AI module classification and filtering and the rigor of POC code. The number of vulnerability verifications in this detection platform depends on the number of POC codes. Adding new POC codes can support new vulnerability attack detection. Classifying request parameters and page topics through NLP algorithms solves the problems of long scanning time and more invalid parameter attack tests, reduces traffic output and the possibility of network storms, and implements attack detection with minimal impact on the target system.

[0041] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A Web attack detection method based on NLP, characterized in that: The steps include: A. Use scanning commands to collect basic information of the target system. According to the scanning results, such as the scanned port, determine whether it is a Web service by sending a request. If it is determined to be a Web service, send instructions to the crawler module to crawl page content and capture requests; B. Use web crawler technology to imitate the mouse click recording behavior of humans browsing pages, record each click interaction of the current entire page HTML code, and pass the HTML request to the AI ​​module for classification and recognition; C. The AI ​​module receives the HTTP request data passed by the crawler, performs NLP-based data processing, classifies it using the fastText fast text classification algorithm, and then performs batch labeling based on the rules. The batch labeling rule is that the request contains an ID and the character is labeled as 1; Parameters with Chinese characters and values ​​are marked as 1; short or random parameters with numbers are marked as 0; the text is digitized to achieve text vectorization, some special characters are segmented and cleaned and vectorized, and then the data is modeled to start deep learning; the modeling includes three models, ANN model, TEXTCNN model, and DQN model; in the ANN model, the word embedding layer is used to convert positive integers into dense vectors of fixed size, and the global pooling shortens the length of the entire fixed-size dense vector, takes the average value, reduces the dimension of the next layer of the network, and suppresses the deviation of the estimated mean caused by parameter errors. The modeling output is performed through the ANN model; the same operation as the ANN model is used in the TEXTCNN model; in the DQN model, the state information obtained from each interaction with the environment is vectorized, an action is obtained based on the state information, and then the action and state are passed to the neural network to obtain the Q value; D. Through the deep learning model, the information is output according to the one-hot encoding classification, and the corresponding one-hot encoding label is mapped to the corresponding binary classification label; E. For the data passed by the AI ​​module, the vulnerability verification module selects the POC plug-in according to the information contained in it, and dynamically executes the corresponding POC file code through the python importlib library. After successful execution, if it is confirmed that there is a vulnerability, the module will send the running result of the POC code to the signal module. The signal module will confirm that this vulnerability attack exists and store the information in the corresponding database.

2. According to the NLP-based Web attack detection method of claim 1, it is characterized by: After the above steps, the results of detection and verification are stored and interacted with the database through signals. The signal system allows one or more senders to send notifications or signals to a group of receivers.

3. The NLP-based Web attack detection method according to claim 1, characterized in that: Before performing step A, the user enters the target information on the client, and the user's input is sent to the server through an HTTP request. After the server verifies the data, it stores the information in the database and sends it to the scanning engine through the message queue to complete the scan.

4. The NLP-based Web attack detection method according to claim 1, characterized in that: In step B, specifically, after the crawler module requests the target for the first time, it will obtain all visible 〈a〉 tags of the target, parse the href attributes therein and archive them into the queue for the next crawl. After the extraction is completed, it will look for the form of the current page, and automatically identify the type and content required in the form by simulating the submission of the form. After filling in the form, a simulated submission action will be performed to submit the corresponding form to the remote server, and the corresponding HTTP request will be captured and sent to the AI ​​module. After operating all forms, all buttons on the page will be identified and clicked. After clicking the button, the HTTP request triggered by clicking the button will be captured and sent to the AI ​​module. The second request will get the previously parsed URL from the queue and submit the form and click the button again until the data in the queue is empty.

5. The NLP-based Web attack detection method according to claim 1, characterized in that: During the numerical processing, the Tokenizer class of Keras is used to perform serialized text operations, including the establishment of the bag-of-words model TF-IDF. When analyzing the request parameters of the hacker attack, it was found that dimension and semantics were not factors that had a close impact on the success of the attack. Word2vec was used for text vectorization.

Images