A Web attack end-to-end detection method, system and medium based on deep learning

Abstract

The present invention discloses a Web attack end-to-end detection method, system and medium based on deep learning. For the HTTP request to be detected, the word features of the URL are learned through the DFFN branch, and the combination relationship and local relationship of the payload are learned through the CNN branch, so as to obtain the URL word feature matrix and the payload word feature matrix respectively, and then the URL word feature matrix and the payload word feature matrix are sequentially connected as the input matrix of the Bi-LSTM with a self-attention mechanism, so as to capture the complex link relationship between the URL and the payload, and output the detection result. Compared with the prior art that cannot fully identify potential threats, the embodiment of the present invention adopts the method of hybrid neural network, and considers the link relationship between the URL and the payload, and the local and combination relationship between the payloads at the same time, so as to improve the accuracy and comprehensiveness of detection and identification.

Description

Technical Field

[0001] The present invention relates to the field of network technology, and in particular to a deep learning-based end-to-end Web attack detection method, system and medium. Background Art

[0002] Web applications have become an indispensable part of modern society and business activities, but they are also accompanied by an increase in security threats, making web application security a hot research topic. With the increase in the number and complexity of network attacks, Web attack detection faces a series of challenges and complexities, such as: (1) Attackers use a variety of means and strategies, including but not limited to SQL (Structured Query Language) injection, cross-site scripting (XSS), cross-site request forgery (CSRF), etc. This diversity makes it difficult for a single detection method to cover all attack types, requiring the detection system to be able to identify variants of different means to ensure comprehensive attack coverage. (2) Attackers tend to use covert methods to evade detection, such as using encrypted communications, obfuscating attack traffic, or imitating normal user behavior. This makes the detection system face greater challenges, requiring it to not only be able to identify common attack patterns, but also to have the ability to combat covert and deceptive attacks. (3) Traditional Web attack detection methods often rely on static rules, which are based on known attack features. However, static rules have inherent limitations and cannot adapt to the emergence of new attacks and zero-day vulnerabilities. Attackers can circumvent these static rules by modifying the attack payload or using variant attack methods, making the detection system vulnerable to attack. (4) Traditional detection methods often have problems with missed reports (failure to detect actual attacks) and false positives (incorrectly marking normal traffic as attacks). This may cause the system to be unable to respond in a timely manner when facing real attacks, or cause unnecessary interference to legitimate users. In recent years, the field of web attack detection has been noticed by more scholars. In response to the above-mentioned problems and the shortcomings of traditional web detection models, scholars have proposed many different types of algorithm models.

[0003] Common algorithm models are machine learning models and deep learning models, such as KNN (K-NearestNeighbors), DT (Decision Tree) and NB (Naive Bayes) models, or CNN models, RNN models and LSTM neural network models for detecting specific attack types. However, the existing technology does not consider the link relationship between URLs and payloads, nor the local and combined relationship between payloads when detecting Web attacks end-to-end, resulting in the inability of existing detection methods to fully identify various potential threats. Summary of the invention

[0004] The embodiments of the present invention provide a deep learning-based end-to-end Web attack detection method, system and medium, which simultaneously consider the link relationship between URL and payload, and the local and combined relationship between payloads, so as to improve the accuracy and comprehensiveness of detection and identification.

[0005] In a first aspect, an embodiment of the present invention provides an end-to-end detection method for Web attacks based on deep learning, comprising:

[0006] Get the first HTTP request to be detected;

[0007] Performing character embedding processing on the first HTTP request to obtain a first character embedding matrix, and encoding the first character embedding matrix according to a preset flat layer and DFNN network to obtain a first URL word feature matrix corresponding to the first HTTP request;

[0008] Generate a first word frequency matrix corresponding to the first HTTP request according to a preset word frequency dictionary, and generate a first payload word feature matrix corresponding to the first word frequency matrix according to a preset CNN coding layer; wherein the CNN coding layer is used to learn the local and combination relationship of the payload;

[0009] The first URL word feature matrix and the first payload word feature matrix are input into a preset Bi-LSTM classification model, so that the Bi-LSTM classification model concatenates the first URL word feature matrix and the first payload word feature matrix to obtain a first concatenated feature matrix, and performs self-attention weight calculation on the first concatenated feature matrix, and then identifies the first HTTP request as a normal request or a malicious request based on the calculation result.

[0010] For the HTTP request to be detected, the embodiment of the present invention learns the word features of the URL through the DFFN branch, and learns the combination relationship and local relationship of the payload through the CNN branch, thereby obtaining the URL word feature matrix and the payload word feature matrix respectively, and then sequentially connects the URL word feature matrix and the payload word feature matrix as the input matrix of the Bi-LSTM with a self-attention mechanism, captures the complex link relationship between the URL and the payload, and outputs the detection result. Compared with the prior art that cannot fully identify potential threats, the embodiment of the present invention adopts a hybrid neural network method, which simultaneously considers the link relationship between the URL and the payload, and the local and combination relationship between the payloads, to improve the accuracy and comprehensiveness of detection and identification.

[0011] As a preferred embodiment of this embodiment, the character embedding process is performed on the first HTTP request to obtain a first character embedding matrix, specifically:

[0012] Preprocessing the first HTTP request to remove a header of the first HTTP request;

[0013] According to the preset character statistics dictionary and the pre-trained Glove-50d word embedding vector, the pre-processed first HTTP request is embedded to generate the first character embedding matrix.

[0014] In this preferred example, since the line headers of HTTP requests are usually unified, the URL is encoded after removing the headers to improve the encoding accuracy. Secondly, the HTTP request is converted using a preset character statistical dictionary, and the non-standard words in the HTTP request are represented as vectors. The pre-trained Glove-50d word embedding vector information is used to enrich the semantic information of each character, so that even in the case of a limited data set, the semantic relationship of each word can be more effectively captured. This step is crucial to improving the deep learning model (DFNN)'s ability to understand and express URL data, thereby improving detection accuracy.

[0015] As a preferred embodiment of this embodiment, the first character embedding matrix is ​​encoded according to the preset flat layer and DFNN network to obtain the first URL word feature matrix corresponding to the first HTTP request, which is specifically:

[0016] Encoding the first character embedding matrix according to the word encoding layer, capturing the features of the words segmented by the preset segmentation characters in the first character embedding matrix, and obtaining the first URL word feature matrix;

[0017] Among them, the word encoding layer is composed of the flat layer and the DFNN network. The ReLu function is used as the activation function after each hidden layer of the DFNN network, which is used to adjust the output vector obtained by the DFNN network to a matrix of a preset size according to the average length of the words in the URL.

[0018] In this preferred embodiment, by introducing a word encoding layer combining a flat layer and DFNN, the features of words separated by preset segmentation characters (such as " / ") can be better captured to form a final word feature matrix of the URL. This processing makes the model more adaptable, and can more accurately understand and represent URL data composed of non-standard words, further improving the detection accuracy.

[0019] As a preferred embodiment of this embodiment, generating a first word frequency matrix corresponding to the first HTTP request according to a preset word frequency dictionary is specifically:

[0020] According to the word frequency dictionary, convert the first HTTP request into a two-dimensional array of the same size as the word frequency dictionary to obtain the first word frequency matrix;

[0021] The word frequency dictionary is generated by calculating the occurrence frequency of the effective load in the training sample in the entire training sample according to the BoW model, and pairing the occurrence frequency with the effective load.

[0022] In this preferred example, in order to solve the problem of disorder in the payload in the HTTP request, the bag-of-words (BoW) technology is used to represent the payload as a feature vector, which does not depend on the order of words but is based on their frequency of occurrence, thus better considering the disorder.

[0023] As a preferred embodiment of this embodiment, the CNN encoding layer is provided with two superimposed convolutional layers for extracting representative features of the word frequency matrix and learning the local and combinational relationships of the payload;

[0024] A maximum pooling layer and a ReLu activation function are set after each convolutional layer, and the output results are processed by a flat layer and a fully connected network to generate a payload word feature matrix.

[0025] In this preferred example, after using the BoW technology to consider the disorder of the payload, a CNN encoding layer is introduced. Through convolution operations, maximum pooling and activation functions, this layer can capture the local and combinational relationships of the payload without paying attention to the specific arrangement of the payload in the HTTP request. This preferred example enables the model to learn and express the semantic relationship of the payload more robustly, further improving the recognition accuracy.

[0026] As a preferred embodiment of this embodiment, the Bi-LSTM classification model concatenates the first URL word feature matrix and the first payload word feature matrix to obtain a first concatenated feature matrix, which is specifically:

[0027] The Bi-LSTM classification model is provided with a first Bi-LSTM layer and a second Bi-LSTM layer;

[0028] The Bi-LSTM classification model concatenates the first URL word feature matrix and the first payload word feature matrix according to the column axis to obtain a first concatenated feature matrix;

[0029] The first Bi-LSTM layer processes the first concatenated feature matrix according to a preset time step, calculates a first hidden state output of each time step, and outputs a first sequence of the entire time step;

[0030] The second Bi-LSTM layer processes the first hidden state output of each time step according to the time step, calculates the second hidden state output of each time step, and outputs the second sequence of the entire time step and the second hidden state output of the last time step.

[0031] In this preferred example, a multi-layer Bi-LSTM structure is used to solve the problem of link relationship between URL and payload. Bi-LSTM can capture the complex temporal dependency between URL and payload by processing the word feature matrix of http request, and provides effective modeling of the correlation between them.

[0032] As a preferred embodiment of this embodiment, the Bi-LSTM classification model performs self-attention weight calculation on the first concatenated feature matrix, specifically as follows:

[0033] The Bi-LSTM classification model establishes a self-attention relationship between the second sequence and the second hidden state output of the last time step, calculates the self-attention weight of each time step, and weightedly processes the time step weight and self-attention weight of each time step to generate a feature vector of self-attention of the first HTTP request.

[0034] The step of identifying the first HTTP request as a normal request or a malicious request according to the calculation result is specifically:

[0035] The feature vector of the self-attention is flattened and processed by a fully connected layer, and the output of the fully connected layer processing is subjected to a sigmoid activation function to generate a classification result, so as to determine whether the first HTTP request is a normal request or a malicious request.

[0036] In this preferred example, a self-attention mechanism is introduced to model the situation where different types of web attacks may have different dependencies on URLs and payloads. The self-attention mechanism can more flexibly adjust the degree of attention to different parts of the input sequence by calculating the self-attention weight at each time step, thereby better adapting to the impact of different attack types on specific parts.

[0037] In a second aspect, an embodiment of the present invention provides an end-to-end Web attack detection system based on deep learning, comprising: an acquisition module, a first matrix generation module, a second matrix generation module and a detection module;

[0038] The acquisition module is used to acquire the first HTTP request to be detected;

[0039] The first matrix generation module is used to perform character embedding processing on the first HTTP request to obtain a first character embedding matrix, and to encode the first character embedding matrix according to a preset flat layer and DFNN network to obtain a first URL word feature matrix corresponding to the first HTTP request;

[0040] The second matrix generation module is used to generate a first word frequency matrix corresponding to the first HTTP request according to a preset word frequency dictionary, and to generate a first payload word feature matrix corresponding to the first word frequency matrix according to a preset CNN encoding layer; wherein the CNN encoding layer is used to learn the local and combination relationship of the payload;

[0041] The detection module is used to input the first URL word feature matrix and the first payload word feature matrix into a preset Bi-LSTM classification model, so that the Bi-LSTM classification model splices the first URL word feature matrix and the first payload word feature matrix to obtain a first spliced ​​feature matrix, and performs self-attention weight calculation on the first spliced ​​feature matrix, and then identifies the first HTTP request as a normal request or a malicious request based on the calculation result.

[0042] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the deep learning-based end-to-end Web attack detection method as described in the embodiment of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flowchart of an embodiment of a deep learning-based Web attack end-to-end detection method provided by the present invention;

[0044] Figure 2It is a schematic diagram of the process of generating the word feature matrix of the URL provided by the embodiment of the present invention;

[0045] Figure 3 It is a word encoding layer structure diagram of a URL provided by an embodiment of the present invention;

[0046] Figure 4 is a diagram of a structure of a payload coding layer provided by an embodiment of the present invention;

[0047] Figure 5 It is a structural diagram of a classification model with a self-attention mechanism Bi-LSTM provided in an embodiment of the present invention;

[0048] Figure 6 It is a flow chart of another embodiment of the deep learning-based end-to-end Web attack detection method provided by the present invention;

[0049] Figure 7 It is a structural schematic diagram of an embodiment of the end-to-end detection of Web attacks based on deep learning provided by the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of a deep learning-based Web attack end-to-end detection method provided by the present invention. The method includes steps 101 to 104, each of which is as follows:

[0052] Step 101: Obtain a first HTTP request to be detected.

[0053] Step 102: Perform character embedding processing on the first HTTP request to obtain a first character embedding matrix, and perform encoding processing on the first character embedding matrix according to a preset flat layer and DFNN network to obtain a first URL word feature matrix corresponding to the first HTTP request.

[0054] In this embodiment, step 102 is specifically as follows: preprocessing the first HTTP request and removing the header of the first HTTP request; embedding characters of the preprocessed first HTTP request according to a preset character statistics dictionary and a pre-trained Glove-50d word embedding vector to generate a first character embedding matrix; encoding the first character embedding matrix according to a word encoding layer, capturing features of words segmented by preset segmentation characters in the first character embedding matrix, and obtaining the first URL word feature matrix.

[0055] In this embodiment, the preprocessing of the first HTTP request deletes the header information of the HTTP request line, because the header in the HTTP request is usually unified. In order to encode the URL, a character statistics dictionary containing 67 characters is constructed for character type statistics. Among them, the 67 characters are 26 lowercase letters in the URL, 30 special characters, 10 numbers and a unique token to represent any unknown characters.

[0056] In the URL encoding process, the URL is first converted into a dictionary position encoding vector, where each element represents the position of a character in the dictionary. In order to ensure the consistency of the length of the URL, the maximum length L is set according to the frequency of the character length distribution of the URL. For the CSIC2010 public dataset, L is set to 80. URLs with a length of more than 80 characters will be truncated, while URLs with a length of less than 80 characters will be padded with unique tokens. Subsequently, the pre-trained Glove-50d word embedding vector is used for URL character embedding. The character embedding matrix of the URL with a shape of (80,50) is obtained through character embedding. Finally, the word encoding layer is introduced to capture the features of words in the URL separated by preset separation characters (separated by " / "). The preset separation characters can be, but are not limited to, set to other characters.

[0057] The word encoding layer is composed of the flat layer and the DFNN network. After each hidden layer of the DFNN network, the ReLu function is used as the activation function to adjust the output vector obtained by the DFNN network to a matrix of a preset size (4,50) according to the average length of the words in the URL, the word feature matrix of the URL. Its structure can be but is not limited to see Figure 3 .like Figure 3 As shown in the figure, the entire neural network is considered as a stack of linear layers and ReLU activation functions. Specifically, the input of the first linear layer is the input layer, the output of the last linear layer is the output layer, and the combination of the linear layer and the ReLU activation function in the middle forms the hidden layer. The top (a, b) of the linear layer represents the input vector dimension and the output vector dimension. The top (a, b) of other components represents the shape of the data.

[0058] In this embodiment, Glove-50d word embedding is a word embedding method based on global vectors, where "Glove" stands for "Global Vectors for Word Representation". By training on a large-scale text corpus, the method generates a 50-dimensional vector for each word, encoding the semantic relationship in the vocabulary through the distance and direction in the vector space. Such an embedding vector not only retains the semantic relationship between words, but also can be applied to various natural language processing tasks such as text classification, sentiment analysis, and named entity recognition through transfer learning. Its feature of training on a large-scale corpus gives Glove word embedding a strong generalization ability, making it an important tool in processing natural language tasks and providing key semantic information for deep learning models. In URL encoding, the pre-trained Glove-50d word embedding vector is used to map the character position encoding vector in the URL to a character embedding matrix, providing key semantic information for the deep learning model (DFNN), helping the model to better understand and process URL data.

[0059] As an example of this embodiment, the pre-trained Glove-50d word embedding vector can be replaced with a pre-trained word embedding vector of another model, such as BERT (Bidirectional Encoder Representations from Transformers).

[0060] The DFNN network is a deep feedforward neural network that uses a hierarchical structure, including an input layer, multiple hidden layers, and an output layer, to learn and capture abstract representations of data. The mathematical relationship between the input layer, hidden layer, and output layer is as shown in Equation 1. If there are multiple hidden layers, the output of the previous hidden layer is used as the input of the next hidden layer. In the URL feature learning task, the goal of DFNN is to model the complex word feature relationships of URLs by learning nonlinear activation functions. The process of training DFNN involves the use of a backpropagation algorithm, which adjusts the model parameters, including weights and biases in the network, by minimizing the loss function. Through this process, DFNN gradually adapts to the data, enabling it to better fit nonlinear functions and thus capture word feature relationships in URL data. Ultimately, DFNN outputs the URL word feature relationships that have been learned and adjusted, enabling the model to better understand and represent the semantic relationships of the input URL data.

[0061] y=σ(w T x+b), (1)

[0062] Among them, x represents the hidden layer input vector (input layer); w Tis the weight parameter of the hidden layer of the DFFN neural network, which is adjusted by the back propagation algorithm; σ is a nonlinear activation function, and the Relu activation function is used in the present invention; y represents the output vector of the hidden layer (output layer), and b is a constant.

[0063] In step 102, non-standard words in the URL are represented as vectors through character-level processing, and the semantic information of each character is enriched using the pre-trained Glove-50d word embedding vector information, wherein the pre-trained Glove-50d word embedding vector can be replaced by other pre-trained word embedding vectors. In this way, the semantic relationship of each word can be captured more effectively even in the case of a limited data set. This step is crucial to improving the understanding and expression capabilities of the deep learning model (DFNN) for URL data. By introducing a word encoding layer that combines a flat layer and DFNN, the model can better capture the features of words separated by " / " to form the final word feature matrix of the URL. This processing process makes the model more adaptable and can more accurately understand and represent URL data composed of non-standard words.

[0064] Step 103: Generate a first word frequency matrix corresponding to the first HTTP request according to a preset word frequency dictionary, and generate a first payload word feature matrix corresponding to the first word frequency matrix according to a preset CNN encoding layer; wherein the CNN encoding layer is used to learn the local and combinational relationships of the payload.

[0065] In this embodiment, in order to obtain the representation of the payload, a word feature matrix of the payload is introduced. First, a payload frequency dictionary is constructed to generate a word frequency matrix of the payload. Subsequently, the word frequency matrix is ​​used as the input of the CNN encoding layer to learn the local and combinational relationships of the payload. Finally, the output of the CNN encoding layer constitutes the word feature matrix of the payload.

[0066] Specifically, according to a preset word frequency dictionary, a first word frequency matrix corresponding to the first HTTP request is generated, specifically: according to the word frequency dictionary, the first HTTP request is converted into a two-dimensional array of the same size as the word frequency dictionary to obtain the first word frequency matrix; wherein the word frequency dictionary is generated by calculating the frequency of occurrence of the payload in the training sample in the entire training sample according to the BoW model, and pairing the frequency of occurrence with the payload.

[0067] Before the conversion, a word frequency dictionary needs to be created. The filter first segments all payloads from the http request using special characters such as " / ", "&", and "+". Then, the bag of words (BoW) technique is used to calculate the number of times each payload appears in the entire dataset. In the BoW model, the words in the text are considered as features, and the number of occurrences of each payload constitutes a feature vector. The feature vector can reflect the degree of repetition of each payload in the entire dataset. Finally, each payload is paired with its corresponding frequency to generate a word frequency dictionary.

[0068] The generated word frequency dictionary is used to create a word frequency matrix. For each HTTP request, a two-dimensional array with the same dictionary size is created, which is conducive to the use of CNN. Based on the CSIC2010 dataset, the size of this two-dimensional array is 200×170 (set according to the type of payload contained in the dataset). If the payload matches a word in the dictionary, the corresponding position in the array is set to the frequency of the payload in the frequency dictionary. Otherwise, it is set to 0.

[0069] In the present embodiment, the bag of words (BoW) technology is a text representation method that aims to convert text information into a vector form. In the bag of words model, the text is regarded as an unordered (combination relationship) word set, ignoring the order and structure of the words, and only focusing on the frequency of occurrence of the words (the prototype of the present invention is frequency). Specifically, the process of the bag of words technology includes first decomposing the text into words and counting the number of times each word appears in the text. Then, by constructing a vector, each dimension represents a word, and the value of the vector represents the frequency of occurrence of the corresponding word in the text.

[0070] As an example of this embodiment, the word frequency matrix of the payload can be replaced by other feature matrices representing disordered relationships, indicating that when processing the payload, the word frequency matrix is ​​not the only representation method, and other feature matrices can be used to capture the disordered relationships in the payload.

[0071] In order to better capture the local and combinatorial relationships of the payload, a CNN encoding layer is introduced to learn the word representation of the payload. The CNN encoding layer is set with two superimposed convolutional layers to extract representative features of the word frequency matrix and learn the local and combinatorial relationships of the payload; after each convolutional layer, a maximum pooling layer and a ReLu activation function are set, and the output results are processed through a flat layer and a fully connected network to generate a payload word feature matrix.

[0072] The structure of the CNN encoding layer is as follows Figure 4As shown in Figure 1. First, two stacked convolutional layers are used to extract representative features of the word frequency matrix to capture the local combination relationship between different payloads, where each convolutional layer is followed by a maximum pooling layer and a ReLu activation function. The convolutional layers consist of 6 and 16 channels, respectively, with a 3×3 kernel size and a stride of 1. The kernel size of the maximum pooling layer is 2×2, with a stride of 2, and the convolutional layer operation and the maximum pooling layer operation are calculated according to Equation 2 and Equation 3, respectively. After that, the flattening layer converts the output matrix of the convolutional layer into a vector, which is then processed by a fully connected neural network. The network has an input layer with 31,488 neurons (obtained from the results of the convolutional neural network structure), a hidden layer with 2,000 neurons, and an output layer with 1,000 neurons, using a ReLu activation function. Finally, based on the average number of payloads in HTTP requests, the output vector of the neural network is reshaped into a 20×50 matrix representing the word features of the payload.

[0073] Convolutional Neural Network (CNN) is a neural network model with two key features: weight sharing of convolutional layers and spatial pooling of pooling layers. These mechanisms give CNN unique advantages in local relationship modeling. It not only performs well in accuracy, but also significantly surpasses traditional fully connected neural networks in computational efficiency. In CNN, the input multi-dimensional tensor is subjected to feature extraction and mapping through convolution operations. The convolution operation generates a new feature map by performing a dot product of each element of the input tensor with the corresponding weight of the convolution kernel tensor, accumulating the bias term, and then processing it through pooling operations. This process is performed by continuously moving the convolution kernel to gradually construct the entire feature map. For a two-dimensional tensor X, during the convolution pooling process, each element q of the new feature map Q ij It can be calculated by the following formula, where y is the result of the convolution operation, then y i,j It can be calculated by formula (2):

[0074]

[0075] Where h and w are the height and width of the convolution kernel. i+m,j+n is the element of the input tensor X, w m.n represents the weight of the convolution kernel, and b represents the bias. Commonly used pooling operations include Max-pooling and Average-pooling. The pooling operator used in the delay layer of the present invention is Max-pooling, and its calculation formula is as follows:

[0076] q i，j =max(y i，j ,y i , j+1 ..., y i，j+k-1 , ..., yi+s-1，y ,y i+s-1，j+1 ,.....,y i+s-1，j+k-1 ) (3)

[0077] Where s and k are the height and width of the kernel of the pooling layer, y is the result of the convolution operation, and q is the result of the pooling operation, that is, the new feature map element.

[0078] In this step, multiple strategies are adopted to address the problem of disorder in the payload in HTTP requests. First, in the preprocessing stage, the payload is segmented by using special characters such as " / ", "&", and "+", ensuring that each key-value pair is processed independently regardless of the order of their appearance. Secondly, the bag-of-words (BoW) technique is used to represent the payload as a feature vector, which does not depend on the order of words but is based on their frequency of occurrence, thus better considering the disorder. The most critical is the introduction of the CNN encoding layer, which, through convolution operations, maximum pooling, and activation functions, is able to capture the local and combinatorial relationships of the payload regardless of its arrangement in the HTTP request. These comprehensive strategies enable the model to learn and express the semantic relationships of the payload more robustly.

[0079] Step 104: Input the first URL word feature matrix and the first payload word feature matrix into a preset Bi-LSTM classification model, so that the Bi-LSTM classification model concatenates the first URL word feature matrix and the first payload word feature matrix to obtain a first concatenated feature matrix, and performs self-attention weight calculation on the first concatenated feature matrix, and then identifies the first HTTP request as a normal request or a malicious request based on the calculation result.

[0080] In this embodiment, step 104 is specifically as follows: a first Bi-LSTM layer and a second Bi-LSTM layer are provided in the Bi-LSTM classification model;

[0081] The Bi-LSTM classification model concatenates the first URL word feature matrix and the first payload word feature matrix according to the column axis to obtain a first concatenated feature matrix; the first Bi-LSTM layer processes the first concatenated feature matrix according to a preset time step, calculates the first hidden state output of each time step, and outputs the first sequence of the entire time step; the second Bi-LSTM layer processes the first hidden state output of each time step according to the time step, calculates the second hidden state output of each time step, and outputs the second sequence of the entire time step and the second hidden state output of the last time step; the Bi-LSTM classification model establishes a self-attention relationship between the second sequence and the second hidden state output of the last time step, calculates the self-attention weight of each time step, and weightedly processes the time step weight and the self-attention weight of each time step to generate a feature vector of self-attention of the first HTTP request; the Bi-LSTM classification model flattens the feature vector of the self-attention and processes it with a fully connected layer, uses a sigmoid activation function to generate a classification result for the output processed by the fully connected layer, and determines whether the first HTTP request is a normal request or a malicious request.

[0082] In order to model the link relationship between the URL and the payload, the embodiment of the present invention adopts a multi-layer Bi-LSTM with a self-attention mechanism. The structure of this layer is shown in Figure 5 . It includes two Bi-LSTM layers, a self-attention layer, a flattening layer, and a fully connected layer. In addition, we use the dropout strategy in both Bi-LSTM layers to mitigate potential overfitting problems. Bi-LSTM refers to the schematic diagram of the bidirectional long short-term memory network structure; LSTM unit refers to the schematic diagram of the internal structure of a unit in the long short-term memory network; self-attention mechanism refers to the basic step flow chart of the self-attention mechanism, and Scale is the dot product of Q and K multiplied by a scaling factor The classification model refers to the basic flow chart of the classification model based on Bi-LSTM with self-attention mechanism.

[0083] In this classification model, the URL word feature matrix and the payload word feature matrix are first concatenated to obtain the word feature matrix of the http request, denoted as Q f . Use Q f As the input of the first Bi-LSTM, the output of the first Bi-LSTM at time step t is calculated according to Formula 8, denoted as O 1t Then O 1tIt is used as the input of the second Bi-LSTM, which performs the same operation as the first Bi-LSTM to obtain the output of time step t, denoted as O 2t , and remember H 2D Represents the hidden state output of the last time step D. Among them, the input dimension of the first Bi-LSTM layer is 50, and the dimension of the hidden state is 128. The input dimension and hidden state dimension of the second Bi-LSTM layer are both 128.

[0084] Therefore, O 2t =[O 21 ,O 22 ,...,O 2D Each element in ] represents a partial feature of the HTTP request, and H 2D Represents the characteristics of the entire HTTP request. In order to evaluate the importance of different parts in the HTTP request, it is necessary to 2D and O 2t A self-attention relationship is established between them, which is achieved by calculating the self-attention weight of each time step t. In this process, H 2D and O 2t As the input of the self-attention mechanism. For each time step, the Key is calculated according to Formula 9 t 、Value t 、Query t , score e t and weight a t Then according to formula 10, the weight a of each time step is t and Value t The weighted addition is performed to obtain the feature vector z of the self-attention http request. Subsequently, z is flattened and processed through a fully connected layer. Finally, the output of the fully connected layer is activated using the sigmoid function to produce the final classification result. The number of neurons in the input layer and output layer of the fully connected neural network is 128 and 1 respectively.

[0085] As an example of this embodiment, the implementation method of the LSTM self-attention mechanism is dot product attention, and the implementation method of the self-attention mechanism can be but is not limited to being replaced by additive attention.

[0086] In this embodiment, LSTM is an improvement on the traditional RNN, capable of capturing and retaining long-term dependencies and patterns in the sequence. It effectively solves the long-term distance dependency and gradient vanishing problems in RNN through the gate mechanism. In addition, Bi-LSTM combines forward and backward information transfer to more comprehensively capture the dependencies and features in the sequence. The core of LSTM lies in its cell state, which traverses the neural network as a horizontal connection. LSTM regulates the cell state by selectively discarding or retaining information through a gate structure. These gates include a forget gate, an input gate, and an output gate, defined as:

[0087]

[0088] Among them, f t represents the forget gate, i t represents the input gate, o t represents the input gate, σ represents the sigmoid function, W f , W i and W o is the parameter of the neural network, which is adjusted with back propagation, [h t-1 , x t ] represents the hidden state h of the previous time step t-1 , and the input x at the current time step t , b f , b i and b o are the bias items of the forget gate, input gate, and output gate respectively. LSTM uses a special mechanism to choose which information to remember and forget, that is, to create candidate information vectors and update the cell state, which is defined as follows:

[0089]

[0090] in represents the candidate information vector, W C represents the weight matrix of the tanh layer of the input gate for linear transformation, b C represents the bias term of the candidate information vector, and tanh represents the hyperbolic tangent activation function. t represents the cell state at time step t, f t is the forget gate, C t-1 is the cell state at time step t-1, i t is the input gate. Finally, the output of LSTM at the current time step t is defined as follows:

[0091] h t =o t *tanh(C t ), (6)

[0092] where ht represents the output at time step t, o t represents the output of the output gate, C t represents the cell state at time step t, and tanh represents the hyperbolic tangent activation function. The feature learning of Bi-LSTM requires forward and backward structures, and M(·) is used to define the LSTM unit (the output is calculated according to Formula 5). Therefore, and Used to define

[0093]

[0094] Forward and backward LSTM, which are defined as follows:

[0095] in is the weight of the forward LSTM, is the weight of the backward LSTM, and are the bias items of the forward and backward LSTM layers respectively. Then the final outputs of the forward and backward are connected and transformed by identity or neural network as the final output of Bi-LSTM, which is defined as follows:

[0096]

[0097] Among them, t is the output of Bi-LSTM time step t, and are the outputs of the forward and backward LSTM units, respectively. Indicates the connection between the two, g can be an identity function or a function trained with a neural network.

[0098] LSTM self-attention mechanism: Bi-LSTM with self-attention mechanism allows Bi-LSTM to assign different weights to each time step to pay more attention to the information related to the current task. This weighted processing enables Bi-LSTM to better adapt to changes and complex patterns in the sequence. Specifically, after the sequence passes through Bi-LSTM, the output O of each time step t is obtained. t and the hidden state output H at the last time step D D . Introduce the attention mechanism into LSTM, record O = [O1, O2, ..., O D ], that is, it is necessary to establish H D The self-attention relationship with O. The implementation of the LSTM self-attention mechanism is dot product attention. The output 0t of each time step is linearly transformed to obtain Key t and Value t , the output HD of the last time step is multiplied by the transformation matrix ω Q As Query. At time step t, Keyt 、Value t 、Query t , score e t and weight a t The calculation formula is as follows:

[0099]

[0100] Among them, Query does not change with time steps, ω Q ,ω V and ω K are the parameters of the neural network, which are modified with back propagation, d k is the dimension of Key, is the scaling factor to prevent the inner product value from being too large and affecting the learning of the neural network. Finally, the weight a of each time step is t and Value t The weighted addition obtains the text feature vector z with self-attention, which is defined as follows:

[0101]

[0102] In this embodiment, a multi-layer Bi-LSTM structure is used to solve the problem of the link relationship between the URL and the payload. By processing the word feature matrix of the http request, Bi-LSTM can capture the complex temporal dependency between the URL and the payload, and provide effective modeling of the correlation between them. On the other hand, a self-attention mechanism is introduced to model the situation where different types of web attacks may have different dependencies on the URL and the payload. The self-attention mechanism can more flexibly adjust the degree of attention to different parts of the input sequence by calculating the attention weight of each time step, so as to better adapt to the impact of different attack types on specific parts.

[0103] In order to better illustrate the application process of the present invention, it is possible but not limited to refer to Figure 6 , Figure 6 A schematic diagram of a flow chart of another embodiment of the deep learning-based end-to-end Web attack detection method provided by the present invention.

[0104] 1. Feature stitching

[0105] After the URL word feature matrix and the payload word feature matrix are generated, the two are concatenated to obtain the word feature matrix of the http request, denoted as Q f .

[0106] enter:

[0107] (1) The word feature matrix of URL (denoted as URL_matrix), with the dimension of (number of words × feature dimension).

[0108] (2) Payload word feature matrix (denoted as Payload_matrix), with dimension (number of words × feature dimension).

[0109] Processing: Concatenate URL_matrix and Payload_matrix along the column axis to form the word feature matrix Qf of the entire http request.

[0110] Output: Qf, with dimension (number of URL words + number of payload words) × feature dimension - (24 × 50).

[0111] 2. Link relationship learning

[0112] The first Bi-LSTM layer:

[0113] Input: URL and payload word feature matrix Qf.

[0114] Processing: The Bi-LSTM layer processes the input sequence one by one at time step t. At each time step, it considers the current input (word features of the URL and payload) and the hidden state of the previous time step, and calculates (see Bi-LSTM below for specific calculations) the output O of the current time step 1t The bidirectional structure of Bi-LSTM enables it to capture both forward and backward information in the sequence.

[0115] Output: O 1t is the output of Bi-LSTM at time step t, which represents the representation of the input sequence at this time step. The output of the entire time step sequence is O 1t =[O 11 ,O 12 ,...,O 1D ], where D is the length of the sequence.

[0116] The second Bi-LSTM layer:

[0117] Input: The output sequence O of the first Bi-LSTM 1t =[O 11 ,O 12 ,...,O 1D ].

[0118] Processing: Similar to the first Bi-LSTM, the second Bi-LSTM processes the output sequence of the first Bi-LSTM one by one at time step t. At each time step, it considers the current input (the output O of the first Bi-LSTM) 1t ) and the hidden state of the previous time step, and calculate the output O of the current time step 2tThe bidirectional structure of this layer allows the model to learn more complex contextual information at each time step in the sequence.

[0119] Output: O2t is the output of the second Bi-LSTM at time step t, which represents the representation of the entire input sequence at a given time step. The output of the entire time step sequence is O 2t =[O 21 ,O 22 ,...,O 2D ] and the hidden state output H at the last time step D 2D .

[0120] 3. Self-Attention Calculation

[0121] In this step, the importance of different parts of the HTTP request is evaluated, the self-attention relationship is calculated, and the calculated weight a is used t and Value t Perform weighted addition to obtain the feature vector z of the self-attention HTTP request.

[0122] Input: O 2t =[O 21 ,O 22 ,...,O 2D Each element in ] represents a partial feature of the HTTP request, and H 2D Represents the characteristics of the entire HTTP request.

[0123] Processing: In order to evaluate the importance of different parts of the HTTP request, it is necessary to 2D and O 2t A self-attention relationship is established between them, which is achieved by calculating the attention weight of each time step t. In this process, H 2D and O 2t As the input of the self-attention mechanism. For each time step, the Key is calculated according to Formula 9 t 、Value t 、Query t , score e t and weight a t Then according to formula 10, the weight a of each time step is t and Value t Perform weighted addition to obtain the feature vector z of the self-attention http request.

[0124] Output: feature vector z of the self-attention http request.

[0125] 4. Final classification results

[0126] Input: feature vector z of self-attention http request

[0127] Processing: z is flattened and then processed by a fully connected layer (feedforward neural network). Finally, the output of the fully connected layer is passed through the Sigmoid activation function to obtain a probability value between 0 and 1, indicating the malicious probability of the HTTP request. The number of neurons in the input layer and output layer of the fully connected neural network are 128 and 1 respectively.

[0128] Output: classification results

[0129] 5. Classification result determination

[0130] Input: The output of the Sigmoid activation function, i.e., the malicious probability value.

[0131] Processing: Compare the probability value with a predetermined threshold.

[0132] Output: If the probability value is greater than the threshold, the HTTP request is judged as a malicious request; otherwise, it is judged as a normal request.

[0133] In order to verify the accuracy of the present invention, as shown in Table 1, the detection method of the present invention performs well on the CSIC2010 data set. By comparing the performance indicators in the table, it can be seen that the detection method of the present invention has achieved significant improvements in accuracy, precision, recall and F1 score. On the validation set, the accuracy rate increased by 1.86%. This shows that the method has achieved better results in detecting Web attacks.

[0134]

[0135] Table 1 Comparison results of the proposed method and existing methods on the CSIC 2010 dataset

[0136] In addition, the accuracy of the present invention on the CSIC 2010 dataset exceeds 99% in about 15 training cycles, reaching a relatively high threshold, which shows that the method performs well in processing the CSIC 2010 dataset and can quickly learn and adapt to the patterns in the dataset.

[0137] Accordingly, see Figure 7 , Figure 7 701 is a schematic diagram of a structure of an embodiment of a Web attack end-to-end detection system based on deep learning provided by the present invention. The system includes: an acquisition module 701, a first matrix generation module 702, a second matrix generation module 703 and a detection module 704;

[0138] The acquisition module 701 is used to acquire the first HTTP request to be detected;

[0139] The first matrix generation module 702 is used to perform character embedding processing on the first HTTP request to obtain a first character embedding matrix, and to encode the first character embedding matrix according to a preset flat layer and DFNN network to obtain a first URL word feature matrix corresponding to the first HTTP request;

[0140] The second matrix generation module 703 is used to generate a first word frequency matrix corresponding to the first HTTP request according to a preset word frequency dictionary, and generate a first payload word feature matrix corresponding to the first word frequency matrix according to a preset CNN coding layer; wherein the CNN coding layer is used to learn the local and combination relationship of the payload;

[0141] The detection module 704 is used to input the first URL word feature matrix and the first payload word feature matrix into a preset Bi-LSTM classification model, so that the Bi-LSTM classification model splices the first URL word feature matrix and the first payload word feature matrix to obtain a first spliced ​​feature matrix, and performs self-attention weight calculation on the first spliced ​​feature matrix, and then identifies the first HTTP request as a normal request or a malicious request based on the calculation result.

[0142] The more detailed working principle of this system can be referred to but not limited to the relevant contents recorded above.

[0143] Accordingly, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the deep learning-based end-to-end Web attack detection method as described in an embodiment of the present invention.

[0144] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the terminal device.

[0145] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0146] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0147] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the terminal device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile terminal, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0148] Wherein, if the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0149] As can be seen from the above, the end-to-end detection method and system for Web attacks based on deep learning provided by the embodiment of the present invention, for the HTTP request to be detected, learns the word features of the URL through the DFFN branch, and learns the combination relationship and local relationship of the payload through the CNN branch, thereby obtaining the URL word feature matrix and the payload word feature matrix respectively, and then sequentially connects the URL word feature matrix and the payload word feature matrix as the input matrix of the Bi-LSTM with a self-attention mechanism, captures the complex link relationship between the URL and the payload, and outputs the detection result. Compared with the prior art that cannot fully identify potential threats, the embodiment of the present invention adopts a hybrid neural network method, which simultaneously considers the link relationship between the URL and the payload, and the local and combination relationship between the payloads, to improve the accuracy and comprehensiveness of detection and identification.

[0150] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A deep learning-based end-to-end detection method for Web attacks, characterized in that: include: Get the first HTTP request to be detected; Performing character embedding processing on the first HTTP request to obtain a first character embedding matrix, and encoding the first character embedding matrix according to a preset flat layer and DFNN network to obtain a first URL word feature matrix corresponding to the first HTTP request; Generate a first word frequency matrix corresponding to the first HTTP request according to a preset word frequency dictionary, and generate a first payload word feature matrix corresponding to the first word frequency matrix according to a preset CNN coding layer; wherein the CNN coding layer is used to learn the local and combination relationship of the payload; Inputting the first URL word feature matrix and the first payload word feature matrix into a preset Bi-LSTM classification model, so that the Bi-LSTM classification model concatenates the first URL word feature matrix and the first payload word feature matrix to obtain a first concatenated feature matrix, and performs self-attention weight calculation on the first concatenated feature matrix, and then identifies the first HTTP request as a normal request or a malicious request according to the calculation result; The Bi-LSTM classification model concatenates the first URL word feature matrix and the first payload word feature matrix to obtain a first concatenated feature matrix, which is specifically: The Bi-LSTM classification model is provided with a first Bi-LSTM layer and a second Bi-LSTM layer; The Bi-LSTM classification model concatenates the first URL word feature matrix and the first payload word feature matrix according to a column axis to obtain a first concatenated feature matrix; The first Bi-LSTM layer processes the first concatenated feature matrix according to a preset time step, calculates a first hidden state output of each time step, and outputs a first sequence of the entire time step; The second Bi-LSTM layer processes the first hidden state output of each time step according to the time step, calculates the second hidden state output of each time step, and outputs the second sequence of the entire time step and the second hidden state output of the last time step; The Bi-LSTM classification model performs self-attention weight calculation on the first concatenated feature matrix, specifically: The Bi-LSTM classification model establishes a self-attention relationship between the second sequence and the second hidden state output of the last time step, calculates the self-attention weight of each time step, and weights the time step weight and the self-attention weight of each time step to generate a feature vector of the self-attention of the first HTTP request; The step of identifying the first HTTP request as a normal request or a malicious request according to the calculation result is specifically as follows: The feature vector of the self-attention is flattened and processed by a fully connected layer, and the output of the fully connected layer processing is subjected to a sigmoid activation function to generate a classification result, so as to determine whether the first HTTP request is a normal request or a malicious request.

2. The deep learning-based end-to-end Web attack detection method according to claim 1, characterized in that: The character embedding process is performed on the first HTTP request to obtain a first character embedding matrix, specifically: Preprocessing the first HTTP request to remove a header of the first HTTP request; According to the preset character statistics dictionary and the pre-trained Glove-50d word embedding vector, the pre-processed first HTTP request is embedded to generate the first character embedding matrix.

3. The end-to-end detection method for Web attacks based on deep learning according to claim 2 is characterized in that: According to the preset flat layer and DFNN network, the first character embedding matrix is ​​encoded to obtain the first URL word feature matrix corresponding to the first HTTP request, which is specifically: Encoding the first character embedding matrix according to the word encoding layer, capturing the features of the words segmented by the preset segmentation characters in the first character embedding matrix, and obtaining the first URL word feature matrix; Among them, the word encoding layer is composed of the flat layer and the DFNN network. The ReLu function is used as the activation function after each hidden layer of the DFNN network, which is used to adjust the output vector obtained by the DFNN network to a matrix of a preset size according to the average length of the words in the URL.

4. The end-to-end detection method for Web attacks based on deep learning according to claim 1 is characterized in that: The first word frequency matrix corresponding to the first HTTP request is generated according to the preset word frequency dictionary, specifically: According to the word frequency dictionary, convert the first HTTP request into a two-dimensional array of the same size as the word frequency dictionary to obtain the first word frequency matrix; The word frequency dictionary is generated by calculating the occurrence frequency of the effective load in the training sample in the entire training sample according to the BoW model, and pairing the occurrence frequency with the effective load.

5. The end-to-end detection method for Web attacks based on deep learning according to claim 4 is characterized in that: The CNN encoding layer is provided with two superimposed convolutional layers for extracting representative features of the word frequency matrix and learning the local and combinational relations of the payload; A maximum pooling layer and ReLu activation function are set after each convolution layer, and the output results are processed by a flat layer and a fully connected network to generate a payload word feature matrix.

6. A deep learning-based end-to-end Web attack detection system, characterized in that: include: An acquisition module, a first matrix generation module, a second matrix generation module and a detection module; The acquisition module is used to acquire the first HTTP request to be detected; The first matrix generation module is used to perform character embedding processing on the first HTTP request to obtain a first character embedding matrix, and to encode the first character embedding matrix according to a preset flat layer and DFNN network to obtain a first URL word feature matrix corresponding to the first HTTP request; The second matrix generation module is used to generate a first word frequency matrix corresponding to the first HTTP request according to a preset word frequency dictionary, and to generate a first payload word feature matrix corresponding to the first word frequency matrix according to a preset CNN coding layer; wherein the CNN coding layer is used to learn the local and combination relationship of the payload; The detection module is used to input the first URL word feature matrix and the first payload word feature matrix into a preset Bi-LSTM classification model, so that the Bi-LSTM classification model splices the first URL word feature matrix and the first payload word feature matrix to obtain a first spliced ​​feature matrix, and performs self-attention weight calculation on the first spliced ​​feature matrix, and then identifies the first HTTP request as a normal request or a malicious request based on the calculation result.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the deep learning-based end-to-end Web attack detection method according to any one of claims 1 to 5.

Images