WAF log anomaly analysis method and system based on space-time convolutional network

Abstract

The invention discloses a WAF log anomaly analysis method and system based on a space-time convolutional network, belongs to the technical field of network security, and aims to solve the technical problem of how to fully utilize time and spatiality of log data to realize WAF log anomaly detection. Comprising the steps that a log anomaly analysis model is constructed based on a space-time convolutional neural network, the log anomaly analysis model takes a one-dimensional vector formed by splicing an access IP and preprocessed log content as input, spatial feature extraction is conducted on WAF log data, a one-dimensional vector representing spatial features is output, and a log anomaly analysis model is constructed; the WAF log data detection module is used for performing anomaly analysis on the WAF log data by taking splicing of time and one-dimensional vectors representing spatial characteristics as input and outputting a detection result; and performing anomaly analysis on the current WAF log data through the trained log anomaly analysis model, and outputting a detection result.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a WAF log anomaly analysis method and system based on a spatiotemporal convolutional network. Background Art

[0002] Traditional WAF anomaly detection methods mainly rely on predefined rules and signatures. These methods are effective in dealing with known attacks, but are often inadequate in the face of unknown and variant attacks. In addition, these methods are difficult to capture the temporal and spatial characteristics of log data, resulting in limited detection accuracy and real-time performance.

[0003] How to make full use of the time and space of log data to implement WAF log anomaly detection is a technical problem that needs to be solved. Summary of the invention

[0004] The technical task of the present invention is to address the above shortcomings and provide a WAF log anomaly analysis method and system based on a spatiotemporal convolutional network to solve the technical problem of how to fully utilize the temporal and spatial properties of log data to realize WAF log anomaly detection.

[0005] In a first aspect, the present invention provides a WAF log anomaly analysis method based on a spatiotemporal convolutional network, comprising the following steps:

[0006] Data collection: collect historical WAF log data;

[0007] Data preprocessing: Use historical WAF log data as target data and perform data preprocessing on the target data. After preprocessing, the target data is in the data format of time + access IP + log content. The text content in the log content is vectorized to obtain the preprocessed log content.

[0008] Sample construction: For preprocessed log data, the access IP address and preprocessed log content are concatenated into a vector as a vector sample, and a sample set is constructed based on the sample vector and time corresponding to the preprocessed target data;

[0009] Model construction: A log anomaly analysis model is constructed based on a spatiotemporal convolutional neural network, and the log anomaly analysis model is trained based on a sample set to obtain a trained log anomaly analysis model, wherein the log anomaly analysis model takes a one-dimensional vector of the access IP and the pre-processed log content as input, is used to extract spatial features from WAF log data, and outputs a one-dimensional vector representing spatial features, and takes time and the concatenation of one-dimensional vectors representing spatial features as input, is used to perform anomaly analysis on WAF log data, and outputs detection results;

[0010] Log anomaly analysis: Collect the current WAF log data, take the current WAF log data as the target data, perform data preprocessing on the target data, and use the data format of the preprocessed target data as time + access IP + log content. Vectorize the text content in the log content to obtain the preprocessed log content. Take the vector concatenated by the access IP and the preprocessed log content as input, extract spatial features of the current WAF log data through the trained log anomaly analysis model, and output a one-dimensional vector representing the spatial features. Take the time and the concatenation of the one-dimensional vector representing the spatial features as input, perform anomaly analysis on the current WAF log data through the trained log anomaly analysis model, and output the detection results.

[0011] Preferably, data preprocessing is performed on the target data, including the following operations:

[0012] Perform data cleaning operations on the target data to remove or revise outliers, irrelevant or duplicate data, and incomplete data;

[0013] Annotate the cleaned target data and modify the target data into the format of time+access IP+log content;

[0014] For the standardized target data, the text content in the log content is vectorized by word embedding;

[0015] The vectorized target data is subjected to dimensionality reduction processing, and feature extraction is performed through feature engineering to obtain the preprocessed target data.

[0016] Preferably, the log anomaly analysis model includes a spatial feature extraction network unit constructed based on CNN, an anomaly analysis network unit constructed based on LSTM network, and an output layer;

[0017] The spatial feature extraction network unit is configured with a convolution layer and a pooling layer, which takes a one-dimensional vector of the access IP and the pre-processed log content as input, is used to extract spatial features from the WAF log data through convolution operations and pooling operations, and outputs a one-dimensional vector representing the spatial features;

[0018] The anomaly analysis network unit takes the concatenation of time and a one-dimensional vector representing spatial characteristics as input, and is used to perform time correlation analysis on WAF log data;

[0019] The output layer takes the output of the anomaly analysis network unit as input, is used to judge whether the WAF log data is abnormal through a binary classifier, and outputs the predicted detection result;

[0020] Correspondingly, when training the log anomaly analysis model, the sample set also includes real results indicating whether the historical WAF log data is abnormal. A loss function is constructed based on the real results and the predicted detection results, and the model training is performed by minimizing the loss function.

[0021] Preferably, the log anomaly analysis model includes a spatial feature extraction network unit constructed based on CNN, a time correlation network unit constructed based on LSTM network, and an output layer;

[0022] The spatial feature extraction network unit is configured with a convolution layer and a pooling layer, which takes a one-dimensional vector of the access IP and the pre-processed log content as input, is used to extract spatial features from the WAF log data through convolution operations and pooling operations, and outputs a one-dimensional vector representing the spatial features;

[0023] The time correlation network unit takes the concatenation of time and a one-dimensional vector representing spatial characteristics as input, and is used to perform time correlation analysis on WAF log data;

[0024] The output layer uses the output of the time correlation network unit as input, and is used to perform anomaly analysis on the WAF log data through a multi-class classifier, and output a detection result, which includes anomaly types and probability values ​​corresponding to the anomaly types;

[0025] Correspondingly, when training the log anomaly analysis model, the sample set also includes anomaly types representing historical WAF log data as real results. A loss function is constructed based on the real results and the predicted detection results, and the model training is performed by minimizing the loss function.

[0026] In a second aspect, the present invention provides a WAF log anomaly analysis method system based on a spatiotemporal convolutional network, comprising a data acquisition module, a data preprocessing module, a sample construction module, a model construction module and a log anomaly analysis module;

[0027] The data collection module is used to perform the following: collect historical WAF log data;

[0028] The data preprocessing module is used to perform the following: take the historical WAF log data as the target data, perform data preprocessing on the target data, and the data format of the preprocessed target data is time + access IP + log content, and the text content in the log content is vectorized to obtain the preprocessed log content;

[0029] The sample construction module is used to perform the following: for the preprocessed log data, the access IP and the preprocessed log content are spliced ​​into a vector as a vector sample, and a sample set is constructed based on the sample vector and time corresponding to the preprocessed target data;

[0030] The model building module is used to perform the following: construct a log anomaly analysis model based on a spatiotemporal convolutional neural network, and perform model training on the log anomaly analysis model based on a sample set to obtain a trained log anomaly analysis model, wherein the log anomaly analysis model takes a one-dimensional vector of an access IP and pre-processed log content as input, is used to extract spatial features from WAF log data, output a one-dimensional vector representing spatial features, and takes time and a one-dimensional vector representing spatial features as input, is used to perform anomaly analysis on WAF log data, and outputs a detection result;

[0031] The log anomaly analysis module cooperates with the data collection module and the data preprocessing module to perform the following: collect the current WAF log data, take the current WAF log data as the target data, perform data preprocessing on the target data, the data format of the preprocessed target data is time + access IP + log content, and the text content in the log content is vectorized to obtain the preprocessed log content, take the vector spliced ​​by the access IP and the preprocessed log content as input, extract the spatial features of the current WAF log data through the trained log anomaly analysis model, output a one-dimensional vector representing the spatial features, and take the splicing of time and the one-dimensional vector representing the spatial features as input, perform anomaly analysis on the current WAF log data through the trained log anomaly analysis model, and output the detection results.

[0032] Preferably, the data preprocessing module is used to perform the following data preprocessing on the target data:

[0033] Perform data cleaning operations on the target data to remove or revise outliers, irrelevant or duplicate data, and incomplete data;

[0034] Annotate the cleaned target data and modify the target data into the format of time+access IP+log content;

[0035] For the standardized target data, the text content in the log content is vectorized by word embedding;

[0036] The vectorized target data is subjected to dimensionality reduction processing, and feature extraction is performed through feature engineering to obtain the preprocessed target data.

[0037] Preferably, the log anomaly analysis model includes a spatial feature extraction network unit constructed based on CNN, an anomaly analysis network unit constructed based on LSTM network, and an output layer;

[0038] The spatial feature extraction network unit is configured with a convolution layer and a pooling layer, which takes a one-dimensional vector of the access IP and the pre-processed log content as input, is used to extract spatial features from the WAF log data through convolution operations and pooling operations, and outputs a one-dimensional vector representing the spatial features;

[0039] The anomaly analysis network unit takes the concatenation of time and one-dimensional vectors representing spatial characteristics as input, and is used to perform time correlation analysis on WAF log data;

[0040] The output layer takes the output of the anomaly analysis network unit as input, is used to judge whether the WAF log data is abnormal through a binary classifier, and outputs the predicted detection result;

[0041] Correspondingly, when training the log anomaly analysis model, the sample set also includes real results indicating whether the historical WAF log data is abnormal. A loss function is constructed based on the real results and the predicted detection results, and the model training is performed by minimizing the loss function.

[0042] Preferably, the log anomaly analysis model includes a spatial feature extraction network unit constructed based on CNN, a time correlation network unit constructed based on LSTM network, and an output layer;

[0043] The spatial feature extraction network unit is configured with a convolution layer and a pooling layer, which takes a one-dimensional vector of the access IP and the pre-processed log content as input, is used to extract spatial features from the WAF log data through convolution operations and pooling operations, and outputs a one-dimensional vector representing the spatial features;

[0044] The time correlation network unit takes the concatenation of time and a one-dimensional vector representing spatial characteristics as input, and is used to perform time correlation analysis on WAF log data;

[0045] The output layer uses the output of the time correlation network unit as input, and is used to perform anomaly analysis on the WAF log data through a multi-class classifier, and output a detection result, which includes anomaly types and probability values ​​corresponding to the anomaly types;

[0046] Correspondingly, when training the log anomaly analysis model, the sample set also includes anomaly types representing historical WAF log data as real results. A loss function is constructed based on the real results and the predicted detection results, and the model training is performed by minimizing the loss function.

[0047] The WAF log anomaly analysis method and system based on spatiotemporal convolutional network of the present invention have the following advantages:

[0048] 1. Capture time and space characteristics simultaneously: The log anomaly analysis model based on spatiotemporal convolutional neural network can process time and space data simultaneously, capturing the time sequence of events in the log and the spatial distribution characteristics of different IP addresses, geographic locations, etc.;

[0049] 2. Efficient feature extraction: Building a log anomaly analysis model based on spatiotemporal convolutional neural network. The convolution layer extracts spatial features, which can identify the correlation between different IP addresses or geographical locations. At the same time, the LSTM layer extracts temporal features, which can detect abnormal temporal patterns and trends.

[0050] 3. Processing complex data structures: WAF log data usually has a complex structure, including multi-dimensional features (such as request type, response time, IP address, geographic location, etc.). Building a log anomaly analysis model based on spatiotemporal convolutional neural network can extract complex features and patterns through multiple convolutional layers and LSTM layers to process high-dimensional data. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] The present invention is further described below in conjunction with the accompanying drawings.

[0053] Figure 1 This is a flowchart of a WAF log anomaly analysis method based on a spatiotemporal convolutional network in Example 1. DETAILED DESCRIPTION

[0054] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0055] The embodiments of the present invention provide a WAF log anomaly analysis method and system based on a spatiotemporal convolutional network, which are used to solve the technical problem of how to fully utilize the temporal and spatial properties of log data to realize WAF log anomaly detection.

[0056] Embodiment 1:

[0057] The present invention discloses a WAF log anomaly analysis method based on a spatiotemporal convolutional network, which includes five steps: data collection, data preprocessing, sample construction, model construction, and log anomaly analysis.

[0058] Step S100: Data collection: collecting historical WAF log data.

[0059] In this embodiment, the WAF log data includes a large amount of relevant content such as basic information, network-related information, request and response information, etc.

[0060] Step S200: Data preprocessing: Use historical WAF log data as target data, perform data preprocessing on the target data, and the data format of the target data after preprocessing is time + access IP + log content, and vectorize the text content in the log content to obtain the preprocessed log content.

[0061] As a specific implementation of data preprocessing, the following operations are included:

[0062] (1) Perform data cleaning operations on the target data to remove or revise outliers, irrelevant or duplicate data, and incomplete data;

[0063] (2) Annotate the cleaned target data and modify the target data into the format of time + access IP + log content;

[0064] (3) For the standardized target data, the text content in the log content is vectorized through word embedding;

[0065] (4) Perform dimensionality reduction on the vectorized target data and extract features through feature engineering to obtain the preprocessed target data.

[0066] There are many different WAF log formats. The purpose of preprocessing is to develop a unified specification to facilitate model reading. For example, different logs can be unified into the format of time + IP access + ... + content. The specific format is not fixed and needs to be determined according to the log format. It can be determined that in order to capture time and space information, the time and access IP information must be taken out separately. The rest can be divided in detail or viewed as a whole (viewing it as a whole is also an advantage of machine learning in dealing with problems).

[0067] Necessary operations for data preprocessing:

[0068] (1) Log cleaning: removing or correcting erroneous, incomplete, irrelevant or duplicate data;

[0069] (2) Standardized data, that is, the unified format mentioned above;

[0070] (3) Quantify the text content. Because machine learning or deep learning models cannot understand text content, the text needs to be converted into a 0.1 vector similar to [0, 1, 0..., 0, 1] or a higher-dimensional 0.1 matrix. The text content needs to be embedded in words. There are many models for embedding words. The mainstream one is the Transformer encoder. This method determines that the specific content of the log can be segmented in detail or used directly for supervised learning regardless of the format.

[0071] (4) The remaining preprocessing operations, such as reducing data dimensions and creating features, need to be determined based on the detailed information of the log.

[0072] Step S300: Sample construction: For the preprocessed log data, the access IP and the preprocessed log content are concatenated into a vector as a vector sample, and a sample set is constructed based on the sample vector and time corresponding to the preprocessed target data.

[0073] Step S400 model construction: construct a log anomaly analysis model based on a spatiotemporal convolutional neural network, and train the log anomaly analysis model based on a sample set to obtain a trained log anomaly analysis model, wherein the log anomaly analysis model takes a one-dimensional vector of the access IP and the preprocessed log content as input, is used to extract spatial features from WAF log data, and output a one-dimensional vector representing the spatial features, and takes time and the splicing of the one-dimensional vector representing the spatial features as input, is used to perform anomaly analysis on the WAF log data, and output detection results.

[0074] In this embodiment, the log anomaly analysis model includes a spatial feature extraction network unit constructed based on CNN, an anomaly analysis network unit constructed based on LSTM network, and an output layer.

[0075] As a specific implementation of the model, the spatial feature extraction network unit is configured with a convolution layer and a pooling layer, which takes the one-dimensional vector of the access IP and the preprocessed log content as input, and is used to extract spatial features from the WAF log data through convolution operations and pooling operations, and output a one-dimensional vector representing the spatial features; the anomaly analysis network unit takes the concatenation of time and the one-dimensional vector representing the spatial features as input, and is used to perform time correlation analysis on the WAF log data; the output layer takes the output of the anomaly analysis network unit as input, and is used to judge whether the WAF log data is abnormal through a binary classifier, and output the predicted detection results.

[0076] Correspondingly, when training the log anomaly analysis model, the sample set also includes real results indicating whether the historical WAF log data is abnormal. A loss function is constructed based on the real results and the predicted detection results, and the model training is performed by minimizing the loss function.

[0077] As another specific implementation of the model, the spatial feature extraction network unit is configured with a convolution layer and a pooling layer, which takes the one-dimensional vector of the access IP and the preprocessed log content as input, and is used to extract spatial features from the WAF log data through convolution operations and pooling operations, and output a one-dimensional vector representing the spatial features; the time correlation network unit takes the concatenation of time and the one-dimensional vector representing the spatial features as input, and is used to perform time correlation analysis on the WAF log data; the output layer takes the output of the time correlation network unit as input, and is used to perform anomaly analysis on the WAF log data through a multi-class classifier, and outputs the detection results, which include anomaly types and probability values ​​corresponding to the anomaly types.

[0078] Correspondingly, when training the log anomaly analysis model, the sample set also includes anomaly types representing historical WAF log data as real results. A loss function is constructed based on the real results and the predicted detection results, and the model training is performed by minimizing the loss function.

[0079] Step S500 log anomaly analysis: collect current WAF log data, take the current WAF log data as target data, perform data preprocessing on the target data, the data format of the preprocessed target data is time + access IP + log content, and the text content in the log content is vectorized to obtain the preprocessed log content, take the vector spliced ​​by the access IP and the preprocessed log content as input, extract spatial features of the current WAF log data through the trained log anomaly analysis model, output a one-dimensional vector representing the spatial features, and take time and the splicing of the one-dimensional vector representing the spatial features as input, perform anomaly analysis on the current WAF log data through the trained log anomaly analysis model, and output the detection results.

[0080] The log anomaly analysis model of this embodiment includes a spatial feature extraction network unit built based on CNN, an anomaly analysis network unit built based on LSTM network, and an output layer. WAF logs are essentially time series data, and each time window represents a series of request information within a certain period of time. Considering the characteristics of time series, one-dimensional convolution (1D CNN) is usually the most direct choice. 1D CNN can slide on the feature sequence within the time window and capture different types of local patterns through different convolution kernels. For example, one convolution kernel focuses on identifying the access rules of a specific HTTP method, and another convolution kernel looks for the specific time distribution of a specific URL access. In addition, adding a pooling layer after the convolution layer, such as average pooling, can reduce the computational complexity, while increasing the generalization ability of the model and reducing the risk of overfitting. Use LSTM to extract time features and capture the time dependency in the data. The recurrent layer can remember long-term dependencies, and the feature sequence processed by CNN received by the LSTM layer (the feature vector of each time window) can further analyze the change pattern of these features over time. By looping through each time step in the sequence, LSTM can learn the dynamic characteristics of features evolving over time, such as abnormal changes in request frequency, behavior patterns within a specific time period, etc. In order to perform convolution operations at each time step and keep the time dimension unchanged, a TimeDistributed layer can be used to apply a convolution layer to each time step of the time series.

[0081] Deploy the model to the production environment and receive real-time or scheduled aggregated WAF log data streams. Set anomaly probability thresholds based on business needs, and requests above this threshold are considered anomalies. When the model predicts an anomaly, notify the security team via API, email, or other means.

[0082] The model is trained based on existing data and is difficult to respond to some new anomalies. It may also misjudge existing anomalies. Therefore, it is necessary to continuously optimize the model based on feedback and emerging attack patterns. Comparison data between model prediction results and manual review is collected for model iteration. The model is retrained regularly with newly collected labeled data to maintain its effectiveness.

[0083] Embodiment 2:

[0084] The present invention discloses a WAF log anomaly analysis system based on a spatiotemporal convolutional network, comprising a data acquisition module, a data preprocessing module, a sample construction module, a model construction module and a log anomaly analysis module.

[0085] The data collection module is used to perform the following: Collect historical WAF log data.

[0086] In this embodiment, the WAF log data includes a large amount of relevant content such as basic information, network-related information, request and response information, etc.

[0087] The data preprocessing module is used to perform the following: using historical WAF log data as target data, preprocessing the target data, and formatting the target data into time + access IP + log content after preprocessing, and vectorizing the text content in the log content to obtain the preprocessed log content.

[0088] As a specific implementation of data preprocessing, this module is used to perform the following operations:

[0089] (1) Perform data cleaning operations on the target data to remove or revise outliers, irrelevant or duplicate data, and incomplete data;

[0090] (2) Annotate the cleaned target data and modify the target data into the format of time + access IP + log content;

[0091] (3) For the standardized target data, the text content in the log content is vectorized through word embedding;

[0092] (4) Perform dimensionality reduction on the vectorized target data and extract features through feature engineering to obtain the preprocessed target data.

[0093] There are many different WAF log formats. The purpose of preprocessing is to develop a unified specification to facilitate model reading. For example, different logs can be unified into the format of time + IP access + ... + content. The specific format is not fixed and needs to be determined according to the log format. It can be determined that in order to capture time and space information, the time and access IP information must be taken out separately. The rest can be divided in detail or viewed as a whole (viewing it as a whole is also an advantage of machine learning in dealing with problems).

[0094] Necessary operations for data preprocessing:

[0095] (1) Log cleaning: removing or correcting erroneous, incomplete, irrelevant or duplicate data;

[0096] (2) Standardized data, that is, the unified format mentioned above;

[0097] (3) Quantify the text content. Because machine learning or deep learning models cannot understand text content, the text needs to be converted into a 0.1 vector similar to [0, 1, 0..., 0, 1] or a higher-dimensional 0.1 matrix. The text content needs to be embedded in words. There are many models for embedding words. The mainstream one is the Transformer encoder. This method determines that the specific content of the log can be segmented in detail or used directly for supervised learning regardless of the format.

[0098] (4) The remaining preprocessing operations, such as reducing data dimensions and creating features, need to be determined based on the detailed information of the log.

[0099] The sample construction module is used to perform the following: for the preprocessed log data, the access IP and the preprocessed log content are spliced ​​into a vector as a vector sample, and a sample set is constructed based on the sample vector and time corresponding to the preprocessed target data.

[0100] The model building module is used to perform the following: build a log anomaly analysis model based on a spatiotemporal convolutional neural network, and train the log anomaly analysis model based on a sample set to obtain a trained log anomaly analysis model, wherein the log anomaly analysis model takes a one-dimensional vector of the access IP and the preprocessed log content as input, is used to extract spatial features of WAF log data, output a one-dimensional vector representing the spatial features, and takes time and the splicing of the one-dimensional vector representing the spatial features as input, is used to perform anomaly analysis on the WAF log data, and output the detection results.

[0101] In this embodiment, the log anomaly analysis model includes a spatial feature extraction network unit constructed based on CNN, an anomaly analysis network unit constructed based on LSTM network, and an output layer.

[0102] As a specific implementation of the model, the spatial feature extraction network unit is configured with a convolution layer and a pooling layer, which takes the one-dimensional vector of the access IP and the preprocessed log content as input, and is used to extract spatial features from the WAF log data through convolution operations and pooling operations, and output a one-dimensional vector representing the spatial features; the anomaly analysis network unit takes the concatenation of time and the one-dimensional vector representing the spatial features as input, and is used to perform time correlation analysis on the WAF log data; the output layer takes the output of the anomaly analysis network unit as input, and is used to judge whether the WAF log data is abnormal through a binary classifier, and output the predicted detection results.

[0103] Correspondingly, when training the log anomaly analysis model, the sample set also includes real results indicating whether the historical WAF log data is abnormal. A loss function is constructed based on the real results and the predicted detection results, and the model training is performed by minimizing the loss function.

[0104] As another specific implementation of the model, the spatial feature extraction network unit is configured with a convolution layer and a pooling layer, which takes the one-dimensional vector of the access IP and the preprocessed log content as input, and is used to extract spatial features from the WAF log data through convolution operations and pooling operations, and output a one-dimensional vector representing the spatial features; the time correlation network unit takes the concatenation of time and the one-dimensional vector representing the spatial features as input, and is used to perform time correlation analysis on the WAF log data; the output layer takes the output of the time correlation network unit as input, and is used to perform anomaly analysis on the WAF log data through a multi-class classifier, and outputs the detection results, which include anomaly types and probability values ​​corresponding to the anomaly types.

[0105] Correspondingly, when training the log anomaly analysis model, the sample set also includes anomaly types representing historical WAF log data as real results. A loss function is constructed based on the real results and the predicted detection results, and the model training is performed by minimizing the loss function.

[0106] The log anomaly analysis module cooperates with the data collection module and the data preprocessing module to perform the following: collect the current WAF log data, take the current WAF log data as the target data, perform data preprocessing on the target data, the data format of the preprocessed target data is time + access IP + log content, and the text content in the log content is vectorized to obtain the preprocessed log content, take the vector spliced ​​by the access IP and the preprocessed log content as input, extract the spatial features of the current WAF log data through the trained log anomaly analysis model, output a one-dimensional vector representing the spatial features, and take the splicing of time and the one-dimensional vector representing the spatial features as input, perform anomaly analysis on the current WAF log data through the trained log anomaly analysis model, and output the detection results.

[0107] The system of this embodiment can execute the method disclosed in Example 1 to implement WAF log anomaly analysis.

[0108] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A WAF log anomaly analysis method based on spatiotemporal convolutional network, characterized in that: The steps include: Data collection: collect historical WAF log data; Data preprocessing: Use historical WAF log data as target data and perform data preprocessing on the target data. After preprocessing, the target data is in the data format of time + access IP + log content. The text content in the log content is vectorized to obtain the preprocessed log content. Sample construction: For preprocessed log data, the access IP address and preprocessed log content are concatenated into a vector as a vector sample, and a sample set is constructed based on the sample vector and time corresponding to the preprocessed target data; Model construction: A log anomaly analysis model is constructed based on a spatiotemporal convolutional neural network, and the log anomaly analysis model is trained based on a sample set to obtain a trained log anomaly analysis model, wherein the log anomaly analysis model takes a one-dimensional vector of the access IP and the pre-processed log content as input, is used to extract spatial features from WAF log data, and outputs a one-dimensional vector representing spatial features, and takes time and the concatenation of one-dimensional vectors representing spatial features as input, is used to perform anomaly analysis on WAF log data, and outputs detection results; Log anomaly analysis: Collect the current WAF log data, take the current WAF log data as the target data, perform data preprocessing on the target data, and use the data format of the preprocessed target data as time + access IP + log content. Vectorize the text content in the log content to obtain the preprocessed log content. Take the vector concatenated by the access IP and the preprocessed log content as input, extract spatial features of the current WAF log data through the trained log anomaly analysis model, and output a one-dimensional vector representing the spatial features. Take the time and the concatenation of the one-dimensional vector representing the spatial features as input, perform anomaly analysis on the current WAF log data through the trained log anomaly analysis model, and output the detection results.

2. According to claim 1, the WAF log anomaly analysis method based on spatiotemporal convolutional network is characterized in that: Perform data preprocessing on the target data, including the following operations: Perform data cleaning operations on the target data to remove or revise outliers, irrelevant or duplicate data, and incomplete data; Annotate the cleaned target data and modify the target data into the format of time+access IP+log content; For the standardized target data, the text content in the log content is vectorized by word embedding; The vectorized target data is subjected to dimensionality reduction processing, and feature extraction is performed through feature engineering to obtain the preprocessed target data.

3. According to claim 1, the WAF log anomaly analysis method based on spatiotemporal convolutional network is characterized in that: The log anomaly analysis model includes a spatial feature extraction network unit constructed based on CNN, an anomaly analysis network unit constructed based on LSTM network, and an output layer; The spatial feature extraction network unit is configured with a convolution layer and a pooling layer, which takes a one-dimensional vector of the access IP and the pre-processed log content as input, is used to extract spatial features from the WAF log data through convolution operations and pooling operations, and outputs a one-dimensional vector representing the spatial features; The anomaly analysis network unit takes the concatenation of time and one-dimensional vectors representing spatial characteristics as input, and is used to perform time correlation analysis on WAF log data; The output layer takes the output of the anomaly analysis network unit as input, is used to judge whether the WAF log data is abnormal through a binary classifier, and outputs the predicted detection result; Correspondingly, when training the log anomaly analysis model, the sample set also includes real results indicating whether the historical WAF log data is abnormal. A loss function is constructed based on the real results and the predicted detection results, and the model training is performed by minimizing the loss function.

4. According to claim 1, the WAF log anomaly analysis method based on spatiotemporal convolutional network is characterized in that: The log anomaly analysis model includes a spatial feature extraction network unit based on CNN, a time correlation network unit based on LSTM network, and an output layer; The spatial feature extraction network unit is configured with a convolution layer and a pooling layer, which takes a one-dimensional vector of the access IP and the pre-processed log content as input, is used to extract spatial features from the WAF log data through convolution operations and pooling operations, and outputs a one-dimensional vector representing the spatial features; The time correlation network unit takes the concatenation of time and a one-dimensional vector representing spatial characteristics as input, and is used to perform time correlation analysis on WAF log data; The output layer uses the output of the time correlation network unit as input, and is used to perform anomaly analysis on the WAF log data through a multi-class classifier, and output a detection result, which includes anomaly types and probability values ​​corresponding to the anomaly types; Correspondingly, when training the log anomaly analysis model, the sample set also includes anomaly types representing historical WAF log data as real results. A loss function is constructed based on the real results and the predicted detection results, and the model training is performed by minimizing the loss function.

5. A WAF log anomaly analysis method system based on spatiotemporal convolutional network, characterized in that: It includes data collection module, data preprocessing module, sample construction module, model construction module and log anomaly analysis module; The data collection module is used to perform the following: collect historical WAF log data; The data preprocessing module is used to perform the following: take the historical WAF log data as the target data, perform data preprocessing on the target data, and the data format of the preprocessed target data is time + access IP + log content, and the text content in the log content is vectorized to obtain the preprocessed log content; The sample construction module is used to perform the following: for the preprocessed log data, the access IP and the preprocessed log content are spliced ​​into a vector as a vector sample, and a sample set is constructed based on the sample vector and time corresponding to the preprocessed target data; The model building module is used to perform the following: construct a log anomaly analysis model based on a spatiotemporal convolutional neural network, and perform model training on the log anomaly analysis model based on a sample set to obtain a trained log anomaly analysis model, wherein the log anomaly analysis model takes a one-dimensional vector of an access IP and pre-processed log content as input, is used to extract spatial features from WAF log data, output a one-dimensional vector representing spatial features, and takes time and a one-dimensional vector representing spatial features as input, is used to perform anomaly analysis on WAF log data, and outputs a detection result; The log anomaly analysis module cooperates with the data collection module and the data preprocessing module to perform the following: collect the current WAF log data, take the current WAF log data as the target data, perform data preprocessing on the target data, the data format of the preprocessed target data is time + access IP + log content, and the text content in the log content is vectorized to obtain the preprocessed log content, take the vector spliced ​​by the access IP and the preprocessed log content as input, extract the spatial features of the current WAF log data through the trained log anomaly analysis model, output a one-dimensional vector representing the spatial features, and take the splicing of time and the one-dimensional vector representing the spatial features as input, perform anomaly analysis on the current WAF log data through the trained log anomaly analysis model, and output the detection results.

6. According to claim 5, the WAF log anomaly analysis method system based on spatiotemporal convolutional network is characterized in that: The data preprocessing module is used to perform the following data preprocessing on the target data: Perform data cleaning operations on the target data to remove or revise outliers, irrelevant or duplicate data, and incomplete data; Annotate the cleaned target data and modify the target data into the format of time+access IP+log content; For the standardized target data, the text content in the log content is vectorized by word embedding; The vectorized target data is subjected to dimensionality reduction processing, and feature extraction is performed through feature engineering to obtain the preprocessed target data.

7. According to claim 5, the WAF log anomaly analysis method system based on spatiotemporal convolutional network is characterized in that: The log anomaly analysis model includes a spatial feature extraction network unit constructed based on CNN, an anomaly analysis network unit constructed based on LSTM network, and an output layer; The spatial feature extraction network unit is configured with a convolution layer and a pooling layer, which takes a one-dimensional vector of the access IP and the pre-processed log content as input, is used to extract spatial features from the WAF log data through convolution operations and pooling operations, and outputs a one-dimensional vector representing the spatial features; The anomaly analysis network unit takes the concatenation of time and one-dimensional vectors representing spatial characteristics as input, and is used to perform time correlation analysis on WAF log data; The output layer takes the output of the anomaly analysis network unit as input, is used to judge whether the WAF log data is abnormal through a binary classifier, and outputs the predicted detection result; Correspondingly, when training the log anomaly analysis model, the sample set also includes real results indicating whether the historical WAF log data is abnormal. A loss function is constructed based on the real results and the predicted detection results, and the model training is performed by minimizing the loss function.

8. The WAF log anomaly analysis method system based on spatiotemporal convolutional network according to claim 5 is characterized in that: The log anomaly analysis model includes a spatial feature extraction network unit based on CNN, a time correlation network unit based on LSTM network, and an output layer; The spatial feature extraction network unit is configured with a convolution layer and a pooling layer, which takes a one-dimensional vector of the access IP and the pre-processed log content as input, is used to extract spatial features from the WAF log data through convolution operations and pooling operations, and outputs a one-dimensional vector representing the spatial features; The time correlation network unit takes the concatenation of time and a one-dimensional vector representing spatial characteristics as input, and is used to perform time correlation analysis on WAF log data; The output layer uses the output of the time correlation network unit as input, and is used to perform anomaly analysis on the WAF log data through a multi-class classifier, and output a detection result, which includes anomaly types and probability values ​​corresponding to the anomaly types; Correspondingly, when training the log anomaly analysis model, the sample set also includes anomaly types representing historical WAF log data as real results. A loss function is constructed based on the real results and the predicted detection results, and the model training is performed by minimizing the loss function.

Images