Methods, devices, and storage media for detecting multi-level malicious HTTP traffic.

Abstract

This application provides a method, apparatus, and storage medium for detecting multi-layered malicious HTTP traffic. The method includes: reading HTTP protocol data and performing preliminary filtering using a multi-layered Transformer model to obtain initial abnormal data; extracting features using an NLP model and / or anomaly detection model to obtain feature data for each initial abnormal data, wherein the anomaly detection model is a model that extracts parameter features from the initial abnormal data; inputting the feature data into an Attack model to obtain a malicious prediction value; and determining whether the initial abnormal data is malicious attack HTTP data traffic based on the relationship between the malicious prediction value and a preset threshold. This method identifies normal and malicious traffic by constructing a machine learning model and comprehensively utilizes the Transformer model, the anomaly detection model, and the attack feature model to achieve in-depth analysis and accurate classification of HTTP traffic.

Description

Technical Field

[0001] This application relates to the field of network security, and more specifically, to a method for detecting multi-layered malicious HTTP traffic, a device for detecting multi-layered malicious HTTP traffic, a computer-readable storage medium, and an electronic device. Background Technology

[0002] With the rapid development of internet technology, network attack methods have become increasingly diverse and complex. Among these, the detection and defense against malicious HTTP traffic has become a key issue in the field of network security. HTTP, as the most common protocol in web applications, directly affects the reliability of network services and the security of user data. However, traditional signature-matching-based detection methods are proving inadequate in the face of new attack methods, making effective identification and defense difficult. Summary of the Invention

[0003] The main objective of this application is to provide a method, device, computer-readable storage medium, and electronic device for detecting multi-layered malicious HTTP traffic, so as to at least solve the problem that existing HTTP traffic detection methods are difficult to effectively identify new network attack methods and have weak defense capabilities.

[0004] To achieve the above objectives, according to one aspect of this application, a method for detecting multi-layered malicious HTTP traffic is provided, comprising: reading HTTP protocol data and performing preliminary filtering on the HTTP protocol data using a multi-layer transformer model to obtain initial abnormal data; extracting features from the initial abnormal data using an NLP model and / or anomaly detection model to obtain feature data for each of the initial abnormal data, wherein the feature data includes text features and / or parameter features, wherein the NLP model is a model for extracting text features from the initial abnormal data, and the anomaly detection model is a model for extracting parameter features from the initial abnormal data; inputting the feature data into an Attack model to obtain a malicious prediction value corresponding to the initial abnormal data; and determining whether the initial abnormal data is malicious HTTP traffic based on the relationship between the malicious prediction value of the initial abnormal data and a preset threshold, wherein the malicious HTTP traffic is HTTP data traffic of a malicious attack.

[0005] Optionally, before using a multi-layer transformer model to initially filter the HTTP protocol data and obtain initial abnormal data, the method further includes: acquiring historical normal traffic and historical malicious traffic; structuring the historical normal traffic and historical malicious traffic to obtain initial normal traffic and initial malicious traffic in token format; constructing an initial transformer model based on an input embedding mechanism, a multi-layer transformer encoder, and a classification output layer; converting the initial normal traffic and the initial malicious traffic into dense vectors in a high-dimensional vector space and inputting them into the initial transformer model; controlling the initial transformer model to perform deep feature learning through a self-attention mechanism and a multi-layer feedforward network, and using a fully connected layer to map the high-level features of the transformer encoder to predefined normal and malicious category labels to train the initial transformer model and obtain a pre-trained transformer model; and optimizing the pre-trained transformer model based on the cross-entropy loss function to obtain the multi-layer transformer model.

[0006] Optionally, before using an NLP model and / or anomaly detection model to extract features from the initial abnormal data to obtain feature data for each of the initial abnormal data, the method further includes: acquiring historical normal traffic; acquiring historical malicious traffic and attack tags for each of the historical malicious traffic; and training an initial anomaly detection model using a random forest model based on the historical normal traffic, the historical malicious traffic, and the attack tags for each of the historical malicious traffic to obtain the anomaly detection model.

[0007] Optionally, before inputting the feature data into the Attack model to obtain the malicious prediction value corresponding to the initial abnormal data, the method further includes: obtaining a training dataset, the training dataset including feature data of the first historical abnormal data and malicious prediction values ​​corresponding to the feature data of the first historical abnormal data; training an initial Attack model using a random forest model based on the data in the training dataset to obtain a pre-trained Attack model; obtaining a test dataset, and performing parameter tuning on the pre-trained Attack model based on the test dataset to obtain the Attack model, the test dataset including feature data of the second historical abnormal data and malicious prediction values ​​corresponding to the feature data of the second historical abnormal data.

[0008] Optionally, determining whether the initial abnormal data is malicious HTTP traffic based on the relationship between the malicious prediction value of the initial abnormal data and a preset threshold includes: determining the initial abnormal data as malicious HTTP traffic if the malicious prediction value of the initial abnormal data is greater than the preset threshold; and determining the initial abnormal data as non-malicious HTTP traffic if the malicious prediction value of the initial abnormal data is less than or equal to the preset threshold.

[0009] Optionally, after determining whether the initial abnormal data is malicious HTTP traffic based on the relationship between the malicious prediction value of the initial abnormal data and a preset threshold, the method further includes: if the initial abnormal data is malicious HTTP traffic, determining the malicious attack type of the malicious HTTP traffic based on the characteristics of the malicious HTTP traffic; determining the malicious attack type as an attack type label of the malicious HTTP traffic; performing window aggregation within a preset time period based on SIP to obtain an aggregation window, and detecting the number of malicious HTTP traffic with the attack type label set to the preset label within the aggregation window; if the number of malicious HTTP traffic with the attack type label set to the preset label within the aggregation window exceeds a preset number, setting the attack type label of all malicious HTTP traffic within the aggregation window to the preset label to obtain a detection result; and outputting the detection result to Kafka.

[0010] Optionally, reading HTTP protocol data includes: reading initial HTTP protocol data from a Kafka topic; and using the PyTorch deep learning framework to convert the data format of the initial HTTP protocol data into tokens format to obtain the HTTP protocol data.

[0011] According to another aspect of this application, a multi-layered malicious HTTP traffic detection device is provided, comprising: a first processing unit, configured to read HTTP protocol data and perform preliminary filtering on the HTTP protocol data using a multi-layer transformer model to obtain initial abnormal data; an extraction unit, configured to perform feature extraction on the initial abnormal data using an NLP model and / or an outlier detection model to obtain feature data for each of the initial abnormal data, wherein the feature data includes text features and / or parameter features, wherein the NLP model is a model for extracting text features from the initial abnormal data, and the outlier detection model is a model for extracting parameter features from the initial abnormal data; a second processing unit, configured to input the feature data into an Attack model to obtain a malicious prediction value corresponding to the initial abnormal data; and a determination unit, configured to determine whether the initial abnormal data is malicious HTTP traffic based on the relationship between the malicious prediction value of the initial abnormal data and a preset threshold, wherein the malicious HTTP traffic is HTTP data traffic of a malicious attack.

[0012] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform any of the aforementioned multi-level malicious HTTP traffic detection methods.

[0013] According to another aspect of this application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for performing any of the above-described methods for detecting multi-level malicious HTTP traffic.

[0014] Applying the technical solution of this application, the above-mentioned multi-layered malicious HTTP traffic detection method first reads HTTP protocol data and uses a multi-layer transformer model to perform preliminary filtering on the HTTP protocol data to obtain initial abnormal data; then, an NLP model and / or anomaly detection model are used to extract features from the initial abnormal data to obtain feature data for each initial abnormal data. The feature data includes text features and / or parameter features. The NLP model is a model for extracting text features from the initial abnormal data, and the anomaly detection model is a model for extracting parameter features from the initial abnormal data; then, the feature data is input into the Attack model to obtain the malicious prediction value corresponding to the initial abnormal data; finally, based on the relationship between the malicious prediction value of the initial abnormal data and a preset threshold, it is determined whether the initial abnormal data is malicious HTTP traffic, and malicious HTTP traffic is HTTP data traffic of malicious attacks. This method identifies and classifies normal and malicious traffic by constructing an advanced machine learning model. It comprehensively utilizes the Transformer model, outlier detection model, and attack signature model to achieve in-depth analysis and accurate classification of HTTP traffic. This not only improves the accuracy and efficiency of detection but also enhances the adaptability and interpretability of the model, providing a powerful tool for the field of network security. It solves the problem that existing HTTP traffic detection methods are unable to effectively identify new network attack methods and have weak defense capabilities. Attached Figure Description

[0015] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0016] Figure 1 A hardware structure block diagram of a mobile terminal for performing a multi-level malicious HTTP traffic detection method according to an embodiment of this application is shown.

[0017] Figure 2 A flowchart illustrating a method for detecting multi-layered malicious HTTP traffic according to an embodiment of this application is shown.

[0018] Figure 3 An example diagram of HTTP traffic formatting provided according to an embodiment of this application is shown;

[0019] Figure 4 A flowchart illustrating an HTTP traffic detection process according to an embodiment of this application is shown.

[0020] Figure 5 A flowchart illustrating another method for detecting multi-layered malicious HTTP traffic according to an embodiment of this application is shown.

[0021] Figure 6 A flowchart illustrating another method for detecting multi-layered malicious HTTP traffic according to an embodiment of this application is shown.

[0022] Figure 7 A traffic overall architecture diagram according to an embodiment of this application is shown;

[0023] Figure 8 A structural block diagram of a multi-layered malicious HTTP traffic detection device provided according to an embodiment of this application is shown.

[0024] The above figures include the following reference numerals:

[0025] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. Detailed Implementation

[0026] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] As described in the background section, traditional signature-matching-based detection methods are inadequate in the face of new attack methods, making it difficult to effectively identify and defend against them. To address the problem that existing HTTP traffic detection methods are unable to effectively identify new network attack methods and have weak defense capabilities, embodiments of this application provide a multi-layered malicious HTTP traffic detection method, a multi-layered malicious HTTP traffic detection device, a computer-readable storage medium, and an electronic device.

[0030] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0031] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a multi-layered malicious HTTP traffic detection method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0032] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the multi-level malicious HTTP traffic detection method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] This embodiment provides a method for detecting multi-layered malicious HTTP traffic that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] Figure 2 This is a flowchart of a method for detecting multi-layered malicious HTTP traffic according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0035] Step S201: Read the HTTP protocol data and use a multi-layer transformer model to perform preliminary filtering on the HTTP protocol data to obtain initial abnormal data;

[0036] Specifically, the model employs comprehensive HTTP request analysis, covering key components such as URI, request body, and request headers, ensuring full identification of potential attack behaviors. The parallel processing capability of the Transformer model makes it more efficient when processing large-scale datasets.

[0037] Reading HTTP protocol data includes the following steps:

[0038] Step S301: Read initial HTTP protocol data from the Kafka topic;

[0039] Step S302: Using the PyTorch deep learning framework, the data format of the initial HTTP protocol data is converted into the tokens format to obtain the HTTP protocol data.

[0040] Specifically, this extraction of multi-dimensional features enables the model to analyze data from different perspectives, improving the accuracy and robustness of detection.

[0041] Before using a multi-layer transformer model to perform preliminary filtering on the HTTP protocol data to obtain initial abnormal data, the above method also includes the following steps:

[0042] Step S401: Obtain historical normal traffic and historical malicious traffic;

[0043] The specific data sources mainly include attack sample data and normal traffic data. The attack sample data sources are as follows:

[0044] 1. Malicious web traffic detected by CSIC2010;

[0045] 2. Malicious traffic intercepted by WAF (data from GitHub): Some of the samples are not malicious, and the filtering scripts have been included in the project;

[0046] 3. Web risk events on PRS products: Some of these risks are not malicious samples. The types of non-malicious samples are: "weak password", "threat intelligence domain name detection", "Nginx directory traversal response", "Apache Tomcat form authentication username enumeration request", and "threat intelligence - domain name n".

[0047] 4. HTTP protocol generated by vulnerability scanning tools: This mainly refers to scanning tools scanning network traffic;

[0048] 5. Simulate web traffic for various combined attacks.

[0049] The sources of normal traffic data are as follows:

[0050] 1. Normal CSIC2010 web traffic;

[0051] 2. Freebuf's daily traffic: How much of it contains attack traffic that needs to be filtered?

[0052] 3. Capture high volumes of HTTP traffic from web access via the network card;

[0053] 4. Simulate web traffic in various normal scenarios.

[0054] Step S402: The above-mentioned historical normal traffic and the above-mentioned historical malicious traffic are structured to obtain the initial normal traffic and the initial malicious traffic in the format of tokens.

[0055] In some instances, in-depth protocol parsing and structuring of raw HTTP traffic data are performed into a unified and structured format. Specifically, Figure 3 An example diagram of HTTP traffic formatting provided according to an embodiment of this application is shown, such as... Figure 3 As shown, request and response traffic packets are grouped together. A vocabulary specifically tailored for this task is constructed, which implements the mapping transformation from text tokens to unique numerical IDs based on all collected unique tokens.

[0056] Step S403: Construct the initial transformer model based on the input embedding mechanism, multi-layer Transformer encoder, and classification output layer;

[0057] In some instances, an input embedding mechanism is introduced to convert numerical tokens into dense vectors in a high-dimensional vector space. A multi-layer Transformer encoder is constructed, employing self-attention and multi-layer feedforward networks for deep feature learning to efficiently capture and abstract complex sequence patterns. A classification output layer is designed, utilizing fully connected layers to map the high-level features of the Transformer encoder onto predefined normal / malicious category labels, thus completing the final classification task.

[0058] Step S404: Convert the initial normal traffic and the initial malicious traffic into dense vectors in a high-dimensional vector space and input them into the initial transformer model.

[0059] Step S405: Control the initial transformer model to perform deep feature learning through self-attention mechanism and multi-layer feedforward network, and use fully connected layer to map the high-level features of the transformer encoder to predefined normal category labels and malicious category labels, so as to train the initial transformer model and obtain a pre-trained transformer model.

[0060] Step S406: Optimize the pre-trained transformer model based on the cross-entropy loss function to obtain the multi-layer transformer model.

[0061] Specifically, the Transformer model utilizes a self-attention mechanism to efficiently capture long-range dependencies and complex patterns in HTTP traffic. This mechanism can simultaneously focus on multiple features in the traffic, thereby achieving more accurate feature extraction. By extracting features such as parameters, special symbols, and encoding methods, the model can capture subtle anomalies in HTTP requests, enhancing its ability to identify complex attack patterns.

[0062] The Transformer model employs a multi-layered structure, with each layer learning feature representations at different levels. In HTTP protocol traffic, different levels of traffic features (such as URLs, header information, and content) can have varying impacts on traffic analysis. The multi-layered structure allows for the simultaneous learning of these different levels of feature representations, thereby improving the accuracy of traffic analysis. Furthermore, the Transformer model's parallel processing capabilities make it more efficient when handling large-scale datasets.

[0063] Furthermore, the model combines the powerful representational capabilities of deep learning Transformer models with the interpretability of traditional machine learning models, achieving a complementary advantage. Deep learning models can automatically extract features and perform complex pattern recognition, while traditional machine learning models offer better interpretability and stability in certain situations. This fusion strategy allows the model to maintain high accuracy while also possessing a degree of interpretability, helping security analysts understand the model's decision-making process.

[0064] In some instances, the loss function is cross-entropy loss, which serves as a quantitative measure of the difference between the model's predicted probability distribution and the true label.

[0065] The model is trained iteratively on the training and validation sets, and its parameters are regularly evaluated and adjusted to prevent overfitting and ensure the model's generalization ability. The model's effectiveness in identifying malicious HTTP traffic is comprehensively evaluated using multi-dimensional metrics such as precision, recall, and F1 score.

[0066] The dataset was divided into a training set and a test set in an 8:2 ratio. Training was performed on the training set, and validation was performed on the test set. The test results are shown in Table 1.

[0067] Table 1. Test Results

[0068] Model Accuracy illustrate TestCNN 0.987 / Transformer 0.9996 /

[0069] Step S202: Use an NLP model and / or an outlier detection model to extract features from the initial outlier data to obtain feature data for each of the initial outlier data. The feature data includes text features and / or parameter features. The NLP model is a model for extracting text features from the initial outlier data, and the outlier detection model is a model for extracting parameter features from the initial outlier data.

[0070] Specifically, by analyzing network traffic, system behavior, or application behavior, activities that deviate significantly from normal patterns can be identified. Anomaly-based detection schemes can adapt to changes in normal behavior and system and user behavior. Furthermore, this method does not rely on predefined attack patterns, thus enabling the identification of new or unknown threats.

[0071] The model employs comprehensive HTTP request analysis, covering key components such as URI, request body, and request headers, ensuring thorough identification of potential attack behaviors. By extracting features such as parameters, special symbols, and encoding methods, the model can capture subtle anomalies in HTTP requests, enhancing its ability to identify complex attack patterns. This multi-dimensional feature extraction allows the model to analyze data from different perspectives, improving the accuracy and robustness of detection.

[0072] Before using an NLP model and / or anomaly detection model to extract features from the initial abnormal data to obtain the feature data of each of the initial abnormal data, the above method further includes the following steps:

[0073] Step S501: Obtain historical parameters for normal traffic flow;

[0074] Step S502: Obtain historical parameters of malicious traffic and attack tags for each of the aforementioned historical parameters of malicious traffic;

[0075] Step S503: Based on the above-mentioned historical parameters of normal traffic, the above-mentioned historical parameters of malicious traffic, and the attack labels of each of the above-mentioned historical parameters of malicious traffic, an initial outlier detection model is trained using a random forest model to obtain the above-mentioned outlier detection model.

[0076] Specifically, the model is built using a parameterized approach and optimized for different types of attacks, such as SQL injection and XSS attacks. This targeted detection method allows the model to learn specific patterns for particular attack types, thereby improving the recognition rate of these attack types. Parameterized construction also allows the model to flexibly adapt to new attack patterns; by adjusting parameters, it can quickly adapt to new threats, enhancing the model's adaptability and scalability.

[0077] In some instances, outlier models primarily detect the presence of malicious activity in POST (equivalent to the request body), GET requests (equivalent to the content of the URI, such as pageId=93179875) and the value in the header.

[0078] Before training the model, it is necessary to obtain normal and malicious traffic data. Normal traffic mainly comes from two sources: HTTP traffic obtained from the internal network, from which the parameter portion is extracted; and simulated normal data traffic.

[0079] Malicious traffic primarily originates from artificially generated data. First, datasets containing SQL injection, XSS attacks, and command execution techniques are collected online. Then, these attacks are randomly inserted into the generated data.

[0080] Because it is a multi-classification model, malicious traffic needs to be labeled as different categories (SQL injection: 1, XSS attack: 2, command execution: 3).

[0081] Feature extraction: See Table 2 for specific features:

[0082] Table 2. Parametric Model Feature Table

[0083]

[0084]

[0085] In addition, a random forest model was used to train the model. Since the model using the random forest model can achieve an accuracy of 99.9%, no model comparison or parameter tuning was performed.

[0086] Step S203: Input the above feature data into the Attack model to obtain the malicious prediction value corresponding to the above initial abnormal data;

[0087] Specifically, the attack model inputs feature data into its algorithm for analysis and prediction. Based on the training and learning results of the model, it provides corresponding malicious prediction values. This targeted detection method allows the model to learn specific patterns of specific attack types, thereby improving the recognition rate of these attack types.

[0088] Before inputting the aforementioned feature data into the Attack model to obtain the malicious prediction value corresponding to the initial abnormal data, the method further includes the following steps:

[0089] Step S601: Obtain a training dataset, which includes feature data of the first historical abnormal data and malicious prediction values ​​corresponding to the feature data of the first historical abnormal data.

[0090] Step S602: Based on the data in the above training dataset, train the initial Attack model using a random forest model to obtain the pre-trained Attack model.

[0091] Step S603: Obtain the test dataset and perform parameter tuning on the pre-trained Attack model based on the test dataset to obtain the Attack model. The test dataset includes the feature data of the second historical abnormal data and the malicious prediction value corresponding to the feature data of the second historical abnormal data.

[0092] Specifically, these data are used to fine-tune the parameters of the pre-trained Attack model to obtain the final Attack model. Parameter tuning aims to improve the model's performance and accuracy, ensuring that the model can accurately predict malicious behavior from second-generation historical anomalies. Parameterized construction also allows the model to flexibly adapt to new attack patterns; by adjusting parameters, it can quickly adapt to new threats, enhancing the model's adaptability and scalability.

[0093] In some instances, the data used to train the Attack model comes from the same sources as the NLP model. When performing feature extraction for the Attack model, both models are used. These features are categorized according to the HTTP protocol: global features, URI features, header features, and parameter features. Specific features are shown in Tables 3 to 6.

[0094] Table 3. Global Feature Table

[0095]

[0096] Table 4. URI Characteristics Table

[0097]

[0098]

[0099] Table 5. Header Feature Table

[0100]

[0101] Table 6. Parameter Characteristics Table

[0102]

[0103]

[0104] After obtaining the above feature values, the Attack model is trained using a random forest and then optimized.

[0105] The dataset was then divided into training and testing sets in an 8:2 ratio. Training was performed on the training set, and validation was performed on the testing set. The test results are shown in Table 7.

[0106] Table 7. Schematic diagram of test results

[0107] Model Accuracy explain Logistic Regression Model 0.988 / Random Forest Model 0.993 / SVM model 0.977 /

[0108] Therefore, logistic regression and random forest models performed similarly and were both better than SVM, so the random forest model was ultimately chosen.

[0109] Step S204: Based on the relationship between the malicious prediction value of the initial abnormal data and the preset threshold, determine whether the initial abnormal data is malicious HTTP traffic, and whether the malicious HTTP traffic is HTTP data traffic of malicious attack.

[0110] Specifically, by comparing the magnitude of the malicious prediction value with the preset threshold, it is possible to determine whether the initial abnormal data is malicious HTTP traffic, which improves the accuracy and efficiency of detection and enhances the adaptability and interpretability of the model.

[0111] The process of determining whether the initial abnormal data is malicious HTTP traffic, based on the relationship between the predicted malicious value of the initial abnormal data and a preset threshold, includes the following steps:

[0112] Step S701: If the malicious prediction value of the initial abnormal data is greater than the preset threshold, the initial abnormal data is determined to be malicious HTTP traffic.

[0113] Step S702: If the malicious prediction value of the initial abnormal data is less than or equal to the preset threshold, the initial abnormal data is determined to be non-malicious HTTP traffic.

[0114] Specifically, after identifying malicious HTTP traffic, the malicious attack type to which the traffic belongs is deduced. Attack tags include command attacks, SQL injection, XSS injection, Java attacks, parameters, URLs, headers, and vulnerability scanning, in preparation for subsequent scan type attack tag confirmation.

[0115] After determining whether the initial abnormal data is malicious HTTP traffic based on the relationship between the malicious prediction value of the initial abnormal data and the preset threshold, the method further includes the following steps:

[0116] Step S801: If the initial abnormal data is malicious HTTP traffic, determine the malicious attack type of the malicious HTTP traffic based on the characteristics of the malicious HTTP traffic.

[0117] Step S802: Determine the above-mentioned malicious attack type as the attack type label of the above-mentioned malicious HTTP traffic;

[0118] Step S803: Perform window aggregation within a preset time period based on SIP to obtain an aggregated window, and detect the number of malicious HTTP traffic with the attack type label set to the preset label within the aggregated window.

[0119] Step S804: If the number of malicious HTTP traffic with the attack type label set to the preset label in the above aggregation window exceeds the preset number, set the attack type label of all malicious HTTP traffic in the above aggregation window to the preset label to obtain the detection result.

[0120] Step S805: Output the above detection results to Kafka.

[0121] Specifically, this method can quickly and accurately identify the attack type of malicious HTTP traffic, and effectively filter out genuine malicious attack traffic through window aggregation and quantity judgment, improving the accuracy and efficiency of detection. Furthermore, outputting the detection results to Kafka facilitates subsequent processing and analysis.

[0122] Figure 4 This diagram illustrates an HTTP traffic detection flowchart according to an embodiment of this application. First, the HTTP protocol is read from Kafka. Then, detection begins by loading the param model, attack model, and NLP model. Combining whitelist and blacklist strategies, the attack model is extracted, the attack model is predicted, and a label is determined. Afterward, a scan is performed, and malicious HTTP traffic is output to Kafka.

[0123] Among them, the Transformer model demonstrates superior performance in processing sequence data due to its self-attention mechanism, while the outlier detection model quickly identifies abnormal behavior of parameters in requests. The attack signature model extracts key features from the HTTP protocol and combines them with expert knowledge for comprehensive analysis to determine the attack type and anomaly location.

[0124] For model training, a large amount of real-world HTTP traffic data was used, including both normal and malicious traffic samples, to ensure the model's generalization ability and accuracy. The data sources were diverse, covering public datasets, internal company data, and simulated attack traffic, ensuring data diversity and representativeness. Through in-depth preprocessing and feature engineering of this data, a rich feature set was constructed, providing a solid foundation for model training.

[0125] Furthermore, this embodiment pays special attention to model evaluation and optimization, employing the cross-entropy loss function and multi-dimensional evaluation metrics such as precision, recall, and F1 score to comprehensively assess model performance. Through training and validation cycles, model parameters are continuously adjusted to prevent overfitting, ensuring the model's robustness and generalization ability.

[0126] Ultimately, the malicious HTTP traffic detection algorithm proposed in this embodiment is not only theoretically innovative, but also demonstrates high detection accuracy and efficiency in practical applications, providing an effective technical means for network security protection.

[0127] Figure 5 This is a flowchart of a method for detecting multi-layered malicious HTTP traffic according to an embodiment of this application. Figure 5 As shown, this method first acquires normal and malicious traffic, processes the data, extracts features from the param model, and then trains the param model to obtain the param model. Next, it processes the malicious HTTP samples. Then, it processes the normal HTTP samples, performs word segmentation and word vector generation, and trains the NLP model to obtain the NLP model. Finally, it preloads the NLP model and the param model, combines them with the data from the Attack model, extracts features from the Attack model, and finally obtains the Attack model.

[0128] Among them, the param model is the outlier detection model, and the NLP model is the transformer model.

[0129] The multi-layered malicious HTTP traffic detection method of this application first reads HTTP protocol data and uses a multi-layer transformer model to perform preliminary filtering on the HTTP protocol data to obtain initial abnormal data; then, it uses an NLP model and / or anomaly detection model to extract features from the initial abnormal data to obtain feature data for each initial abnormal data. The feature data includes text features and / or parameter features. The NLP model is a model for extracting text features from the initial abnormal data, and the anomaly detection model is a model for extracting parameter features from the initial abnormal data; then, the feature data is input into the Attack model to obtain the malicious prediction value corresponding to the initial abnormal data; finally, based on the relationship between the malicious prediction value of the initial abnormal data and a preset threshold, it is determined whether the initial abnormal data is malicious HTTP traffic, and malicious HTTP traffic is HTTP data traffic of malicious attacks. This method identifies and classifies normal and malicious traffic by constructing an advanced machine learning model. It comprehensively utilizes the Transformer model, outlier detection model, and attack signature model to achieve in-depth analysis and accurate classification of HTTP traffic. This not only improves the accuracy and efficiency of detection but also enhances the adaptability and interpretability of the model, providing a powerful tool for the field of network security. It solves the problem that existing HTTP traffic detection methods are unable to effectively identify new network attack methods and have weak defense capabilities.

[0130] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the multi-level malicious HTTP traffic detection method of this application will be described in detail below with reference to specific embodiments.

[0131] This embodiment relates to a specific method for detecting multi-layered malicious HTTP traffic, such as... Figure 6 As shown, it includes the following steps:

[0132] Step S1: Start detection. Read HTTP protocol data from the Kafka topic.

[0133] Step S2: Load the transformer model and use pytorch / torch to further convert the HTTP traffic into tokens.

[0134] Step S3: Whitelist + Blacklist Strategy. First, determine if the whitelist strategy is applicable:

[0135] a) If the source IP and destination IP are in the whitelist, filter them out and do not perform detection;

[0136] b) If the URL is filtered in the whitelist;

[0137] c) GET requests with a status code of 200 are filtered out directly;

[0138] d) Filter out port 9200, because Elasticsearch requests are generally internal network requests.

[0139] Blacklist strategy:

[0140] a) First check if the NLP model has been loaded. If not, load the NLP model.

[0141] b) Combine the uri, request_body, and request_header_values ​​into your own string and use an NLP model to detect HTTP. If the predicted value is less than 0.6, then filter out this information.

[0142] Step S4: Load the outlier detection model and the Attack model.

[0143] Step S5: Extract the features of the Attack model sequentially according to the feature tables (Tables 2 to 5) in the second part of the specific Attack model scheme.

[0144] Step S6: Use the Attack model to predict the features extracted in the previous step. If the predicted value is >0.6, it is considered malicious traffic; otherwise, it is considered normal traffic.

[0145] Step S7: Once the traffic is determined to be malicious HTTP traffic, the characteristics obtained in Step 5 are used to further deduce the type of malicious attack to which the traffic belongs. Attack tags include command attacks, SQL injection, XSS injection, Java attacks, parameters, URL, header, and vulnerability scanning.

[0146] Step S8: Scan type attack label confirmation: Perform 1-minute window aggregation based on SIP. If the traffic containing the vulnerability scan label exceeds the threshold within the aggregation window, then add the "vulnerability scan" label to all malicious traffic in this aggregation window.

[0147] Step S9: Output the detection results to Kafka.

[0148] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0149] Figure 7This diagram illustrates a traffic architecture provided by an embodiment of this application, comprising system implementation and key technologies. The system implementation involves traffic collection through interface, traffic splitting, aggregation, and filtering steps. The collected traffic is input into a traffic identification engine and processed by a traffic security analysis engine, an HTTP traffic restoration engine, and an HTTPS traffic decryption engine. HTTP traffic is input into the HTTP traffic restoration engine, and HTTPS traffic is input into the HTTPS traffic decryption engine, ultimately resulting in blacklists and whitelists. The key technologies of the system implementation are a multi-layered traffic identification model, decompression acceleration technology, and high-speed regular expression matching technology.

[0150] This application also provides a device for detecting multi-level malicious HTTP traffic. It should be noted that this device can be used to execute the method for detecting multi-level malicious HTTP traffic provided in this application. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0151] The following describes the multi-layered malicious HTTP traffic detection device provided in the embodiments of this application.

[0152] Figure 8 This is a schematic diagram of a multi-layered malicious HTTP traffic detection device according to an embodiment of this application. Figure 8 As shown, the device includes: a first processing unit 10, an extraction unit 20, a second processing unit 30, and a determination unit 40. The first processing unit 10 is used to read HTTP protocol data and perform preliminary filtering on the HTTP protocol data using a multi-layer transformer model to obtain initial abnormal data. The extraction unit 20 is used to perform feature extraction on the initial abnormal data using an NLP model and / or anomaly detection model to obtain feature data for each of the initial abnormal data. The feature data includes text features and / or parameter features. The NLP model is a model for extracting text features from the initial abnormal data, and the anomaly detection model is a model for extracting parameter features from the initial abnormal data. The second processing unit 30 is used to input the feature data into an Attack model to obtain a malicious prediction value corresponding to the initial abnormal data. The determination unit 40 is used to determine whether the initial abnormal data is malicious HTTP traffic based on the relationship between the malicious prediction value of the initial abnormal data and a preset threshold. The malicious HTTP traffic is HTTP data traffic of a malicious attack.

[0153] The multi-layered malicious HTTP traffic detection device of this application includes: a first processing unit, an extraction unit, a second processing unit, and a determination unit. The first processing unit is used to read HTTP protocol data and perform preliminary filtering on the HTTP protocol data using a multi-layer transformer model to obtain initial abnormal data. The extraction unit is used to extract features from the initial abnormal data using an NLP model and / or anomaly detection model to obtain feature data for each initial abnormal data. The feature data includes text features and / or parameter features. The NLP model is a model for extracting text features from the initial abnormal data, and the anomaly detection model is a model for extracting parameter features from the initial abnormal data. The second processing unit is used to input the feature data into an Attack model to obtain a malicious prediction value corresponding to the initial abnormal data. The determination unit is used to determine whether the initial abnormal data is malicious HTTP traffic based on the relationship between the malicious prediction value of the initial abnormal data and a preset threshold. Malicious HTTP traffic is HTTP data traffic of malicious attacks. This device identifies and classifies normal and malicious traffic by constructing an advanced machine learning model. It comprehensively utilizes the Transformer model, outlier detection model, and attack signature model to achieve in-depth analysis and accurate classification of HTTP traffic. This not only improves the accuracy and efficiency of detection but also enhances the adaptability and interpretability of the model, providing a powerful tool for the field of network security. It solves the problem that existing HTTP traffic detection methods are unable to effectively identify new network attack methods and have weak defense capabilities.

[0154] In some instances, before initially filtering the HTTP protocol data using a multi-layer Transformer model to obtain initial abnormal data, the device further includes a first acquisition module, a first processing module, a first construction module, a first conversion membrane, a first control module, and a first optimization module. The first acquisition module acquires historical normal traffic and historical malicious traffic; the first processing module structures the historical normal traffic and historical malicious traffic to obtain initial normal traffic and initial malicious traffic in token format; the first construction module constructs an initial Transformer model based on an input embedding mechanism, a multi-layer Transformer encoder, and a classification output layer; and the first conversion membrane is used to process the initial normal traffic and malicious traffic into a structured format. The traffic and the initial malicious traffic are converted into dense vectors in a high-dimensional vector space and input into the initial transformer model. The first control module controls the initial transformer model to perform deep feature learning through a self-attention mechanism and a multi-layer feedforward network, and uses a fully connected layer to map the high-level features of the transformer encoder onto predefined normal and malicious class labels to train the initial transformer model, resulting in a pre-trained transformer model. The first optimization module optimizes the pre-trained transformer model based on the cross-entropy loss function, resulting in the multi-layer transformer model. The Transformer model utilizes a self-attention mechanism, which can efficiently capture long-distance dependencies and complex patterns in HTTP traffic. By extracting features such as parameters, special symbols, and encoding methods, the model can capture subtle anomalies in HTTP requests, enhancing its ability to identify complex attack patterns.

[0155] In some instances, before using NLP models and / or outlier detection models to extract features from the initial outlier data and obtain the feature data of each initial outlier data, the device further includes a second acquisition module, a third acquisition module, and a second processing module. The second acquisition module is used to acquire historical normal traffic; the third acquisition module is used to acquire historical malicious traffic and attack tags for each of the aforementioned historical malicious traffic; the second processing module is used to train an initial outlier detection model using a random forest model based on the aforementioned historical normal traffic, the aforementioned historical malicious traffic, and the attack tags for each of the aforementioned historical malicious traffic, to obtain the aforementioned outlier detection model. The model is constructed using a parameterized approach and is specifically optimized for different types of attack behaviors such as SQL injection and XSS attacks. A random forest model is used to train the model; since the accuracy of the random forest model can reach 99.9%, no model comparison or parameter tuning is performed.

[0156] In some instances, before inputting the aforementioned feature data into the Attack model to obtain the malicious prediction values ​​corresponding to the initial anomalous data, the device further includes a fourth acquisition module, a third processing module, and a fifth acquisition module. The fourth acquisition module is used to acquire a training dataset, which includes feature data of the first historical anomalous data and the malicious prediction values ​​corresponding to the feature data of the first historical anomalous data. The third processing module is used to train an initial Attack model using a random forest model based on the data in the training dataset, obtaining a pre-trained Attack model. The fifth acquisition module is used to acquire a test dataset and perform parameter tuning on the pre-trained Attack model based on the test dataset, obtaining the Attack model. The test dataset includes feature data of the second historical anomalous data and the malicious prediction values ​​corresponding to the feature data of the second historical anomalous data. Parameterized construction also allows the model to flexibly adapt to new attack patterns. By adjusting parameters, it can quickly adapt to new threats, enhancing the model's adaptability and scalability.

[0157] In some instances, the determining unit includes a first determining module and a second determining module. The first determining module is used to determine that the initial abnormal data is malicious HTTP traffic if the malicious prediction value of the initial abnormal data is greater than the preset threshold. The second determining module is used to determine that the initial abnormal data is non-malicious HTTP traffic if the malicious prediction value of the initial abnormal data is less than or equal to the preset threshold. By comparing the relationship between the malicious prediction value and the preset threshold, it is possible to determine whether the initial abnormal data is malicious HTTP traffic, thereby improving the accuracy and efficiency of detection and enhancing the adaptability and interpretability of the model.

[0158] In some instances, after determining whether the initial abnormal data is malicious HTTP traffic based on the relationship between the predicted malicious value of the initial abnormal data and a preset threshold, the device further includes a third determining module, a fourth determining module, a fourth processing module, a fifth processing module, and a first output module. The third determining module is used to determine the malicious attack type of the malicious HTTP traffic based on the characteristics of the malicious HTTP traffic when the initial abnormal data is malicious HTTP traffic. The fourth determining module is used to determine the malicious attack type as an attack type label for the malicious HTTP traffic. The fourth processing module is used to perform window aggregation within a preset time period based on SIP to obtain an aggregation window and detect the number of malicious HTTP traffic with the attack type label set to the preset label within the aggregation window. The fifth processing module is used to set the attack type label of all malicious HTTP traffic within the aggregation window to the preset label when the number of malicious HTTP traffic with the attack type label set to the preset label within the aggregation window exceeds a preset number, thereby obtaining a detection result. The first output module is used to output the detection result to Kafka. After identifying malicious HTTP traffic, we work backward to determine the type of malicious attack to which the traffic belongs. Attack tags include command attacks, SQL injection, XSS injection, Java attacks, parameters, URLs, headers, and vulnerability scanning, which prepares us for subsequent scan type attack tag confirmation.

[0159] In some examples, the first processing unit includes a first reading module and a second transformation module. The first reading module reads initial HTTP protocol data from a Kafka topic; the second transformation module uses the PyTorch deep learning framework to convert the initial HTTP protocol data into tokens format, thus obtaining the HTTP protocol data. This multi-dimensional feature extraction enables the model to analyze data from different perspectives, improving the accuracy and robustness of detection.

[0160] The aforementioned multi-layered malicious HTTP traffic detection device includes a processor and a memory. The first processing unit and other components are stored as program units in the memory, and the processor executes these program units to achieve the corresponding functions. All of the above modules reside in the same processor; alternatively, the modules may be located in different processors in any combination.

[0161] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can address the limitations of existing HTTP traffic detection methods in effectively identifying new network attack techniques and their weak defense capabilities.

[0162] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0163] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the multi-level malicious HTTP traffic detection method.

[0164] This invention provides a processor for running a program, wherein the program executes the multi-level malicious HTTP traffic detection method.

[0165] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of a method for detecting at least multi-level malicious HTTP traffic.

[0166] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform the steps of initializing a detection method for at least multi-level malicious HTTP traffic.

[0167] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0168] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0169] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0170] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0172] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0173] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0174] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0175] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0176] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0177] 1) The multi-layered malicious HTTP traffic detection method of this application first reads HTTP protocol data and uses a multi-layer transformer model to perform preliminary filtering on the HTTP protocol data to obtain initial abnormal data; then, an NLP model and / or anomaly detection model are used to extract features from the initial abnormal data to obtain feature data for each initial abnormal data. The feature data includes text features and / or parameter features. The NLP model is a model for extracting text features from the initial abnormal data, and the anomaly detection model is a model for extracting parameter features from the initial abnormal data; then, the feature data is input into the Attack model to obtain the malicious prediction value corresponding to the initial abnormal data; finally, based on the relationship between the malicious prediction value of the initial abnormal data and a preset threshold, it is determined whether the initial abnormal data is malicious HTTP traffic, and malicious HTTP traffic is HTTP data traffic of malicious attacks. This method identifies and classifies normal and malicious traffic by constructing an advanced machine learning model. It comprehensively utilizes the Transformer model, outlier detection model, and attack signature model to achieve in-depth analysis and accurate classification of HTTP traffic. This not only improves the accuracy and efficiency of detection but also enhances the adaptability and interpretability of the model, providing a powerful tool for the field of network security. It solves the problem that existing HTTP traffic detection methods are unable to effectively identify new network attack methods and have weak defense capabilities.

[0178] 2) The multi-layered malicious HTTP traffic detection device of this application includes: a first processing unit, an extraction unit, a second processing unit, and a determination unit. The first processing unit is used to read HTTP protocol data and perform preliminary filtering on the HTTP protocol data using a multi-layer transformer model to obtain initial abnormal data. The extraction unit is used to extract features from the initial abnormal data using an NLP model and / or anomaly detection model to obtain feature data for each initial abnormal data. The feature data includes text features and / or parameter features. The NLP model is a model for extracting text features from the initial abnormal data, and the anomaly detection model is a model for extracting parameter features from the initial abnormal data. The second processing unit is used to input the feature data into an Attack model to obtain the malicious prediction value corresponding to the initial abnormal data. The determination unit is used to determine whether the initial abnormal data is malicious HTTP traffic based on the relationship between the malicious prediction value of the initial abnormal data and a preset threshold. Malicious HTTP traffic is HTTP data traffic of malicious attacks. This device identifies and classifies normal and malicious traffic by constructing an advanced machine learning model. It comprehensively utilizes the Transformer model, outlier detection model, and attack signature model to achieve in-depth analysis and accurate classification of HTTP traffic. This not only improves the accuracy and efficiency of detection but also enhances the adaptability and interpretability of the model, providing a powerful tool for the field of network security. It solves the problem that existing HTTP traffic detection methods are unable to effectively identify new network attack methods and have weak defense capabilities.

[0179] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for detecting multi-layered malicious HTTP traffic, characterized in that, include: Read HTTP protocol data and use a multi-layer transformer model to perform preliminary filtering on the HTTP protocol data to obtain initial abnormal data; The initial abnormal data are feature extracted using an NLP model and / or an outlier detection model to obtain feature data for each initial abnormal data. The feature data includes text features and / or parameter features. The NLP model is a model for extracting text features from the initial abnormal data, and the outlier detection model is a model for extracting parameter features from the initial abnormal data. The feature data is input into the Attack model to obtain the malicious prediction value corresponding to the initial abnormal data; Based on the relationship between the malicious prediction value of the initial abnormal data and the preset threshold, it is determined whether the initial abnormal data is malicious HTTP traffic, and the malicious HTTP traffic is HTTP data traffic of malicious attacks. After determining whether the initial abnormal data is malicious HTTP traffic based on the relationship between the malicious prediction value of the initial abnormal data and a preset threshold, the method further includes: In the case where the initial abnormal data is malicious HTTP traffic, the malicious attack type of the malicious HTTP traffic is determined based on the characteristics of the malicious HTTP traffic; The malicious attack type is identified as the attack type tag of the malicious HTTP traffic; Based on SIP, windows are aggregated within a preset time period to obtain an aggregated window, and the number of malicious HTTP traffic with the attack type label set to a preset label within the aggregated window is detected. If the number of malicious HTTP traffic with the attack type label set to the preset label in the aggregation window exceeds the preset number, the attack type label of all malicious HTTP traffic in the aggregation window is set to the preset label to obtain the detection result. The detection results are output to Kafka.

2. The method according to claim 1, characterized in that, Before using a multi-layer transformer model to perform preliminary filtering of the HTTP protocol data to obtain initial abnormal data, the method further includes: Obtain historical normal traffic and historical malicious traffic; The historical normal traffic and the historical malicious traffic are structured to obtain the initial normal traffic and the initial malicious traffic in the format of tokens; An initial transformer model is constructed based on an input embedding mechanism, a multi-layer Transformer encoder, and a classification output layer. The initial normal traffic and the initial malicious traffic are converted into dense vectors in a high-dimensional vector space and input into the initial transformer model; The initial transformer model is controlled to perform deep feature learning through a self-attention mechanism and a multi-layer feedforward network, and the high-level features of the transformer encoder are mapped to predefined normal and malicious class labels using a fully connected layer to train the initial transformer model and obtain a pre-trained transformer model. The pre-trained transformer model is optimized based on the cross-entropy loss function to obtain the multi-layer transformer model.

3. The method according to claim 1, characterized in that, Before using an NLP model and / or anomaly detection model to extract features from the initial anomalous data to obtain feature data for each of the initial anomalous data, the method further includes: Obtain historical normal traffic parameters; Obtain historical parameters of malicious traffic and attack tags for each of the historical parameters of malicious traffic; Based on the historical parameters of normal traffic, the historical parameters of malicious traffic, and the attack tags of each of the historical parameters of malicious traffic, an initial outlier detection model is trained using a random forest model to obtain the outlier detection model.

4. The method according to claim 1, characterized in that, Before inputting the feature data into the Attack model to obtain the malicious prediction value corresponding to the initial abnormal data, the method further includes: Obtain a training dataset, which includes feature data of the first historical abnormal data and malicious prediction values ​​corresponding to the feature data of the first historical abnormal data. Based on the data in the training dataset, an initial Attack model is trained using a random forest model to obtain a pre-trained Attack model. Obtain a test dataset and perform parameter tuning on the pre-trained Attack model based on the test dataset to obtain the Attack model. The test dataset includes feature data of the second historical abnormal data and malicious prediction values ​​corresponding to the feature data of the second historical abnormal data.

5. The method according to claim 1, characterized in that, Determining whether the initial abnormal data is malicious HTTP traffic based on the relationship between the predicted malicious value of the initial abnormal data and a preset threshold includes: If the malicious prediction value of the initial abnormal data is greater than the preset threshold, the initial abnormal data is determined to be malicious HTTP traffic. If the malicious prediction value of the initial abnormal data is less than or equal to the preset threshold, the initial abnormal data is determined to be non-malicious HTTP traffic.

6. The method according to any one of claims 1 to 5, characterized in that, Read HTTP protocol data, including: Read initial HTTP protocol data from a Kafka topic; Using the PyTorch deep learning framework, the initial HTTP protocol data is converted into tokens format to obtain the HTTP protocol data.

7. A device for detecting multi-level malicious HTTP traffic, characterized in that, include: The first processing unit is used to read HTTP protocol data and perform preliminary filtering on the HTTP protocol data using a multi-layer transformer model to obtain initial abnormal data. An extraction unit is used to extract features from the initial abnormal data using an NLP model and / or an outlier detection model to obtain feature data for each of the initial abnormal data. The feature data includes text features and / or parameter features. The NLP model is a model for extracting text features from the initial abnormal data, and the outlier detection model is a model for extracting parameter features from the initial abnormal data. The second processing unit is used to input the feature data into the Attack model to obtain the malicious prediction value corresponding to the initial abnormal data; The determining unit is used to determine whether the initial abnormal data is malicious HTTP traffic based on the relationship between the malicious prediction value of the initial abnormal data and a preset threshold, wherein the malicious HTTP traffic is HTTP data traffic of a malicious attack. The above-mentioned device also includes a third determining module, a fourth determining module, a fourth processing module, a fifth processing module, and a first output module. The third determining module is used to determine whether the initial abnormal data is malicious HTTP traffic based on the relationship between the malicious prediction value of the initial abnormal data and a preset threshold. If the initial abnormal data is malicious HTTP traffic, the module determines the malicious attack type of the malicious HTTP traffic based on the characteristics of the malicious HTTP traffic. The fourth determining module is used to determine the malicious attack type as the attack type label of the malicious HTTP traffic; the fourth processing module is used to perform window aggregation within a preset time period based on SIP to obtain an aggregation window, and detect the number of malicious HTTP traffic with the attack type label set to the preset label within the aggregation window; the fifth processing module is used to set the attack type label of all malicious HTTP traffic within the aggregation window to the preset label if the number of malicious HTTP traffic with the attack type label set to the preset label within the aggregation window exceeds a preset number, thereby obtaining a detection result; the first output module is used to output the detection result to Kafka.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the multi-level malicious HTTP traffic detection method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a method for performing a multi-level malicious HTTP traffic detection method as described in any one of claims 1 to 6.

Images