Multi-level malicious HTTP traffic detection method and device and storage medium

Through multi-layer Transformer model and feature extraction technology, combined with NLP and outlier detection models, identifying and classifying HTTP traffic, the problem that traditional detection methods are difficult to identify new network attacks is solved, and more efficient malicious traffic detection and defense is achieved.

CN119030776AActive Publication Date: 2024-11-26ELECTRIC POWER RES INST CHINA SOUTHERN POWER GRID CO LTD

Patent Information

Application Number
CN202411194154.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-11-26
Estimated Expiration
2044-08-28

AI Technical Summary

Technical Problem

Traditional HTTP traffic detection methods based on signature matching are difficult to effectively identify new cyber attack methods, and have weak defenses, so they cannot effectively identify and defend complex malicious HTTP traffic.

Method used

The multi-layer Transformer model is used to perform preliminary filtering of HTTP protocol data, combine the NLP model and outlier detection model to extract feature data, and malicious prediction is made through the Attack model, and determine whether the data is malicious HTTP traffic based on the preset threshold.

Benefits of technology

It improves the accuracy and efficiency of HTTP traffic detection, enhances the adaptability and interpretability of the model, and provides more powerful network security tools that can effectively identify and defend against new types of cyberattacks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119030776A_ABST
    Figure CN119030776A_ABST
Patent Text Reader

Abstract

The invention provides a multi-layer malicious HTTP traffic detection method and device and a storage medium, and the method comprises the steps: reading HTTP protocol data, and carrying out the preliminary filtering through employing a multi-layer transformer model, and obtaining initial abnormal data; performing feature extraction by adopting an NLP model and / or an abnormal value detection model to obtain feature data of each initial abnormal data, the abnormal value detection model being a model for extracting parameter features in the initial abnormal data; inputting the feature data into an Attack model to obtain a malicious predicted value; according to the size relation between the malicious predicted value and the preset threshold value, whether the initial abnormal data is the HTTP data traffic of the hostile attack is determined, according to the method, the normal traffic and the malicious traffic are recognized by constructing a machine learning model, and deep analysis and accurate classification of the HTTP traffic are achieved by comprehensively using a Transform model, an abnormal value detection model and an attack feature model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of network security, and in particular to a method for detecting multi-level malicious HTTP traffic, a device for detecting multi-level malicious HTTP traffic, a computer-readable storage medium, and an electronic device. Background Art

[0002] With the rapid development of Internet technology, network attack methods are becoming increasingly diverse and complex. Among them, the detection and defense of malicious HTTP traffic has become a key issue in the field of network security. As the most common protocol in Web applications, the security of HTTP is directly related to the reliability of network services and the security of user data. However, traditional detection methods based on signature matching are powerless in the face of new attack methods and are difficult to effectively identify and defend. Summary of the invention

[0003] The main purpose of this application is to provide a multi-level malicious HTTP traffic detection method, a multi-level malicious HTTP traffic detection device, a computer-readable storage medium and an electronic device, so as to at least solve the problem that the HTTP traffic detection method in the prior art is difficult to effectively identify new network attack methods and has weak defense capabilities.

[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a multi-level malicious HTTP traffic detection method is provided, comprising: reading HTTP protocol data, and using a multi-layer transformer model to preliminarily filter the HTTP protocol data to obtain initial abnormal data; using an NLP model and / or an outlier detection model to extract features of the initial abnormal data to obtain feature data of each of the initial abnormal data, the feature data including text features and / or parameter features, the NLP model is a model for extracting text features in the initial abnormal data, and the outlier detection model is a model for extracting parameter features in the initial abnormal data; inputting the feature data into an Attack model to obtain a malicious prediction value corresponding to the initial abnormal data; and determining whether the initial abnormal data is malicious HTTP traffic based on the size relationship between the malicious prediction value of the initial abnormal data and a preset threshold, wherein the malicious HTTP traffic is HTTP data traffic of a malicious attack.

[0005] Optionally, before using a multi-layer transformer model to perform preliminary filtering on the HTTP protocol data to obtain initial abnormal data, the method also includes: obtaining historical normal traffic and historical malicious traffic; structuring the historical normal traffic and the historical malicious traffic to obtain initial normal traffic and initial malicious traffic in the format of tokens; constructing an initial transformer model based on an input embedding mechanism, a multi-layer transformer encoder and a classification output layer; converting the initial normal traffic and the initial malicious traffic into dense vectors in a high-dimensional vector space, and inputting them into the initial transformer model; controlling the initial transformer model to perform deep feature learning through a self-attention mechanism and a multi-layer feedforward network, and using a fully connected layer to map the high-level features of the transformer encoder to predefined normal category labels and malicious category labels to train the initial transformer model to obtain a pre-trained transformer model; optimizing the pre-trained transformer model based on a cross-entropy loss function to obtain the multi-layer transformer model.

[0006] Optionally, before using the NLP model and / or the outlier detection model to extract features from the initial anomaly data to obtain feature data of each of the initial anomaly data, the method also includes: obtaining historical parameter normal traffic; obtaining historical parameter malicious traffic and attack labels of each of the historical parameter malicious traffic; based on the historical parameter normal traffic, the historical parameter malicious traffic and the attack labels of each of the historical parameter malicious traffic, using a random forest model to train an initial outlier detection model to obtain the outlier detection model.

[0007] Optionally, before inputting the feature data into the Attack model to obtain the malicious prediction value corresponding to the initial abnormal data, the method also includes: obtaining a training data set, the training data set including feature data of first historical abnormal data and malicious prediction values ​​corresponding to the feature data of the first historical abnormal data; training an initial Attack model using a random forest model based on data in the training data set to obtain a pre-trained Attack model; obtaining a test data set, and tuning parameters of the pre-trained Attack model based on the test data set to obtain the Attack model, the test data set including feature data of second historical abnormal data and malicious prediction values ​​corresponding to the feature data of the second historical abnormal data.

[0008] Optionally, determining whether the initial abnormal data is malicious HTTP traffic is based on a size relationship between the malicious prediction value of the initial abnormal data and a preset threshold, including: when the malicious prediction value of the initial abnormal data is greater than the preset threshold, determining that the initial abnormal data is malicious HTTP traffic; when the malicious prediction value of the initial abnormal data is less than or equal to the preset threshold, determining that the initial abnormal data is non-malicious HTTP traffic.

[0009] Optionally, after determining whether the initial abnormal data is malicious HTTP traffic based on the size relationship between the malicious prediction value of the initial abnormal data and a preset threshold, the method further includes: when the initial abnormal data is the malicious HTTP traffic, determining the malicious attack type of the malicious HTTP traffic based on the characteristics of the malicious HTTP traffic; determining the malicious attack type as the attack type label of the malicious HTTP traffic; performing window aggregation within a preset time period according to SIP to obtain an aggregation window, and detecting the number of malicious HTTP traffic whose attack type label is a preset label within the aggregation window; when the number of malicious HTTP traffic whose attack type label is a preset label within the aggregation window exceeds a preset number, setting the attack type labels of all malicious HTTP traffic within the aggregation window to the preset label to obtain a detection result; and outputting the detection result to Kafka.

[0010] Optionally, reading HTTP protocol data includes: reading initial HTTP protocol data from a kafka topic; using a pytorch deep learning framework to convert a data format of the initial HTTP protocol data into a tokens format to obtain the HTTP protocol data.

[0011] According to another aspect of the present application, a multi-level malicious HTTP traffic detection device is provided, comprising: a first processing unit, configured to read HTTP protocol data, and use a multi-layer transformer model to preliminarily filter the HTTP protocol data to obtain initial abnormal data; an extraction unit, configured to use an NLP model and / or an outlier detection model to perform feature extraction on the initial abnormal data to obtain feature data of each of the initial abnormal data, wherein the feature data comprises text features and / or parameter features, the NLP model is a model for extracting text features from the initial abnormal data, and the outlier detection model is a model for extracting parameter features from the initial abnormal data; a second processing unit, configured to input the feature data into an Attack model to obtain a malicious prediction value corresponding to the initial abnormal data; and a determination unit, configured to determine whether the initial abnormal data is malicious HTTP traffic according to a size relationship between the malicious prediction value of the initial abnormal data and a preset threshold, wherein the malicious HTTP traffic is HTTP data traffic of a malicious attack.

[0012] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the multi-level malicious HTTP traffic detection methods.

[0013] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any of the above-mentioned multi-level malicious HTTP traffic detection methods.

[0014] Applying the technical solution of the present application, the above-mentioned multi-level malicious HTTP traffic detection method first reads HTTP protocol data, and uses a multi-layer transformer model to preliminarily filter the HTTP protocol data to obtain initial abnormal data; then uses an NLP model and / or an outlier detection model to extract features of the initial abnormal data to obtain feature data of each initial abnormal data, the feature data includes text features and / or parameter features, the NLP model is a model for extracting text features in the initial abnormal data, and the outlier detection model is a model for extracting parameter features in the initial abnormal data; then the feature data is input into the Attack model to obtain a malicious prediction value corresponding to the initial abnormal data; finally, according to the size relationship between the malicious prediction value of the initial abnormal data and a preset threshold, it is determined whether the initial abnormal data is malicious HTTP traffic, and the malicious HTTP traffic is HTTP data traffic of a malicious attack. This method builds an advanced machine learning model to identify and classify normal and malicious traffic. It comprehensively uses the Transformer model, outlier detection model and attack feature model to achieve in-depth analysis and precise classification of HTTP traffic. It not only improves the accuracy and efficiency of detection, but also enhances the adaptability and interpretability of the model. It provides a powerful tool for the field of network security and solves the problem that the HTTP traffic detection method in the existing technology is difficult to effectively identify new network attack methods and has weak defense capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 A hardware structure block diagram of a mobile terminal for performing a multi-level malicious HTTP traffic detection method provided in an embodiment of the present application is shown;

[0017] Figure 2 A schematic flow chart of a multi-level malicious HTTP traffic detection method provided according to an embodiment of the present application is shown;

[0018] Figure 3 An example diagram of HTTP traffic formatting provided according to an embodiment of the present application is shown;

[0019] Figure 4 A flow chart of HTTP traffic detection provided according to an embodiment of the present application is shown;

[0020] Figure 5 A schematic flow chart of another multi-level malicious HTTP traffic detection method provided according to an embodiment of the present application is shown;

[0021] Figure 6 A schematic diagram of a process flow of another multi-level malicious HTTP traffic detection method provided according to an embodiment of the present application is shown;

[0022] Figure 7 A flow overall architecture diagram provided according to an embodiment of the present application is shown;

[0023] Figure 8 A structural block diagram of a multi-level malicious HTTP traffic detection device provided according to an embodiment of the present application is shown.

[0024] The above drawings include the following reference numerals:

[0025] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0026] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] As introduced in the background technology, traditional detection methods based on signature matching are powerless when faced with new attack methods and are difficult to effectively identify and defend. In order to solve the problem that HTTP traffic detection methods in the prior art are difficult to effectively identify new network attack methods and have weak defense capabilities, the embodiments of the present application provide a multi-level malicious HTTP traffic detection method, a multi-level malicious HTTP traffic detection device, a computer-readable storage medium and an electronic device.

[0030] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0031] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a multi-level malicious HTTP traffic detection method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0032] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the detection method of multi-level malicious HTTP traffic in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. The transmission device 106 is used to receive or send data via a network. The specific examples of the above networks may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] In this embodiment, a multi-level malicious HTTP traffic detection method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] Figure 2 1 is a flow chart of a method for detecting multi-level malicious HTTP traffic according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0035] Step S201, reading HTTP protocol data, and using a multi-layer transformer model to preliminarily filter the HTTP protocol data to obtain initial abnormal data;

[0036] Specifically, the model uses comprehensive HTTP request analysis, covering key parts such as URI, request body, and request header, ensuring comprehensive identification of potential attack behaviors. The parallel processing capability of the Transformer model makes the model more efficient when processing large-scale data sets.

[0037] Among them, reading HTTP protocol data includes the following steps:

[0038] Step S301, read the initial HTTP protocol data from the Kafka topic;

[0039] Step S302: Using the pytorch deep learning framework, the data format of the above-mentioned initial HTTP protocol data is converted into a tokens format to obtain the above-mentioned HTTP protocol data.

[0040] Specifically, the extraction of such multi-dimensional features enables the model to analyze data from different angles, thus improving the accuracy and robustness of detection.

[0041] Before using the multi-layer transformer model to preliminarily filter the HTTP protocol data to obtain initial abnormal data, the method further includes the following steps:

[0042] Step S401, obtaining historical normal traffic and historical malicious traffic;

[0043] The specific data sources mainly include attack sample data and normal traffic data. The attack sample data sources are as follows:

[0044] 1. CSIC2010 WEB malicious traffic;

[0045] 2. Malicious traffic intercepted by WAF (data from GitHub): Some samples are not malicious, and the filtering scripts have been put into the project;

[0046] 3. Web risk events on PRS products: Some of these risks do not belong to malicious samples. The types of samples that do not belong to malicious samples are: "weak password", "threat intelligence domain name detection", "Nginx directory traversal response", "Apache Tomcat form authentication user name enumeration request", "threat intelligence-domain name n";

[0047] 4. Leakage scanning tools generate HTTP protocol: mainly some scanning tools scan network traffic;

[0048] 5. Simulate web traffic of various combined attacks.

[0049] The sources of normal traffic data are as follows:

[0050] 1. Normal traffic of CSIC2010 WEB;

[0051] 2. Daily traffic of freebuf: It contains some attack traffic that needs to be filtered;

[0052] 3. Capture a large amount of HTTP traffic that accesses the web on a regular basis through the network card;

[0053] 4. Simulate web traffic in various normal scenarios.

[0054] Step S402, structuring the above historical normal traffic and the above historical malicious traffic to obtain initial normal traffic and initial malicious traffic in the format of tokens;

[0055] In some instances, in-depth protocol parsing and structuring of raw HTTP traffic data into a unified and structured format are performed. Specifically, Figure 3 An example diagram of HTTP traffic formatting provided according to an embodiment of the present application is shown, such as Figure 3 As shown in the figure, the request and response traffic packets are put together. A vocabulary table customized for this task is constructed, which realizes the mapping transformation from text tokens to unique numerical IDs based on all the unique tokens collected.

[0056] Step S403, constructing an initial transformer model based on the input embedding mechanism, the multi-layer transformer encoder and the classification output layer;

[0057] In some instances, an input embedding mechanism is introduced to convert numerical tokens into dense vectors in a high-dimensional vector space. A multi-layer Transformer encoder is constructed to perform deep feature learning through a self-attention mechanism and a multi-layer feedforward network to achieve efficient capture and abstraction of complex sequence patterns. A classification output layer is designed, and a fully connected layer is used to map the high-level features of the Transformer encoder to predefined normal / malicious category labels to complete the final classification task.

[0058] Step S404, converting the initial normal traffic and the initial malicious traffic into dense vectors in a high-dimensional vector space, and inputting them into the initial transformer model;

[0059] Step S405, controlling the initial transformer model to perform deep feature learning through a self-attention mechanism and a multi-layer feedforward network, and using a fully connected layer to map the high-level features of the transformer encoder to predefined normal category labels and malicious category labels, so as to train the initial transformer model and obtain a pre-trained transformer model;

[0060] Step S406, optimizing the pre-trained transformer model based on the cross entropy loss function to obtain the multi-layer transformer model.

[0061] Specifically, the Transformer model uses the self-attention mechanism to efficiently capture long-distance dependencies and complex patterns in HTTP traffic. This mechanism can focus on multiple features in the traffic at the same time, thereby achieving more accurate feature extraction. By extracting features such as parameters, special symbols, and encoding methods, the model can capture subtle anomalies in HTTP requests and enhance the ability to identify complex attack patterns.

[0062] The Transformer model uses a multi-layer structure, and each layer can learn different levels of feature representation. In HTTP protocol traffic, different levels of traffic features (such as URL, header information, content, etc.) may have different effects on traffic analysis. The multi-layer structure can simultaneously learn these different levels of feature representation, thereby improving the accuracy of traffic analysis. In addition, the parallel processing capability of the Transformer model makes the model more efficient when processing large-scale data sets.

[0063] In addition, the model combines the powerful representation capabilities of the deep learning Transformer model and the interpretability of the traditional machine learning model, achieving complementary advantages. The deep learning model can automatically extract features and perform complex pattern recognition, while the traditional machine learning model provides better interpretability and stability in some cases. This fusion strategy enables the model to maintain high accuracy while also having a certain degree of interpretability, which helps security analysts understand the decision-making process of the model.

[0064] In some instances, the loss function uses cross-entropy loss as a quantitative measure of the difference between the model's predicted probability distribution and the true label.

[0065] Iterate the training model on the training set and validation set, regularly evaluate and adjust the model parameters to prevent overfitting and ensure the generalization ability of the model. Comprehensively evaluate the effectiveness of the model in identifying malicious HTTP traffic through multi-dimensional indicators such as precision, recall, and F1 score.

[0066] The data set is divided into a training set and a test set in a ratio of 8:2. Then training is performed on the training set and verification is performed on the test set. The test results are shown in Table 1:

[0067] Table 1. Test results

[0068] Model Accuracy illustrate TestCNN 0.987 / Transformer 0.9996 /

[0069] Step S202, using an NLP model and / or an outlier detection model to perform feature extraction on the initial abnormal data to obtain feature data of each of the initial abnormal data, wherein the feature data includes text features and / or parameter features, the NLP model is a model for extracting text features from the initial abnormal data, and the outlier detection model is a model for extracting parameter features from the initial abnormal data;

[0070] Specifically, by analyzing network traffic, system behavior, or application behavior to identify activities that deviate significantly from normal patterns, anomaly-based detection schemes can adjust according to changes in normal behavior and adapt to changes in system and user behavior. This method does not rely on pre-defined attack patterns and can therefore identify new or unknown threats.

[0071] The model uses comprehensive HTTP request analysis, covering key parts such as URI, request body, and request header, ensuring comprehensive identification of potential attack behaviors. By extracting features such as parameters, special symbols, and encoding methods, the model can capture subtle anomalies in HTTP requests and enhance the ability to identify complex attack patterns. This multi-dimensional feature extraction enables the model to analyze data from different angles, improving the accuracy and robustness of detection.

[0072] Before extracting features from the initial abnormal data using the NLP model and / or the outlier detection model to obtain feature data of each of the initial abnormal data, the method further includes the following steps:

[0073] Step S501, obtaining historical parameter normal flow;

[0074] Step S502, obtaining historical parameter malicious traffic and attack labels of each of the above historical parameter malicious traffic;

[0075] Step S503, based on the above historical parameter normal traffic, the above historical parameter malicious traffic and the attack labels of each of the above historical parameter malicious traffic, a random forest model is used to train an initial outlier detection model to obtain the above outlier detection model.

[0076] Specifically, the model is constructed in a parameterized way and optimized for different types of attack behaviors such as SQL injection and XSS attacks. This targeted detection method allows the model to learn specific patterns of specific attack types, thereby improving the recognition rate of these attack types. Parameterized construction also allows the model to flexibly adapt to new attack patterns. By adjusting parameters, it can quickly adapt to new threats, enhancing the adaptability and scalability of the model.

[0077] In some instances, the outlier model mainly detects the value in the POST (equivalent to the request body), GET request (equivalent to the URI content, for example: pageId=93179875) and header to detect whether there is an attack behavior.

[0078] Before training the model, you need to obtain the parameters of normal traffic and malicious traffic. Normal traffic mainly comes from two aspects: HTTP traffic obtained from the intranet, extracting the parameter part; and simulating normal data traffic.

[0079] Malicious traffic mainly comes from artificial data. First, we collect data sets such as SQL injection, XSS attack, and command execution from the Internet. When artificially creating data, we randomly insert corresponding attacks.

[0080] Since it is a multi-classification model, malicious traffic must be labeled as different categories (SQL injection: 1, XSS attack: 2, command execution: 3).

[0081] Feature extraction: See Table 2 for specific features:

[0082] Table 2. Parameter model characteristics table

[0083]

[0084]

[0085] In addition, the random forest model is used to train the model. Since the model uses the random forest model, the accuracy rate can reach 99.9%, so there is no need to compare the models and adjust the parameters.

[0086] Step S203, inputting the above-mentioned characteristic data into the Attack model to obtain the malicious prediction value corresponding to the above-mentioned initial abnormal data;

[0087] Specifically, the attack model inputs feature data into its algorithm for analysis and prediction, and gives corresponding malicious prediction values ​​based on the model's training and learning results. This targeted detection method allows the model to learn specific patterns of specific attack types, thereby improving the recognition rate of these attack types.

[0088] Before inputting the characteristic data into the Attack model to obtain the malicious prediction value corresponding to the initial abnormal data, the method further includes the following steps:

[0089] Step S601, obtaining a training data set, wherein the training data set includes feature data of first historical abnormal data and malicious prediction values ​​corresponding to the feature data of the first historical abnormal data;

[0090] Step S602, according to the data in the training data set, the initial Attack model is trained using a random forest model to obtain a pre-trained Attack model;

[0091] Step S603, obtaining a test data set, and tuning parameters of the pre-trained Attack model according to the test data set to obtain the Attack model, wherein the test data set includes feature data of the second historical abnormal data and malicious prediction values ​​corresponding to the feature data of the second historical abnormal data.

[0092] Specifically, these data are used to tune the parameters of the pre-trained Attack model to obtain the final Attack model. Parameter tuning is to improve the performance and accuracy of the model and ensure that the model can accurately predict the malicious behavior of the second historical anomaly data. Parameterized construction also allows the model to flexibly adapt to new attack patterns. By adjusting parameters, it can quickly adapt to new threats, enhancing the adaptability and scalability of the model.

[0093] In some instances, the data used for Attack model training comes from the same source as the NLP model. The above two models are used when performing Attack model features. These features are divided into global features, URI features, header features, and parameter features according to the HTTP protocol. The specific features are shown in Tables 3 to 6:

[0094] Table 3. Global feature table

[0095]

[0096] Table 4. URI feature table

[0097]

[0098]

[0099] Table 5. Header feature table

[0100]

[0101] Table 6. Parameter characteristics table

[0102]

[0103]

[0104] After obtaining the above eigenvalues, use random forest to train the Attack model and tune the model.

[0105] Then the data set is divided into a training set and a test set according to an 8:2 ratio. Then training is performed on the training set and verification is performed on the test set. The test results are shown in Table 7:

[0106] Table 7. Test results

[0107] Model Accuracy explain LogisticRegression Model 0.988 / Random Forest Model 0.993 / SVM Model 0.977 /

[0108] Therefore, the logistic regression and random forest models have the same effect, both are better than SVM, and the random forest model is finally selected.

[0109] Step S204, determining whether the initial abnormal data is malicious HTTP traffic based on the size relationship between the malicious prediction value of the initial abnormal data and a preset threshold, wherein the malicious HTTP traffic is HTTP data traffic of a malicious attack.

[0110] Specifically, by comparing the malicious prediction value with the preset threshold, it is possible to determine whether the initial abnormal data is malicious HTTP traffic, which improves the accuracy and efficiency of detection and enhances the adaptability and interpretability of the model.

[0111] According to the relationship between the malicious prediction value of the initial abnormal data and the preset threshold, determining whether the initial abnormal data is malicious HTTP traffic includes the following steps:

[0112] Step S701, when the malicious prediction value of the initial abnormal data is greater than the preset threshold, determining that the initial abnormal data is the malicious HTTP traffic;

[0113] Step S702: when the malicious prediction value of the initial abnormal data is less than or equal to the preset threshold, determine that the initial abnormal data is non-malicious HTTP traffic.

[0114] Specifically, after determining the malicious HTTP traffic, the malicious attack type to which the traffic belongs is inferred. The attack tags include command attack, SQL injection, XSS injection, Java attack, parameter, URL, Header, and vulnerability scan, in preparation for the subsequent scan type attack tag confirmation.

[0115] After determining whether the initial abnormal data is malicious HTTP traffic based on the relationship between the malicious prediction value of the initial abnormal data and the preset threshold, the method further includes the following steps:

[0116] Step S801, in the case where the initial abnormal data is the malicious HTTP traffic, determining the malicious attack type of the malicious HTTP traffic according to the characteristics of the malicious HTTP traffic;

[0117] Step S802, determining the malicious attack type as the attack type label of the malicious HTTP traffic;

[0118] Step S803, performing window aggregation within a preset time period according to SIP to obtain an aggregation window, and detecting the amount of the malicious HTTP traffic whose attack type label is a preset label within the aggregation window;

[0119] Step S804, when the number of the malicious HTTP traffic whose attack type label is a preset label in the aggregation window exceeds a preset number, the attack type labels of all malicious HTTP traffic in the aggregation window are set to the preset label to obtain a detection result;

[0120] Step S805: output the above detection results to Kafka.

[0121] Specifically, this method can quickly and accurately identify the attack type of malicious HTTP traffic, and can effectively filter out the real malicious attack traffic through window aggregation and quantity judgment, thereby improving the accuracy and efficiency of detection. At the same time, the detection results are output to Kafka, which can be easily processed and analyzed later.

[0122] Figure 4 A flow chart of HTTP traffic detection provided according to an embodiment of the present application is shown. First, the HTTP protocol is read in kafka, and then the detection is started, the param model, attack model and NLP model are loaded respectively, the attack model is extracted and predicted in combination with the whitelist and blacklist strategies, the label judgment is determined, and then the scanning is performed to output the malicious HTTP traffic to kagka.

[0123] Among them, the Transformer model shows excellent performance in processing sequence data with its self-attention mechanism, while the outlier detection model quickly identifies abnormal behaviors of parameters in the Request. The attack feature model extracts key features from the HTTP protocol and conducts comprehensive analysis based on expert knowledge to determine the attack type and abnormal location.

[0124] In terms of model training, a large amount of real-world HTTP traffic data, including normal traffic and malicious traffic samples, is used to ensure the generalization and accuracy of the model. The data comes from a wide range of sources, including public data sets, internal company data, simulated attack traffic, etc., to ensure the diversity and representativeness of the data. Through in-depth preprocessing and feature engineering of these data, a rich feature set is constructed, providing a solid foundation for model training.

[0125] In addition, this embodiment also pays special attention to the evaluation and optimization of the model, using the cross entropy loss function and multi-dimensional evaluation indicators such as precision, recall and F1 score to comprehensively evaluate the model performance. Through the training and validation cycle, the model parameters are continuously adjusted to prevent overfitting, ensuring the robustness and generalization ability of the model.

[0126] Finally, the malicious HTTP traffic detection algorithm proposed in this embodiment is not only innovative in theory, but also demonstrates high detection accuracy and efficiency in practical applications, providing an effective technical means for network security protection.

[0127] Figure 5 1 is a flow chart of a method for detecting multi-level malicious HTTP traffic according to an embodiment of the present application. Figure 5 As shown, the method first obtains normal traffic and malicious traffic, performs data processing, extracts param model features, and then trains the param model to obtain the param model; then performs data processing on malicious HTTP samples; then performs data processing on normal HTTP samples, performs word segmentation and word vectorization, trains the NLP model, and obtains the NLP model; finally, the NLP model and the param model are preloaded, and the data of the Attack model is combined to extract features of the Attack model to obtain the Attack model.

[0128] Among them, the param model is the outlier detection model, and the NLP model is the transformer model.

[0129] The above-mentioned multi-level malicious HTTP traffic detection method of the present application first reads HTTP protocol data, and uses a multi-layer transformer model to preliminarily filter the HTTP protocol data to obtain initial abnormal data; then uses an NLP model and / or an outlier detection model to extract features of the initial abnormal data to obtain feature data of each initial abnormal data, the feature data includes text features and / or parameter features, the NLP model is a model for extracting text features in the initial abnormal data, and the outlier detection model is a model for extracting parameter features in the initial abnormal data; then the feature data is input into the Attack model to obtain a malicious prediction value corresponding to the initial abnormal data; finally, according to the size relationship between the malicious prediction value of the initial abnormal data and a preset threshold, it is determined whether the initial abnormal data is malicious HTTP traffic, and the malicious HTTP traffic is HTTP data traffic of malicious attacks. This method builds an advanced machine learning model to identify and classify normal and malicious traffic. It comprehensively uses the Transformer model, outlier detection model and attack feature model to achieve in-depth analysis and precise classification of HTTP traffic. It not only improves the accuracy and efficiency of detection, but also enhances the adaptability and interpretability of the model. It provides a powerful tool for the field of network security and solves the problem that the HTTP traffic detection method in the existing technology is difficult to effectively identify new network attack methods and has weak defense capabilities.

[0130] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the multi-level malicious HTTP traffic detection method of the present application will be described in detail below in conjunction with specific embodiments.

[0131] This embodiment relates to a specific multi-level malicious HTTP traffic detection method, such as Figure 6 As shown, the following steps are included:

[0132] Step S1: Start detection. Read HTTP protocol data from the Kafka topic.

[0133] Step S2: Load the transformer model and use pytorch / torch to further convert HTTP traffic into tokens.

[0134] Step S3: Whitelist + blacklist strategy. First determine whether it complies with the whitelist strategy:

[0135] a) If the source IP and target IP are in the whitelist, they are filtered out and not tested;

[0136] b) If the URL is filtered in the whitelist;

[0137] c) The request is get, and the status code is 200, which is directly filtered out;

[0138] d) Filter out port 9200, because elasticsearch requests are generally intranet requests.

[0139] Blacklist strategy:

[0140] a) First check whether the NLP model is loaded. If not, load the NLP model;

[0141] b) Combine uri, request_body and request_header_values ​​into a string and use the NLP model to detect HTTP. If the predicted value is < 0.6, filter out the information.

[0142] Step S4: Load the outlier detection model and the attack model.

[0143] Step S5: according to the feature tables (Table 2 to Table 5) in the second part of the specific scheme of the Attack model, the features of the Attack model are extracted in sequence.

[0144] Step S6: Use the Attack model to predict the features extracted in the previous step. If the predicted value is > 0.6, it is considered malicious traffic, otherwise it is normal traffic.

[0145] Step S7: After determining that the traffic is malicious HTTP traffic, it is necessary to obtain features according to step 5, and then further infer the malicious attack type to which the traffic belongs. The attack tags include command attack, SQL injection, XSS injection, Java attack, parameter, URL, Header, and vulnerability scan.

[0146] Step S8: Scan type attack tag confirmation: Perform 1-minute window aggregation according to SIP. If the traffic containing vulnerability scan tags is detected to exceed the threshold within the aggregation window, all malicious traffic in this aggregation window will be set to add a "vulnerability scan" tag.

[0147] Step S9: Output the detection results to Kafka.

[0148] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0149] Figure 7The overall traffic architecture diagram provided according to the embodiment of the present application is shown, which consists of two aspects: system implementation and key technologies. The system implementation is to collect traffic through the steps of interface, diversion, aggregation and filtering, input the collected traffic into the traffic identification engine, and process it through the traffic security analysis engine, HTTP traffic restoration engine and HTTPs traffic decryption engine, input HTTP traffic into the HTTP traffic restoration engine, and HTTPs traffic into the HTTPs traffic decryption engine, and finally obtain the black and white list. The key technologies of the system implementation are multi-level traffic identification model, decompression acceleration technology and high-speed regular matching technology.

[0150] The embodiment of the present application also provides a multi-level malicious HTTP traffic detection device. It should be noted that the multi-level malicious HTTP traffic detection device of the embodiment of the present application can be used to execute the detection method for multi-level malicious HTTP traffic provided by the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0151] The following is an introduction to the multi-level malicious HTTP traffic detection device provided in the embodiment of the present application.

[0152] Figure 8 Schematic diagram of a multi-level malicious HTTP traffic detection device according to an embodiment of the present application. Figure 8 As shown, the device includes: a first processing unit 10, an extraction unit 20, a second processing unit 30 and a determination unit 40, the first processing unit 10 is used to read HTTP protocol data, and use a multi-layer transformer model to preliminarily filter the above HTTP protocol data to obtain initial abnormal data; the extraction unit 20 is used to use an NLP model and / or an outlier detection model to extract features of the above initial abnormal data to obtain feature data of each of the above initial abnormal data, the above feature data includes text features and / or parameter features, the above NLP model is a model for extracting text features in the above initial abnormal data, and the above outlier detection model is a model for extracting parameter features in the above initial abnormal data; the second processing unit 30 is used to input the above feature data into the Attack model to obtain a malicious prediction value corresponding to the above initial abnormal data; the determination unit 40 is used to determine whether the above initial abnormal data is malicious HTTP traffic according to the size relationship between the malicious prediction value of the above initial abnormal data and a preset threshold, and the above malicious HTTP traffic is HTTP data traffic of malicious attack.

[0153] The above-mentioned multi-level malicious HTTP traffic detection device of the present application includes: a first processing unit, an extraction unit, a second processing unit and a determination unit, wherein the first processing unit is used to read HTTP protocol data, and use a multi-layer transformer model to preliminarily filter the HTTP protocol data to obtain initial abnormal data; the extraction unit is used to use an NLP model and / or an outlier detection model to extract features of the initial abnormal data to obtain feature data of each initial abnormal data, wherein the feature data includes text features and / or parameter features, the NLP model is a model for extracting text features in the initial abnormal data, and the outlier detection model is a model for extracting parameter features in the initial abnormal data; the second processing unit is used to input the feature data into an Attack model to obtain a malicious prediction value corresponding to the initial abnormal data; the determination unit is used to determine whether the initial abnormal data is malicious HTTP traffic according to the size relationship between the malicious prediction value of the initial abnormal data and a preset threshold value, wherein the malicious HTTP traffic is HTTP data traffic of a malicious attack. The device builds an advanced machine learning model to identify and classify normal and malicious traffic, and comprehensively uses the Transformer model, outlier detection model and attack feature model to achieve in-depth analysis and precise classification of HTTP traffic. It not only improves the accuracy and efficiency of detection, but also enhances the adaptability and interpretability of the model. It provides a powerful tool for the field of network security and solves the problem that the HTTP traffic detection method in the existing technology is difficult to effectively identify new network attack methods and has weak defense capabilities.

[0154] In some instances, before the multi-layer transformer model is used to perform preliminary filtering on the HTTP protocol data to obtain initial abnormal data, the device further includes a first acquisition module, a first processing module, a first construction module, a first conversion film, a first control module and a first optimization module, wherein the first acquisition module is used to obtain historical normal traffic and historical malicious traffic; the first processing module is used to structure the historical normal traffic and the historical malicious traffic to obtain initial normal traffic and initial malicious traffic in the format of tokens; the first construction module is used to construct an initial transformer model based on an input embedding mechanism, a multi-layer transformer encoder and a classification output layer; the first conversion film is used to structure the initial normal traffic and the historical malicious traffic to obtain initial normal traffic and initial malicious traffic in the format of tokens; The traffic and the above-mentioned initial malicious traffic are converted into dense vectors in a high-dimensional vector space and input into the above-mentioned initial transformer model; the first control module is used to control the above-mentioned initial transformer model to perform deep feature learning through a self-attention mechanism and a multi-layer feedforward network, and use a fully connected layer to map the high-level features of the above-mentioned transformer encoder to predefined normal category labels and malicious category labels to train the above-mentioned initial transformer model to obtain a pre-trained transformer model; the first optimization module is used to optimize the above-mentioned pre-trained transformer model based on a cross-entropy loss function to obtain the above-mentioned multi-layer transformer model. The Transformer model uses a self-attention mechanism to efficiently capture long-distance dependencies and complex patterns in HTTP traffic. By extracting features such as parameters, special symbols, and encoding methods, the model can capture subtle anomalies in HTTP requests and enhance the ability to recognize complex attack patterns.

[0155] In some examples, before using the NLP model and / or the outlier detection model to extract features from the above initial anomaly data and obtain the feature data of each of the above initial anomaly data, the above device also includes a second acquisition module, a third acquisition module and a second processing module, the second acquisition module is used to obtain the historical parameter normal traffic; the third acquisition module is used to obtain the historical parameter malicious traffic and the attack label of each of the above historical parameter malicious traffic; the second processing module is used to train the initial outlier detection model using the random forest model based on the above historical parameter normal traffic, the above historical parameter malicious traffic and the attack label of each of the above historical parameter malicious traffic to obtain the above outlier detection model. The model is constructed in a parameterized manner, and is specially optimized for different types of attack behaviors such as SQL injection and XSS attack. The random forest model is used to train the model. Since the model uses the random forest model, the accuracy rate can reach 99.9%, so there is no model comparison and parameter adjustment.

[0156] In some instances, before the above-mentioned characteristic data is input into the Attack model to obtain the malicious prediction value corresponding to the above-mentioned initial abnormal data, the above-mentioned device also includes a fourth acquisition module, a third processing module and a fifth acquisition module, the fourth acquisition module is used to obtain a training data set, the above-mentioned training data set includes the characteristic data of the first historical abnormal data and the malicious prediction value corresponding to the characteristic data of the above-mentioned first historical abnormal data; the third processing module is used to train the initial Attack model using a random forest model according to the data in the above-mentioned training data set to obtain a pre-trained Attack model; the fifth acquisition module is used to obtain a test data set, and according to the above-mentioned test data set, the parameters of the above-mentioned pre-trained Attack model are tuned to obtain the above-mentioned Attack model, the above-mentioned test data set includes the characteristic data of the second historical abnormal data and the malicious prediction value corresponding to the characteristic data of the above-mentioned second historical abnormal data. Parameterized construction also allows the model to flexibly adapt to new attack modes, and can quickly adapt to new threats by adjusting parameters, thereby enhancing the adaptability and scalability of the model.

[0157] In some examples, the determination unit includes a first determination module and a second determination module, the first determination module is used to determine that the initial abnormal data is the malicious HTTP traffic when the malicious prediction value of the initial abnormal data is greater than the preset threshold; the second determination module is used to determine that the initial abnormal data is non-malicious HTTP traffic when the malicious prediction value of the initial abnormal data is less than or equal to the preset threshold. By comparing the size relationship between the malicious prediction value and the preset threshold, it can be determined whether the initial abnormal data is malicious HTTP traffic, which improves the accuracy and efficiency of detection and enhances the adaptability and interpretability of the model.

[0158] In some instances, after determining whether the initial abnormal data is malicious HTTP traffic based on the size relationship between the malicious prediction value of the initial abnormal data and a preset threshold, the device further includes a third determination module, a fourth determination module, a fourth processing module, a fifth processing module and a first output module, wherein the third determination module is used to determine the malicious attack type of the malicious HTTP traffic based on the characteristics of the malicious HTTP traffic when the initial abnormal data is the malicious HTTP traffic; the fourth determination module is used to determine the malicious attack type as the attack type label of the malicious HTTP traffic; the fourth processing module is used to perform window aggregation within a preset time period according to sip to obtain an aggregation window, and detect the number of the malicious HTTP traffic whose attack type label is a preset label in the aggregation window; the fifth processing module is used to set the attack type labels of all malicious HTTP traffic in the aggregation window to the preset label to obtain a detection result when the number of the malicious HTTP traffic whose attack type label is a preset label in the aggregation window exceeds a preset number; the first output module is used to output the detection result to kafka. After the malicious HTTP traffic is identified, the malicious attack type to which the traffic belongs is inferred. The attack tags include command attack, SQL injection, XSS injection, Java attack, parameter, URL, Header, and vulnerability scan, in preparation for the subsequent scan type attack tag confirmation.

[0159] In some examples, the first processing unit includes a first reading module and a second conversion module, wherein the first reading module is used to read the initial HTTP protocol data from the topic of kafka; the second conversion module is used to convert the data format of the initial HTTP protocol data into the tokens format using the pytorch deep learning framework to obtain the HTTP protocol data. This multi-dimensional feature extraction enables the model to analyze data from different angles, improving the accuracy and robustness of detection.

[0160] The above-mentioned multi-level malicious HTTP traffic detection device includes a processor and a memory. The above-mentioned first processing unit and the like are stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to implement corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in any combination.

[0161] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem that the HTTP traffic detection method in the prior art is difficult to effectively identify new network attack methods and has weak defense capabilities can be solved by adjusting kernel parameters.

[0162] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0163] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the multi-level malicious HTTP traffic detection method.

[0164] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes the multi-level malicious HTTP traffic detection method when running.

[0165] An embodiment of the present invention provides a device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, steps of a method for detecting malicious HTTP traffic at least at multiple levels are implemented.

[0166] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of initiating a method for detecting malicious HTTP traffic having at least multiple levels.

[0167] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0168] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0169] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0170] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0172] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0173] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0174] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0175] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0176] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0177] 1) The above-mentioned multi-level malicious HTTP traffic detection method of the present application first reads HTTP protocol data, and uses a multi-layer transformer model to preliminarily filter the HTTP protocol data to obtain initial abnormal data; then uses an NLP model and / or an outlier detection model to extract features of the initial abnormal data to obtain feature data of each initial abnormal data, the feature data includes text features and / or parameter features, the NLP model is a model for extracting text features in the initial abnormal data, and the outlier detection model is a model for extracting parameter features in the initial abnormal data; then the feature data is input into the Attack model to obtain a malicious prediction value corresponding to the initial abnormal data; finally, according to the size relationship between the malicious prediction value of the initial abnormal data and a preset threshold, it is determined whether the initial abnormal data is malicious HTTP traffic, and the malicious HTTP traffic is the HTTP data traffic of a malicious attack. This method builds an advanced machine learning model to identify and classify normal and malicious traffic. It comprehensively uses the Transformer model, outlier detection model and attack feature model to achieve in-depth analysis and precise classification of HTTP traffic. It not only improves the accuracy and efficiency of detection, but also enhances the adaptability and interpretability of the model. It provides a powerful tool for the field of network security and solves the problem that the HTTP traffic detection method in the existing technology is difficult to effectively identify new network attack methods and has weak defense capabilities.

[0178] 2) The above-mentioned multi-level malicious HTTP traffic detection device of the present application includes: a first processing unit, an extraction unit, a second processing unit and a determination unit, wherein the first processing unit is used to read HTTP protocol data, and use a multi-layer transformer model to preliminarily filter the HTTP protocol data to obtain initial abnormal data; the extraction unit is used to use an NLP model and / or an outlier detection model to extract features of the initial abnormal data to obtain feature data of each initial abnormal data, wherein the feature data includes text features and / or parameter features, the NLP model is a model for extracting text features in the initial abnormal data, and the outlier detection model is a model for extracting parameter features in the initial abnormal data; the second processing unit is used to input the feature data into the Attack model to obtain a malicious prediction value corresponding to the initial abnormal data; the determination unit is used to determine whether the initial abnormal data is malicious HTTP traffic based on the size relationship between the malicious prediction value of the initial abnormal data and a preset threshold, wherein the malicious HTTP traffic is HTTP data traffic of a malicious attack. The device builds an advanced machine learning model to identify and classify normal and malicious traffic, and comprehensively uses the Transformer model, outlier detection model and attack feature model to achieve in-depth analysis and precise classification of HTTP traffic. It not only improves the accuracy and efficiency of detection, but also enhances the adaptability and interpretability of the model. It provides a powerful tool for the field of network security and solves the problem that the HTTP traffic detection method in the existing technology is difficult to effectively identify new network attack methods and has weak defense capabilities.

[0179] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-level malicious HTTP traffic detection method, characterized in that: include: Read HTTP protocol data, and use a multi-layer transformer model to preliminarily filter the HTTP protocol data to obtain initial abnormal data; Using an NLP model and / or an outlier detection model to extract features from the initial abnormal data to obtain feature data of each of the initial abnormal data, wherein the feature data includes text features and / or parameter features, the NLP model is a model for extracting text features from the initial abnormal data, and the outlier detection model is a model for extracting parameter features from the initial abnormal data; Input the characteristic data into the Attack model to obtain the malicious prediction value corresponding to the initial abnormal data; According to the size relationship between the malicious prediction value of the initial abnormal data and a preset threshold, it is determined whether the initial abnormal data is malicious HTTP traffic, and the malicious HTTP traffic is HTTP data traffic of a malicious attack.

2. The method according to claim 1, characterized in that Before using the multi-layer transformer model to preliminarily filter the HTTP protocol data to obtain initial abnormal data, the method further includes: Obtain historical normal traffic and historical malicious traffic; Structuring the historical normal traffic and the historical malicious traffic to obtain initial normal traffic and initial malicious traffic in the format of tokens; Build an initial transformer model based on the input embedding mechanism, multi-layer transformer encoder, and classification output layer; Convert the initial normal traffic and the initial malicious traffic into dense vectors in a high-dimensional vector space, and input them into the initial transformer model; Controlling the initial transformer model to perform deep feature learning through a self-attention mechanism and a multi-layer feedforward network, and mapping high-level features of the transformer encoder to predefined normal category labels and malicious category labels using a fully connected layer, so as to train the initial transformer model and obtain a pre-trained transformer model; The pre-trained transformer model is optimized based on a cross entropy loss function to obtain the multi-layer transformer model.

3. The method according to claim 1, characterized in that: Before extracting features from the initial abnormal data using the NLP model and / or the outlier detection model to obtain feature data of each of the initial abnormal data, the method further includes: Get historical parameters of normal flow; Obtaining historical parameter malicious traffic and attack labels of each of the historical parameter malicious traffic; Based on the historical parameter normal traffic, the historical parameter malicious traffic and the attack labels of each historical parameter malicious traffic, a random forest model is used to train an initial outlier detection model to obtain the outlier detection model.

4. The method according to claim 1, characterized in that: Before inputting the characteristic data into the Attack model to obtain the malicious prediction value corresponding to the initial abnormal data, the method further includes: Acquire a training data set, wherein the training data set includes feature data of first historical abnormal data and malicious prediction values ​​corresponding to the feature data of the first historical abnormal data; According to the data in the training data set, the initial Attack model is trained using a random forest model to obtain a pre-trained Attack model; A test data set is obtained, and parameters of the pre-trained Attack model are tuned according to the test data set to obtain the Attack model, wherein the test data set includes feature data of the second historical abnormal data and malicious prediction values ​​corresponding to the feature data of the second historical abnormal data.

5. The method according to claim 1, characterized in that Determining whether the initial abnormal data is malicious HTTP traffic according to a size relationship between the malicious prediction value of the initial abnormal data and a preset threshold value includes: When the malicious prediction value of the initial abnormal data is greater than the preset threshold, determining that the initial abnormal data is the malicious HTTP traffic; When the malicious prediction value of the initial abnormal data is less than or equal to the preset threshold, it is determined that the initial abnormal data is non-malicious HTTP traffic.

6. The method according to claim 1, characterized in that After determining whether the initial abnormal data is malicious HTTP traffic according to the size relationship between the malicious prediction value of the initial abnormal data and the preset threshold, the method further includes: In a case where the initial abnormal data is the malicious HTTP traffic, determining the malicious attack type of the malicious HTTP traffic according to the characteristics of the malicious HTTP traffic; Determine the malicious attack type as an attack type label of the malicious HTTP traffic; Perform window aggregation within a preset time period according to SIP to obtain an aggregation window, and detect the amount of malicious HTTP traffic whose attack type label is a preset label within the aggregation window; When the number of malicious HTTP traffic whose attack type label is a preset label in the aggregation window exceeds a preset number, the attack type labels of all malicious HTTP traffic in the aggregation window are set to the preset label to obtain a detection result; The detection result is output to Kafka.

7. The method according to any one of claims 1 to 6, characterized in that: Read HTTP protocol data, including: Read the initial HTTP protocol data from the Kafka topic; The pytorch deep learning framework is used to convert the data format of the initial HTTP protocol data into a tokens format to obtain the HTTP protocol data.

8. A multi-level malicious HTTP traffic detection device, characterized in that: include: The first processing unit is used to read HTTP protocol data and use a multi-layer transformer model to preliminarily filter the HTTP protocol data to obtain initial abnormal data; An extraction unit, configured to perform feature extraction on the initial abnormal data using an NLP model and / or an outlier detection model to obtain feature data of each of the initial abnormal data, wherein the feature data includes text features and / or parameter features, the NLP model is a model for extracting text features from the initial abnormal data, and the outlier detection model is a model for extracting parameter features from the initial abnormal data; A second processing unit, configured to input the feature data into an Attack model to obtain a malicious prediction value corresponding to the initial abnormal data; The determination unit is used to determine whether the initial abnormal data is malicious HTTP traffic according to the size relationship between the malicious prediction value of the initial abnormal data and a preset threshold, and the malicious HTTP traffic is HTTP data traffic of a malicious attack.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the multi-level malicious HTTP traffic detection method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for detecting multi-level malicious HTTP traffic as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Detection method and system for HTTP malicious traffic

    CN109960729A

  • Advanced persistent threat (APT) attack behavior detection method, device and equipment

    CN115603992A

  • Attack detection method and device, equipment and storage medium

    CN115766153A

  • DDos attack identification method and device, electronic equipment and storage medium

    CN116781338A

  • Network event security monitoring method and system

    CN118200019A

Cited By

  • Safety detection method and device for HTTP (Hyper Text Transport Protocol) request

    CN120768698A

  • A method and apparatus for security detection of HTTP requests

    CN120768698B