Message data detection methods, devices, electronic equipment and storage media
By using the skip-gram model to extract feature vectors and the classification model to identify message types, the problem of high false alarm rate in WAF is solved, achieving more accurate malicious attack detection and reducing the cost of manual intervention.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING TOPSEC NETWORK SECURITY TECH
- Filing Date
- 2022-12-08
- Publication Date
- 2026-05-26
AI Technical Summary
Existing Web Application Firewalls (WAFs) have a high false positive rate when detecting malicious attacks, resulting in a large amount of manual intervention and cost.
The skip-gram model is used to extract feature vectors from abnormal message data, and a classification model is used to determine the type of message data or the associated WAF engine ID, thereby reducing the false alarm rate.
By correcting the classification results of false alarm messages, the false alarm rate of message detection was reduced, and the generalization ability and understanding of contextual relationships of the classification model were improved.
Smart Images
Figure CN115865504B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and more specifically, to a message data detection method, apparatus, electronic device, and storage medium. Background Technology
[0002] Web applications are the most common platform for information and service exchange on the Internet. With the continuous development of the World Wide Web, many businesses, such as e-banking, online shopping, government agencies, and social networks, have moved their services online. Furthermore, the adoption of cloud-based systems and services has further accelerated this transformation. However, this has also made the Web a prime target for malicious attackers. Recent studies have found a dramatic increase in the number of reported network vulnerabilities. The latest statistics indicate that web application vulnerabilities have seriously threatened users' privacy and even financial security. Attacks against web applications can lead to serious consequences such as information leaks and program crashes; therefore, it is necessary to research a mechanism specifically designed to protect web applications from web attacks.
[0003] Web Application Firewalls (WAFs) are firewalls specifically designed to protect web systems from malicious attacks. Traditional WAF protection strategies rely on regular expressions to identify attack signatures, and their protective capabilities depend on the accuracy of expert-defined policy rule sets. As business systems evolve from single-business architectures to dozens of different multi-business cross-platform systems, the value of data becomes increasingly prominent, representing a company's most important asset. Most existing WAFs protect servers by matching the characteristics of request and response content. If malicious strings are found in a request or sensitive information in a response, the request is considered illegal, and the response is stopped. Excellent WAF products can design robust signature databases to detect most malicious attacks while maintaining a low false positive rate, ensuring the normal functional requirements of the website.
[0004] However, as attack techniques continue to evolve, existing rule-based detection methods will have a lot of false positives. These false positives require human intervention, which incurs a significant amount of labor costs. Summary of the Invention
[0005] The purpose of this application is to provide a message data detection method, apparatus, electronic device, and storage medium to reduce the false alarm rate of message detection.
[0006] In a first aspect, the present invention provides a message data detection method, the method comprising:
[0007] Obtain exception message data based on the WAF engine;
[0008] The feature vector of the abnormal message data is extracted based on the skip-gram model;
[0009] The detection result of the abnormal message data is determined based on a classification model. The detection result of the abnormal message data is the type of the abnormal message data or the WAF engine ID associated with the abnormal message data. The feature vector of the abnormal message is used as the input of the classification model, and the type of the abnormal message data is the output of the classification model. The type of the abnormal message data includes a first type and a second type. The first type indicates that the abnormal message data is an abnormal message, and the second type indicates that the abnormal message data is a false alarm message.
[0010] In the first aspect of this application, by acquiring abnormal message data obtained based on a WAF engine, feature vectors of the abnormal message data can be extracted based on a skip-gram model, and the detection result of the abnormal message data can be determined based on a classification model. Finally, based on the detection result, it can be determined whether the abnormal message data is an abnormal message or a false alarm message. When the abnormal message data is detected as a false alarm message, the classification result of the abnormal message data can be corrected to prevent the abnormal message data from being treated as an abnormal message, thereby reducing the false alarm rate of message detection.
[0011] Meanwhile, this application can extract feature vectors from the abnormal message data based on the skip-gram model, thereby enhancing the understanding of contextual relationships through the feature vectors of the abnormal message data, and thus improving the generalization ability of the classification model.
[0012] In an optional implementation, before obtaining the abnormal message data based on the WAF engine, the method further includes:
[0013] Training samples of false alarm messages were obtained from the false alarm message database;
[0014] Training samples of abnormal messages are obtained from an abnormal message database;
[0015] Based on the skip-gram model, feature vectors of the false alarm training samples and feature vectors of the abnormal training samples are extracted.
[0016] The feature vectors of the false alarm training samples are labeled based on the ID of the WAF engine to obtain the first labeled data, and the feature vectors of the abnormal message training samples are labeled based on the ID of the WAF engine to obtain the second labeled data.
[0017] The model parameters of the classification model are trained based on the first labeled data and the second labeled data. The model parameters of the classification model are used to identify the type of the abnormal message data or the WAF engine ID associated with the abnormal message data.
[0018] In the above optional implementation, false positive message training samples can be obtained based on the false positive message database, and abnormal message training samples can be obtained based on the abnormal message database. Then, feature vectors of the false positive message training samples and feature vectors of the abnormal message training samples can be extracted based on the skip-gram model. Then, the feature vectors of the false positive message training samples are labeled based on the WAF engine ID to obtain first labeled data. The feature vectors of the abnormal message training samples are labeled based on the WAF engine ID to obtain second labeled data. Then, the model parameters of the classification model can be trained based on the first labeled data and the second labeled data. The model parameters of the classification model are used to identify the type of the abnormal message data or the WAF engine ID associated with the abnormal message data.
[0019] In optional implementations, the classification model includes the GBDT model, the XGBoost model, and the random forest algorithm model;
[0020] And, the identification of the type of the abnormal message data based on the classification model includes:
[0021] Obtain the type identification results output by the GBDT model, the XGBoost model, and the Random Forest algorithm model;
[0022] Obtain the voting value corresponding to the type recognition result output by the GBDT model, and use it as the first value;
[0023] Obtain the voting value corresponding to the type identification result output by the XGBoost model, and use it as the second value;
[0024] Obtain the voting value corresponding to the type identification result output by the random forest algorithm model, and use it as the third value;
[0025] Based on the type identification results output by the GBDT model, the type identification results output by the XGBoost model, the type identification results output by the Random Forest algorithm model, the first value, the second value, and the third value, the type of the abnormal message data or the WAF engine ID associated with the abnormal message data is determined.
[0026] In the above optional implementation, multiple machine learning models such as GBDT model, XGBoost model and random forest model are used for comprehensive detection to determine whether the detected abnormal message data is a false alarm message or an abnormal message.
[0027] In an optional implementation, the feature vector of the abnormal message data is extracted using the skip-gram model, including:
[0028] Extract the message parameter field data from the aforementioned abnormal message data;
[0029] The message parameter field data is converted into feature vectors of the abnormal message data based on the skip-gram model, wherein the skip-gram model learns in advance the mapping relationship between the message parameter field data and the feature vectors.
[0030] In the above optional implementation, by extracting the message parameter field data from the abnormal message data, the message parameter field data can be converted into a feature vector of the abnormal message data based on the skip-gram model, wherein the skip-gram model pre-learns the mapping relationship between the message parameter field data and the feature vector.
[0031] In an optional implementation, the method further includes:
[0032] When the type of the abnormal message data is the first type, an alarm message is generated.
[0033] In the above optional implementation, when the type of the abnormal message data is the first type, alarm information can be generated to ensure that the abnormal message can trigger an alarm normally.
[0034] In an optional implementation, the method further includes:
[0035] When the type of the abnormal message data is the second type, the abnormal message data is sent to the terminal where the operation and maintenance personnel are located.
[0036] In an optional implementation, obtaining the abnormal message data based on the WAF engine further includes:
[0037] Acquire traffic data and extract WEB message data based on the traffic data;
[0038] The WAF engine detects the WEB message data, and when the parameters in the WEB message data match the WAF engine, the WEB message data is identified as the abnormal message data.
[0039] In the above optional implementation, by acquiring traffic data and extracting WEB packet data based on the traffic data, the WAF engine can detect the WEB packet data. Thus, when the parameters in the WEB packet data match the WAF engine, the WEB packet data is identified as the abnormal packet data.
[0040] In a second aspect, the present invention provides a message data detection device, the device comprising:
[0041] The acquisition module is used to acquire exception message data obtained based on the WAF engine;
[0042] The feature extraction module is used to extract feature vectors from the abnormal message data based on the skip-gram model;
[0043] The determination module is used to determine the detection result of the abnormal message data based on a classification model. The detection result of the abnormal message data is the type of the abnormal message data or the WAF engine ID associated with the abnormal message data. The feature vector of the abnormal message is used as the input of the classification model, and the type of the abnormal message data is the output of the classification model. The type of the abnormal message data includes a first type and a second type. The first type indicates that the abnormal message data is an abnormal message, and the second type indicates that the abnormal message data is a false alarm message.
[0044] The apparatus of the second aspect of this application, by executing a message data detection method, can acquire abnormal message data obtained based on a WAF engine, then extract feature vectors of the abnormal message data based on a skip-gram model, then determine the detection result of the abnormal message data based on a classification model, and finally determine whether the abnormal message data is an abnormal message or a false alarm message based on the detection result. When the abnormal message data is detected as a false alarm message, the classification result of the abnormal message data can be corrected to prevent the abnormal message data from being treated as an abnormal message, thereby reducing the false alarm rate of message detection.
[0045] Thirdly, the present invention provides an electronic device, comprising:
[0046] Processor; and
[0047] The memory is configured to store machine-readable instructions that, when executed by the processor, perform the message data detection method as described in any of the foregoing embodiments.
[0048] The electronic device of the third aspect of this application, by executing a message data detection method, can acquire abnormal message data obtained based on a WAF engine, then extract feature vectors of the abnormal message data based on a skip-gram model, then determine the detection result of the abnormal message data based on a classification model, and finally determine whether the abnormal message data is an abnormal message or a false alarm message based on the detection result. When the abnormal message data is detected as a false alarm message, the classification result of the abnormal message data can be corrected to prevent the abnormal message data from being treated as an abnormal message, thereby reducing the false alarm rate of message detection.
[0049] Fourthly, the present invention provides a storage medium storing a computer program, the computer program being executed by a processor using the message data detection method as described in any of the foregoing embodiments.
[0050] The storage medium of the fourth aspect of this application, by executing a message data detection method, can acquire abnormal message data obtained based on a WAF engine, then extract feature vectors of the abnormal message data based on a skip-gram model, then determine the detection result of the abnormal message data based on a classification model, and finally determine whether the abnormal message data is an abnormal message or a false alarm message based on the detection result. When the abnormal message data is detected as a false alarm message, the classification result of the abnormal message data can be corrected to prevent the abnormal message data from being treated as an abnormal message, thereby reducing the false alarm rate of message detection. Attached Figure Description
[0051] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0052] Figure 1 This is a flowchart illustrating a message data detection method disclosed in an embodiment of this application;
[0053] Figure 2 This is a schematic diagram of the scan result of a skip-gram model disclosed in an embodiment of this application;
[0054] Figure 3 This is a schematic diagram of the structure of a message data detection device disclosed in an embodiment of this application;
[0055] Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. Detailed Implementation
[0056] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.
[0057] Example 1
[0058] Please see Figure 1 , Figure 1 This is a flowchart illustrating a message data detection method disclosed in an embodiment of this application, as shown below. Figure 1 As shown, the method in this application embodiment includes the following steps:
[0059] 101. Obtain exception message data based on the WAF engine;
[0060] 102. Extracting feature vectors from abnormal message data based on the skip-gram model;
[0061] 103. Determine the detection result of abnormal message data based on the classification model. The detection result of abnormal message data is the type of abnormal message data or the WAF engine ID associated with the abnormal message data. The feature vector of the abnormal message is used as the input of the classification model, and the type of abnormal message data is the output of the classification model. The types of abnormal message data include a first type and a second type. The first type indicates that the abnormal message data is an abnormal message, and the second type indicates that the abnormal message data is a false alarm message.
[0062] In this embodiment, by acquiring abnormal message data obtained from the WAF engine, feature vectors of the abnormal message data can be extracted based on the skip-gram model. Then, the detection result of the abnormal message data can be determined based on the classification model. Finally, based on the detection result, it can be determined whether the abnormal message data is an abnormal message or a false alarm message. When the abnormal message data is detected as a false alarm message, the classification result of the abnormal message data can be corrected to prevent the abnormal message data from being treated as an abnormal message, thereby reducing the false alarm rate of message detection.
[0063] Meanwhile, the embodiments of this application are able to extract feature vectors from abnormal message data based on the skip-gram model, thereby enhancing the understanding of contextual relationships through the feature vectors of abnormal message data, and thus improving the generalization ability of the classification model.
[0064] In this embodiment of the application, for step 101, the WAF engine can obtain the Web message and use various regular expression rules to detect the Web message, thereby determining whether the Web message is abnormal message data. For example, the WAF engine can use corresponding regular expression rules to detect the Web message based on the parameters in the Web message. The parameters in the Web message can include a variety of different parameters, and each parameter can be detected using different regular expression rules.
[0065] In this embodiment, regarding step 102, the skip-gram model is one of the distributed coding methods with powerful representation capabilities. It can map the smallest semantic unit of a text corpus into a real-valued vector and use the spatial distance between different semantic units to represent semantic similarity. Simultaneously, it can predict contextual information based on a given target character, saving time for subsequent feature extraction. Therefore, this embodiment uses the skip-gram model to densely encode the Web message parameter domain data, completing the information conversion of the Web message parameter domain data.
[0066] Furthermore, the Skip-gram model can obtain a mapping table between characters and vectors, so that the feature vectors corresponding to the characters in the abnormal message data can be obtained through the mapping table.
[0067] Furthermore, as an example, for abnormal message data, the Skip-gram module is used for data feature vectorization. First, a character table V is constructed based on all possible characters appearing in the message parameters, requiring the skip-gram module to form its structure. ω is set as the number of characters in character table V, and d is the dimension of the character vector. Then, w∈R d Let V be a vector representation of ω∈V. Next, the training data (ω, c) is obtained according to the set sliding window size. When the sliding window size is set to 5, the following is obtained: Figure 2 The training data shown Figure 2 This is a schematic diagram of the scan result of a skip-gram model disclosed in an embodiment of this application. The probability of observing the character c in the context of the target character ω is:
[0068]
[0069] The probability that character c is not observed in the context of target character ω is:
[0070]
[0071] Here, vector w and vector C are the vector representations of characters ω and c, respectively.
[0072] In the skip-gram model, the hidden layers are linear neurons without activation functions, the output layer uses the softmax activation function, noise contrast estimation (NCE) is used to predict the target character, and negative sampling is used to maximize the probability of the true target. Given an experimental dataset D obtained from a sliding window scan and a corresponding negatively sampled random dataset D′, the objective function in the skip-gram model is:
[0073] J(θ)=∑ ω.c∈D Pr(D=1|ω,c)+∑ ω.c∈D ′Pr(D=0|ω,c) (3)
[0074] Calculate the dense encoded vector of the character:
[0075] w′=w·W h (4)
[0076] Where vector w′ is the dense encoding vector of character ω, vector w is the one-hot vector of character ω, and W h This is the hidden layer weight matrix obtained after training the skip-gram model.
[0077] Finally, based on the skip-gram encoder and model, the data mapping table S is calculated, thereby obtaining the feature vector of the abnormal message data.
[0078] In an optional implementation, before step 1: obtaining the abnormal message data based on the WAF engine, the method of this application embodiment further includes the following steps:
[0079] Training samples of false alarm messages were obtained from the false alarm message database;
[0080] Training samples of abnormal messages are obtained from an abnormal message database;
[0081] The feature vectors of false alarm training samples and the feature vectors of abnormal training samples are extracted based on the skip-gram model.
[0082] The feature vectors of false alarm training samples are labeled based on the WAF engine ID to obtain the first labeled data, and the feature vectors of abnormal training samples are labeled based on the WAF engine ID to obtain the second labeled data.
[0083] The model parameters of the classification model are trained based on the first and second labeled data. The model parameters of the classification model are used to identify the type of abnormal message data or the WAF engine ID associated with the abnormal message data.
[0084] In the above optional implementation, false positive message training samples can be obtained based on the false positive message database, and abnormal message training samples can be obtained based on the abnormal message database. Then, feature vectors of the false positive message training samples and the abnormal message training samples can be extracted based on the skip-gram model. Then, the feature vectors of the false positive message training samples are labeled based on the WAF engine ID to obtain first labeled data. The feature vectors of the abnormal message training samples are labeled based on the WAF engine ID to obtain second labeled data. Then, the model parameters of the classification model can be trained based on the first labeled data and the second labeled data. The model parameters of the classification model are used to identify the type of abnormal message data or the WAF engine ID associated with the abnormal message data.
[0085] In the above optional implementation, specifically, the abnormal message data detected by the WAF engine is manually judged to determine whether it is false alarm data or abnormal data. The abnormal message data content can be stored in the abnormal database, and the false alarm message data content and corresponding detection rule ID can be stored in the false alarm database and used as training data for machine learning models.
[0086] In optional implementations, the classification models include the GBDT model, the XGBoost model, and the random forest algorithm model;
[0087] And, the steps: Identifying the type of abnormal message data based on a classification model, including the following sub-steps:
[0088] Obtain the type recognition results output by the GBDT model, the XGBoost model, and the Random Forest algorithm model;
[0089] Obtain the voting value corresponding to the type recognition result output by the GBDT model and use it as the first value;
[0090] Obtain the voting value corresponding to the type recognition result output by the XGBoost model, and use it as the second value;
[0091] Obtain the voting value corresponding to the type identification result output by the random forest algorithm model, and use it as the third value;
[0092] Based on the type identification results output by the GBDT model, the XGBoost model, the Random Forest algorithm model, the first value, the second value, and the third value, the type of abnormal message data or the WAF engine ID associated with the abnormal message data is determined.
[0093] In the above optional implementation, multiple machine learning models such as GBDT model, XGBoost model and random forest model are used for comprehensive detection to determine whether the detected abnormal message data is a false alarm message or an abnormal message.
[0094] In the above optional implementation, specifically, the gradient boosting decision tree (GBDT), as an improvement on the AdaBoost algorithm, consists of two parts: a gradient boosting algorithm and a decision tree algorithm. Its core is to reduce residuals, that is, to generate a decision tree in the negative gradient direction to reduce the residuals of the previous iteration. The GBDT algorithm iteratively decreases the loss function when building the model, so that the model continuously improves in the direction of optimization.
[0095] In the aforementioned optional implementation, specifically, the Extreme Gradient Boosting (XGBoost) algorithm is a boosting algorithm based on the gradient boosting algorithm. It improves upon gradient boosting by refining the calculation method of the objective function and transforms the optimization problem of the objective function into finding the minimum of a quadratic function. It utilizes the second derivative information of the loss function to train the decision tree model, thereby accelerating training and reducing model runtime. Simultaneously, the complexity of the tree is added as a regularization term to the objective function, improving the model's generalization performance.
[0096] In the above optional implementation, specifically, the random forest algorithm model is a statistical learning theory that uses decision trees as basic classifiers. The final classification result is determined by voting on the final output results of individual decision trees. The random forest algorithm model has the following advantages: it has good tolerance for noise and outliers; it does not suffer from the overfitting problem of decision trees; and it has good scalability and parallelism for high-dimensional data classification problems. In addition, random forest is a data-driven non-parametric classification method, requiring only training on classification rules for given samples, and does not require prior knowledge of classification.
[0097] In the above optional implementation, specifically, the principle of the GBDT model is as follows:
[0098] Input: Training dataset T = {(x1,y1),(x2,y2),…,(x n ,y n Loss function L(y,f(x));
[0099] initialization The estimate is a constant value used to minimize the loss function;
[0100] For m = 1, 2, ..., M:
[0101] (a) For i = 1, 2, ..., N, calculate Treat it as a remnant
[0102] Poor estimate;
[0103] (b) for r mi Fit a regression tree to obtain the leaf node regions of the m-th tree:
[0104] R mj j = 1, 2, ..., J;
[0105] (c) Calculate for j = 1, 2, ..., J The leaf node region is estimated using a linear search, and the loss function is minimized.
[0106] (d) Update the regression tree:
[0107]
[0108] Output: Regression Tree
[0109] In the above optional implementation, specifically, the difference between the XGBoos model and the GBDT model lies in their objective function, as shown in the following formula:
[0110]
[0111] in, These are the first and second derivatives of the loss function, θ(f) t Let t be the structure of the t-th tree.
[0112] In the above optional implementation, specifically, the principle of the random forest algorithm model is as follows:
[0113] 1. Use the Bootstraping method to randomly sample m samples with replacement from the original training set, perform n_tree sampling times in total, and generate n_tree training sets.
[0114] 2. For each of the n_tree training sets, train n_tree decision tree models respectively.
[0115] 3. For a single decision tree model, assuming the number of training sample features is n, the best feature is selected for splitting based on the Gini index at each split. For a general decision tree, if there are a total of K classes, the probability that a sample belongs to the k-th class is: p k .
[0116]
[0117] 4. Each tree is known to continue splitting in this way, and it is known that all training examples at that node belong to the same class. No pruning is required during the splitting process of the decision tree.
[0118] 5. Combine the generated decision trees into a random forest.
[0119] In an optional implementation, the step of extracting feature vectors from the abnormal message data using the skip-gram model includes:
[0120] Extract the message parameter field data from the abnormal message data;
[0121] The skip-gram model is used to convert message parameter field data into feature vectors of abnormal message data. The skip-gram model learns the mapping relationship between message parameter field data and feature vectors in advance.
[0122] In the above optional implementation, by extracting the message parameter field data from the abnormal message data, the message parameter field data can be converted into a feature vector of the abnormal message data based on the skip-gram model, wherein the skip-gram model pre-learns the mapping relationship between the message parameter field data and the feature vector.
[0123] In an optional implementation, the method of this application embodiment further includes the following steps:
[0124] An alarm message is generated when the type of the abnormal message data is the first type.
[0125] In the above optional implementation, when the type of abnormal message data is the first type, alarm information can be generated to ensure that the abnormal message can trigger the alarm normally.
[0126] In an optional implementation, the method of this application embodiment further includes the following steps:
[0127] When the type of the abnormal message data is the second type, the abnormal message data will be sent to the terminal where the operation and maintenance personnel are located.
[0128] In an optional implementation, the step of obtaining abnormal message data based on the WAF engine specifically includes:
[0129] Acquire traffic data and extract web message data based on the traffic data;
[0130] Web message data is detected based on the WAF engine. When the parameters in the web message data match those of the WAF engine, the web message data is identified as abnormal message data.
[0131] In the above optional implementation, by acquiring traffic data and extracting WEB packet data based on the traffic data, it is possible to detect WEB packet data based on the WAF engine. Thus, when the parameters in the WEB packet data match the WAF engine, the WEB packet data is identified as abnormal packet data.
[0132] Example 2
[0133] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a message data detection device disclosed in an embodiment of this application, as shown below. Figure 3 As shown, the apparatus in this embodiment includes the following functional modules:
[0134] Module 201 is used to acquire abnormal message data obtained based on the WAF engine;
[0135] Feature extraction module 202 is used to extract feature vectors from abnormal message data based on the skip-gram model;
[0136] The determination module 203 is used to determine the detection result of abnormal message data based on the classification model. The detection result of abnormal message data is the type of abnormal message data or the WAF engine ID associated with the abnormal message data. The feature vector of the abnormal message is used as the input of the classification model, and the type of abnormal message data is the output of the classification model. The types of abnormal message data include a first type and a second type. The first type indicates that the abnormal message data is an abnormal message, and the second type indicates that the abnormal message data is a false alarm message.
[0137] The apparatus in this embodiment of the application can acquire abnormal message data based on a WAF engine by executing a message data detection method. It can then extract feature vectors of the abnormal message data based on a skip-gram model, determine the detection result of the abnormal message data based on a classification model, and finally determine whether the abnormal message data is an abnormal message or a false alarm message based on the detection result. When the abnormal message data is detected as a false alarm message, the classification result of the abnormal message data can be corrected to prevent the abnormal message data from being treated as an abnormal message, thereby reducing the false alarm rate of message detection.
[0138] It should be noted that for other detailed descriptions of the apparatus in the embodiments of this application, please refer to the relevant description in Embodiment 1 of this application, which will not be repeated in the embodiments of this application.
[0139] Example 3
[0140] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application, such as... Figure 4As shown, the electronic device in this application embodiment includes:
[0141] Processor 301; and
[0142] The memory 302 is configured to store machine-readable instructions, which, when executed by the processor 301, perform a message data detection method as described in any of the foregoing embodiments.
[0143] The electronic device in this embodiment of the application can obtain abnormal message data based on the WAF engine by executing the message data detection method. Then, it can extract the feature vector of the abnormal message data based on the skip-gram model. Then, it can determine the detection result of the abnormal message data based on the classification model. Finally, it can determine whether the abnormal message data is an abnormal message or a false alarm message based on the detection result. When the abnormal message data is detected as a false alarm message, the classification result of the abnormal message data can be corrected to prevent the abnormal message data from being regarded as an abnormal message. In this way, the false alarm rate of message detection is reduced.
[0144] Fourthly, the present invention provides a storage medium storing a computer program, the computer program being executed by a processor as a message data detection method according to any of the foregoing embodiments.
[0145] The storage medium of the fourth aspect of this application, by executing a message data detection method, can acquire abnormal message data obtained based on a WAF engine, then extract feature vectors of the abnormal message data based on a skip-gram model, and then determine the detection result of the abnormal message data based on a classification model. Finally, based on the detection result, it can determine whether the abnormal message data is an abnormal message or a false alarm message. When the abnormal message data is detected as a false alarm message, the classification result of the abnormal message data can be corrected to prevent the abnormal message data from being treated as an abnormal message, thereby reducing the false alarm rate of message detection.
[0146] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0147] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0148] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0149] It should be noted that if a function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0150] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.
[0151] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A message data detection method, characterized in that, The method includes: Obtain exception message data based on the WAF engine; The feature vector of the abnormal message data is extracted based on the skip-gram model; The detection result of the abnormal message data is determined based on a classification model, wherein the detection result of the abnormal message data is the type of the abnormal message data or the WAF engine ID associated with the abnormal message data, wherein the feature vector of the abnormal message is used as the input of the classification model, and the type of the abnormal message data is the output of the classification model. The type of the abnormal message data includes a first type and a second type, wherein the first type indicates that the abnormal message data is an abnormal message, and the second type indicates that the abnormal message data is a false alarm message. Furthermore, before obtaining the exception message data based on the WAF engine, the method further includes: Training samples of false alarm messages were obtained from the false alarm message database; Training samples of abnormal messages are obtained from an abnormal message database; Based on the skip-gram model, feature vectors are extracted from the training samples of false alarm messages and feature vectors are extracted from the training samples of abnormal messages. The skip-gram model constructs a character table V based on all possible characters in the message parameters, sets ω to a character in character table V, and then... The feature vector of the false alarm message training sample is obtained by the dense coding vector, and the skip-gram model directly extracts the feature vector of the false alarm message training sample; The feature vectors of the false alarm training samples are labeled based on the ID of the WAF engine to obtain the first labeled data, and the feature vectors of the abnormal message training samples are labeled based on the ID of the WAF engine to obtain the second labeled data. The model parameters of the classification model are trained based on the first labeled data and the second labeled data. The model parameters of the classification model are used to identify the type of the abnormal message data or the WAF engine ID associated with the abnormal message data.
2. The method as described in claim 1, characterized in that, The classification models include the GBDT model, the XGBoost model, and the random forest algorithm model; And, the identification of the type of the abnormal message data based on the classification model includes: Obtain the type identification results output by the GBDT model, the XGBoost model, and the Random Forest algorithm model; Obtain the voting value corresponding to the type recognition result output by the GBDT model, and use it as the first value; Obtain the voting value corresponding to the type identification result output by the XGBoost model, and use it as the second value; Obtain the voting value corresponding to the type identification result output by the random forest algorithm model, and use it as the third value; Based on the type identification results output by the GBDT model, the type identification results output by the XGBoost model, the type identification results output by the Random Forest algorithm model, the first value, the second value, and the third value, the type of the abnormal message data or the WAF engine ID associated with the abnormal message data is determined.
3. The method as described in claim 1, characterized in that, The skip-gram model extracts feature vectors from the abnormal message data, including: Extract the message parameter field data from the aforementioned abnormal message data; The message parameter field data is converted into feature vectors of the abnormal message data based on the skip-gram model, wherein the skip-gram model learns in advance the mapping relationship between the message parameter field data and the feature vectors.
4. The method as described in claim 1, characterized in that, The method further includes: When the type of the abnormal message data is the first type, an alarm message is generated.
5. The method as described in claim 4, characterized in that, The method further includes: When the type of the abnormal message data is the second type, the abnormal message data is sent to the terminal where the operation and maintenance personnel are located.
6. The method as described in claim 1, characterized in that, The process of obtaining abnormal message data based on the WAF engine also includes: Acquire traffic data and extract WEB message data based on the traffic data; The WAF engine detects the WEB message data, and when the parameters in the WEB message data match the WAF engine, the WEB message data is identified as the abnormal message data.
7. A message data detection device, characterized in that, The device includes: The acquisition module is used to acquire exception message data obtained based on the WAF engine; The feature extraction module is used to extract feature vectors from the abnormal message data based on the skip-gram model, wherein the skip-gram model directly extracts feature vectors from the training samples of false alarm messages. The determination module is used to determine the detection result of the abnormal message data based on the classification model. The detection result of the abnormal message data is the type of the abnormal message data or the WAF engine ID associated with the abnormal message data. The feature vector of the abnormal message is used as the input of the classification model, and the type of the abnormal message data is the output of the classification model. The type of the abnormal message data includes a first type and a second type. The first type indicates that the abnormal message data is an abnormal message, and the second type indicates that the abnormal message data is a false alarm message. Before acquiring the exception message data obtained based on the WAF engine, the device is also used to: Training samples of false alarm messages were obtained from the false alarm message database; Training samples of abnormal messages are obtained from an abnormal message database; Based on the skip-gram model, feature vectors are extracted from training samples of false positive messages and feature vectors from training samples of abnormal messages. The skip-gram model constructs a character table V based on all possible characters in the message parameters, sets ω to a character in character table V, and then... The feature vector of the false alarm message training sample is obtained from the dense coding vector; The feature vectors of the false alarm training samples are labeled based on the ID of the WAF engine to obtain the first labeled data, and the feature vectors of the abnormal message training samples are labeled based on the ID of the WAF engine to obtain the second labeled data. The model parameters of the classification model are trained based on the first labeled data and the second labeled data. The model parameters of the classification model are used to identify the type of the abnormal message data or the WAF engine ID associated with the abnormal message data.
8. An electronic device, characterized in that, include: processor; as well as A memory configured to store machine-readable instructions that, when executed by the processor, perform the message data detection method as described in any one of claims 1-6.
9. A storage medium, characterized in that, The storage medium stores a computer program, which is executed by a processor according to the message data detection method as described in any one of claims 1-6.