Task attack surface scanning rule weight prediction method, model, system, medium and equipment

Abstract

The invention discloses a task attack surface scanning rule weight prediction method, model, system, medium and equipment, and the system comprises a web page similarity discrimination unit and a task attack surface scanning rule weight prediction unit. And the web page similarity judgment unit sends data information of a web page which is not similar to the task target to the task attack surface scanning rule weight prediction unit for task attack surface scanning rule weight prediction. The problems that a large number of useless data packets are sent and task execution time is long in the automatic vulnerability detection process can be solved, the detection weight can be automatically and intelligently generated according to data such as the task process, task identification assets and task output information, the detection efficiency can be greatly improved through the weight, and the detection efficiency is improved. And useless packet sending quantity and task execution time are reduced.

Description

Technical Field

[0001] This invention relates to the field of network security technology, specifically to a method, model, system, medium, and device for predicting the weight of attack surface scanning rules. Background Technology

[0002] Vulnerability detection involves using different types of Proof-of-Concept (POC) to verify vulnerabilities based on different products and application types. Traditionally, POCs are generated based on website fingerprints or by blind testing to verify the authenticity of vulnerabilities.

[0003] Vulnerability detection uses either fingerprinting or blind scanning methods. Most products on the market currently execute tasks based on these two logics. Blind scanning sends a lot of useless data packets, which may cause network congestion and affect the use of online applications. Fingerprinting relies entirely on fingerprint recognition capabilities. Many websites have undergone secondary development, which may lead to recognition errors or failure to recognize fingerprints, easily resulting in missed or false vulnerability reports. Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is to provide a method, model, system, medium and device for predicting the rule weight of a task attack surface. This method can solve the problems of sending a large number of useless data packets and long task execution time during automated vulnerability detection. It can also automatically and intelligently generate detection weights based on data such as task process, task-identified assets and task output information. These weights can greatly improve detection efficiency and reduce the number of useless packets and task execution time.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] The mission attack surface scanning rule weight prediction model is obtained through the following steps:

[0007] S101) Set up a vulnerability testing range to simulate attack behavior and collect attack data;

[0008] S102) Perform feature engineering processing on the attack data collected in step S101);

[0009] S103) Train the attack data processed in step S102) to obtain the attack surface scanning rule weight prediction model.

[0010] In the above-mentioned attack surface scanning rule weight prediction model, in step S102), the feature engineering processing of the attack data collected in step S101) includes dirty data cleaning, missing value completion, feature importance ranking, and selection and data augmentation for data imbalance; the format of the processed attack data is a unified data format that can be used by the algorithm model.

[0011] In step S103), the above-mentioned task attack surface scanning rule weight prediction model uses the XGBoost algorithm to train and learn the attack data processed in step S102), and uses cross-validation to filter the parameters of the task attack surface scanning rule weight prediction model. The selected parameters are the optimal parameters.

[0012] The aforementioned attack surface scanning rule weight prediction model, when training the attack data processed by step S102) using the XGBoost algorithm, employs a greedy algorithm to optimally select the features of the objective function in the XGBoost algorithm.

[0013] The method for predicting the weights of mission attack surface scanning rules using the aforementioned mission attack surface scanning rule weight prediction model includes the following steps:

[0014] S201) Perform a similarity assessment between the task objective and the web page generated by the user's IP address;

[0015] S202) The task attack surface scanning rule weight prediction model is used to predict the task attack surface scanning rule weights for web pages generated by user IPs that are not similar to the task target.

[0016] In step S201 of the above method, cosine similarity is used to determine the similarity between the task target and the web page generated by the user's IP address. The specific steps are as follows:

[0017] S201-1) Crawl page data from web pages generated by user IP addresses;

[0018] S201-2) Extract the page structure and internal page content from the page data crawled in step S201-1) and obtain the corresponding vector data;

[0019] S201-3) Using the vector data obtained in step S201-2), calculate the page structure similarity value and page content similarity value between the task target and the web page generated by the user IP; wherein, the page structure similarity value and page content similarity value are calculated using the following formula:

[0020] ;

[0021] ;

[0022] ;

[0023] In the formula, α and β are both adjustable preset parameters, and 1.8≤α+β≤2; The average value of all vectors A; The average value of all vectors B;

[0024] S201-4) Using the page structure similarity values ​​and page content similarity values ​​calculated in step S201-3), determine the page structure similarity threshold range and page content similarity threshold range between two identical pages. If, in subsequent use, the page structure similarity value and page content similarity value between the task target and the web page generated by the user IP are both outside the page structure similarity threshold range and page content similarity threshold range, then the task target and the web page generated by the user IP are considered dissimilar.

[0025] In the above method, in step S201-2), the BS4 and XPath frameworks are used to extract the DOM tree structure and page content of the crawled web page data. Then, the nodes of the DOM tree structure are uniformly encoded, and each web page is uniformly formed into a 1024-dimensional vector. The page content is then processed to remove special symbols, high-frequency words and low-frequency words. Finally, the bag-of-words model is used to encode the page content to form a 512-dimensional vector.

[0026] Systems that use the above method to predict attack surface scanning rule weights include:

[0027] The web page similarity discrimination unit is used to discriminate the similarity between a web page and a task target;

[0028] The task attack surface scanning rule weight prediction unit is used to predict the task attack surface scanning rule weights for web pages that are not similar to the task target.

[0029] The web page similarity discrimination unit sends data information of web pages that are not similar to the task target to the task attack surface scanning rule weight prediction unit to predict the task attack surface scanning rule weight.

[0030] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0031] A computer device includes a readable storage medium, a processor, and a computer program stored on the readable storage medium and executable on the processor, wherein the computer program, when executed by the processor, implements the above-described method.

[0032] The technical solution of the present invention achieves the following beneficial technical effects:

[0033] 1. The information generated by the vulnerability can be integrated into the entire task execution process. Compared with existing technologies, fingerprint recognition can be implemented more conveniently, and the original task execution process can be changed. Even if there is no fingerprint information in the early stage, the task process can be dynamically adjusted.

[0034] 2. By combining two artificial intelligence models, the generation of scanning rule weights becomes more accurate and flexible, while also improving the efficiency and accuracy of scanning task execution. Attached Figure Description

[0035] Figure 1 This is a flowchart of the attack surface scanning rule weight prediction model obtained in this invention;

[0036] Figure 2 This is a schematic diagram illustrating the working principle of the attack surface scanning rule weight prediction system in this invention.

[0037] Figure 3 Flowchart for attack surface scanning rule weight prediction;

[0038] Figure 4 This is a schematic diagram of a computer device capable of predicting attack surface scanning rule weights. Detailed Implementation

[0039] The present invention will be further explained below with reference to examples.

[0040] Example 1

[0041] like Figure 1 As shown, the mission attack surface scanning rule weight prediction model in this invention is obtained through the following steps:

[0042] S101) Set up a vulnerability testing range to simulate attack behavior and collect attack data;

[0043] S102) Perform feature engineering processing on the attack data collected in step S101); wherein, the feature engineering processing on the attack data collected in step S101) includes dirty data cleaning, missing value completion, feature importance ranking and selection of data augmentation for imbalanced data; the format of the processed attack data is a unified data format that can be used by the algorithm model;

[0044] S103) Train the attack data processed in step S102) to obtain the attack surface scanning rule weight prediction model.

[0045] In step S103), the XGBoost algorithm is used to train the attack data processed in step S102), and cross-validation is used to select the parameters of the task attack surface scanning rule weight prediction model. The selected parameters are the optimal parameters. Specifically, when training the attack data processed in step S102 using the XGBoost algorithm, a greedy algorithm is used to optimally select the features of the objective function in the XGBoost algorithm.

[0046] The XGBoost algorithm is an implementation of Gradient Boosting. Compared with other implementations, XGBoost has made many optimizations, such as using second-order Taylor expansion to improve training speed and accuracy, and adding regularization to the objective function to improve the model's generalization ability. After the above optimizations, the algorithm has significantly improved both the training speed and accuracy of the model.

[0047] XGBoost is a supervised learning algorithm implemented using Gradient Tree Boosting, which can solve machine learning problems such as classification and regression. Similar to Gradient Boosting, it assumes that the training dataset consists of samples (x...). i ,y i ),in , x i Let y represent an m-dimensional eigenvector. i Let K represent the sample labels. If the model contains K trees, then the XGBoost model is defined as follows:

[0048]

[0049] Let K represent the Kth decision tree. The decision tree maps the features of the samples so that each sample falls on a leaf node of the tree.

[0050] The objective function of XGBoost is defined as follows:

[0051]

[0052] The objective function Obj consists of two terms: the first is the loss function, which evaluates the loss or error between the model's predicted values ​​and the true values; this function must be a differentiable convex function. The second term is the regularization term, which controls the complexity of the model. The regularization term tends to select simpler models to avoid overfitting. The definition of the regularization term is as follows:

[0053]

[0054] First item The complexity of the tree is controlled by the number of leaf nodes and their coefficients; the larger the value, the larger the objective function, thus suppressing the complexity of the model. The second term is the L2 regularization term, used to control the weight scores of the leaf nodes.

[0055] As can be seen from the approximation of the objective function, it is necessary to find an optimal f(x) iThe goal is to optimize the objective function, but traditional methods struggle to optimize this function in Euclidean space. Therefore, XGBoost employs an approximation method to address this issue. The formula is rewritten as:

[0056]

[0057] In the formula, This is the new sub-model trained in the s-th round. XGBoost introduces the Taylor series to approximate and simplify the objective function. Let's first look at the definition of the Taylor series. The Taylor series is a formula that uses information about a function at a certain point to describe the values ​​in its vicinity. If the function curve is smooth enough, a polynomial can be constructed using the derivative values ​​at a certain point to approximate the value of the function in the neighborhood of that point. Here, we only take the second order of the Taylor expansion, defined as follows:

[0058]

[0059] Will If we consider it as Δx, we can perform a Taylor expansion of the XGBoost objective function:

[0060]

[0061] In the above formula, Statistics on the first-order gradient of the loss function; For second-order gradient statistics,

[0062] Since the constant term does not affect the optimization result, the formula can be further simplified by removing the constant term and substituting the expression for Ω(fs) into the formula, which is then rewritten as follows:

[0063]

[0064] For the objective function mentioned above, a greedy algorithm is used to select the optimal features. The model is trained through continuous iteration. Then, cross-validation is used to filter the model parameters, select the optimal parameters, and then save the model.

[0065] Example 2

[0066] like Figure 2As shown, the attack surface scanning rule weight prediction system includes a web page similarity discrimination unit and a task attack surface scanning rule weight prediction unit. The web page similarity discrimination unit sends data information of web pages that are dissimilar to the task target to the task attack surface scanning rule weight prediction unit for task attack surface scanning rule weight prediction. Specifically, the web page similarity discrimination unit is used to determine the similarity between the web page and the task target; the task attack surface scanning rule weight prediction unit is used to predict the task attack surface scanning rule weights for web pages that are dissimilar to the task target.

[0067] like Figure 3 As shown, in this embodiment, the task attack surface rule weights are predicted using the task attack surface scanning rule weight prediction model in Embodiment 1, including the following steps:

[0068] S201) Cosine similarity is used to determine the similarity between the task target and the web page generated by the user's IP address. The specific steps are as follows:

[0069] S201-1) Crawl page data from web pages generated by user IP addresses;

[0070] S201-2) Extract the page structure and internal page content from the page data crawled in step S201-1) and obtain the corresponding vector data. Specifically, the following steps are taken: use BS4 and XPath frameworks to extract the page DOM tree structure and page content from the crawled web page data. Then, uniformly encode the nodes of the page DOM tree structure, forming a 1024-dimensional vector for each web page. Dirty data processing is performed on the page content to remove special symbols, high-frequency words, and low-frequency words. Then, the bag-of-words model is used to encode the page content to form a 512-dimensional vector.

[0071] S201-3) Using the vector data obtained in step S201-2), calculate the page structure similarity value and page content similarity value between the task target and the web page generated by the user IP; wherein, the page structure similarity value and page content similarity value are calculated using the following formula:

[0072] ;

[0073] ;

[0074] ;

[0075] In the formula, α and β are both adjustable preset parameters, and 1.8≤α+β≤2; The average value of all vectors A, preferably the arithmetic mean; Let B be the average of all vectors, preferably the average; A and B are the vector representations of two samples, respectively. and Two samples respectively at the 1st Values ​​in each dimension;

[0076] S201-4) Using the page structure similarity values ​​and page content similarity values ​​calculated in step S201-3), determine the page structure similarity threshold range and page content similarity threshold range between two identical pages. If, in subsequent use, the page structure similarity value and page content similarity value between the task target and the web page generated by the user IP are both outside the page structure similarity threshold range and page content similarity threshold range, then it is considered that the task target and the web page generated by the user IP are not similar.

[0077] S202) The task attack surface scanning rule weight prediction model is used to predict the task attack surface scanning rule weights for web pages generated by user IPs that are not similar to the task target.

[0078] In this invention, a combination of the commonly used cosine similarity algorithm and the Pearson correlation coefficient is employed to calculate the page structure similarity and page content similarity values ​​between the task target and the web page generated by the user's IP address. Compared with the similarity values ​​obtained by the cosine similarity formula used in the commonly used cosine similarity algorithm and the correlation coefficient obtained by the Pearson correlation coefficient formula, this method reduces the dependence on the distribution and characteristics of the data. It also makes up for the deficiency of cosine similarity in that it cannot be calculated when a certain dimension value is missing, reduces the impact of outliers on the final value, and improves the accuracy of the judgment.

[0079] Based on the above attack surface scanning rule weight prediction method, this example also provides a computer-readable storage medium storing a computer program. When the computer program is executed by the processor, it performs the following steps: using cosine similarity to determine the similarity between the task target and the web page generated by the user IP, and then using the task attack surface scanning rule weight prediction model to predict the task attack surface scanning rule weights for the web page generated by the user IP that is not similar to the task target.

[0080] like Figure 4As shown, based on the above attack surface scanning rule weight prediction method and computer-readable storage medium, this embodiment also provides a computer device, which includes a readable storage medium, a processor, and a computer program stored on the readable storage medium and executable on the processor. The readable storage medium and the processor are both mounted on a bus. When the processor executes the computer program, it performs the following steps: using cosine similarity to determine the similarity between the task target and the web page generated by the user IP, and then using the task attack surface scanning rule weight prediction model to predict the task attack surface scanning rule weights for the web page generated by the user IP that is not similar to the task target.

[0081] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of the claims of this patent application.

Claims

1. A task attack surface scanning rule weight prediction model, characterized in that, The mission attack surface scanning rule weight prediction model is obtained through the following steps: S101) Set up a vulnerability testing range to simulate attack behavior and collect attack data; S102) Perform feature engineering processing on the attack data collected in step S101); S103) Train the attack data processed in step S102) to obtain the attack surface scanning rule weight prediction model.

2. The task attack surface scanning rule weight prediction model according to claim 1, characterized in that, In step S102), the feature engineering processing of the attack data collected in step S101 includes dirty data cleaning, missing value completion, feature importance ranking, and data augmentation for imbalanced data. The format of the processed attack data is a unified data format that can be used by the algorithm model.

3. The mission attack surface scanning rule weight prediction model according to claim 1, characterized in that, In step S103), the XGBoost algorithm is used to train and learn the attack data processed in step S102, and cross-validation is used to filter the parameters of the task attack surface scanning rule weight prediction model. The selected parameters are the optimal parameters.

4. The mission attack surface scanning rule weight prediction model according to claim 3, characterized in that, When training the attack data processed in step S102) using the XGBoost algorithm, a greedy algorithm is used to optimally select the features of the objective function in the XGBoost algorithm.

5. A method for predicting the weights of mission attack surface scanning rules using the mission attack surface scanning rule weight prediction model according to any one of claims 1 to 4, characterized in that, Includes the following steps: S201) Perform a similarity assessment between the task objective and the web page generated by the user's IP address; S202) The task attack surface scanning rule weight prediction model is used to predict the task attack surface scanning rule weights for web pages generated by user IPs that are not similar to the task target.

6. The method according to claim 5, characterized in that, In step S201), cosine similarity is used to determine the similarity between the task target and the web page generated by the user's IP address. The specific steps are as follows: S201-1) Crawl page data from web pages generated by user IP addresses; S201-2) Extract the page structure and internal page content from the page data crawled in step S201-1) and obtain the corresponding vector data; S201-3) Using the vector data obtained in step S201-2), calculate the page structure similarity value and page content similarity value between the task target and the web page generated by the user IP; wherein, the page structure similarity value and page content similarity value are calculated using the following formula: ； ； ； In the formula, α and β are both adjustable preset parameters, and 1.8≤α+β≤2; The average value of all vectors A; The average value of all vectors B; S201-4) Using the page structure similarity values ​​and page content similarity values ​​calculated in step S201-3), determine the page structure similarity threshold range and page content similarity threshold range between two identical pages. If, in subsequent use, the page structure similarity value and page content similarity value between the task target and the web page generated by the user IP are both outside the page structure similarity threshold range and page content similarity threshold range, then the task target and the web page generated by the user IP are considered dissimilar.

7. The method according to claim 6, characterized in that, In step S201-2), the BS4 and XPath frameworks are used to extract the DOM tree structure and page content of the crawled web page data. Then, the nodes of the DOM tree structure are uniformly encoded, and each web page is uniformly formed into a 1024-dimensional vector. The page content is then processed to remove special symbols, high-frequency words and low-frequency words. Finally, the bag-of-words model is used to encode the page content to form a 512-dimensional vector.

8. A system for predicting attack surface scanning rule weights using the method described in any one of claims 5 to 7, characterized in that, include: The web page similarity discrimination unit is used to discriminate the similarity between a web page and a task target; The task attack surface scanning rule weight prediction unit is used to predict the task attack surface scanning rule weights for web pages that are not similar to the task target. The web page similarity discrimination unit sends data information of web pages that are not similar to the task target to the task attack surface scanning rule weight prediction unit to predict the task attack surface scanning rule weight.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 5 to 7.

10. A computer device, comprising a readable storage medium, a processor, and a computer program stored on the readable storage medium and executable on the processor, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 5 to 7.

Images