Integrated learning and active learning-based phishing page identification method

Abstract

The invention discloses a phishing page identification method based on ensemble learning and active learning. The method comprises the following steps: collecting a network page and processing the network page into an initial data set; clustering the initial data set to construct a cluster set; training a single-class width learning system for clusters in the cluster set, and allocating weights to construct a first integrated learning model; a buffer area is set to temporarily store unrecognizable pages, types are recognized after clustering, then a single-class width learning system is trained to be added into the first ensemble learning model, and a second ensemble learning model is formed; designing a weight adjustment mechanism to adapt to the change of the phishing page; identifying the page by using a second integrated learning model, and if the page fails, sending the page to a buffer area; and if successful, calculating a confidence score, and selecting the data tag with the highest score to determine whether the page is a phishing page. According to the method, the participation of professionals in a phishing page identification process can be reduced, an excellent identification success rate is kept, and an effective solution is provided for the field of network security.

Description

Technical Field

[0001] The present invention relates to the technical field of network security, and in particular to a phishing page recognition method based on integrated learning and active learning. Background Art

[0002] With the popularity and development of the Internet, phishing pages have become an increasingly serious problem in the field of network security. Phishing pages are fraudulent web pages that attackers use to imitate the real websites of legitimate organizations or individuals in order to trick users into revealing sensitive information.

[0003] At present, the identification methods of network attacks can be roughly divided into two categories: (1) Rule-based matching: Identify phishing web pages by presetting a series of patterns and rules. (2) Traditional machine learning-based methods: Use a large amount of historical annotated data features to build a classifier model for phishing identification. In general, existing methods can identify phishing web pages to a certain extent, but they still have some shortcomings: First, it is expensive and time-consuming to obtain a large amount of accurately annotated phishing page data because professional knowledge is required to distinguish phishing web pages from legitimate web pages. Second, phishing attack methods are constantly evolving, and existing models are easily outdated. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and proposes a phishing page recognition method based on ensemble learning and active learning. In the context of scarce labels, the method can accurately distinguish phishing web pages from legitimate web pages through an ensemble learning framework. By combining the DBSCAN density clustering algorithm with a single-class width learning system and introducing an innovative active learning strategy, the problems of insufficient phishing web page annotation data and changes in phishing web page types in the phishing web page recognition task are effectively solved.

[0005] To achieve the above purpose, the technical solution provided by the present invention is: a phishing page identification method based on ensemble learning and active learning, comprising the following steps:

[0006] S1: Collect web page data and perform preprocessing operations, including web page feature extraction and noise data removal, to obtain an initial data set of numerical features;

[0007] S2: For each category of data in the initial data set, the DBSCAN density clustering algorithm is applied to perform independent clustering to form multiple different clusters to form a cluster set;

[0008] S3: For each cluster in the cluster set, a single-class width learning system is trained respectively, and each single-class width learning system is responsible for identifying a specific class or part of the feature information of a class;

[0009] S4: assigning the same weight to each single-class width learning system, combining all single-class width learning systems together, and constructing the first integrated learning model;

[0010] S5: Setting up a buffer for storing web page data that cannot be recognized by the first integrated learning model, and continuously monitoring the data in the buffer. When the amount of data in the buffer exceeds a preset value, performing a DBSCAN density clustering algorithm on the data in the buffer. If a new cluster can be formed, using the data of the new cluster to train a new single-class width learning system, and requesting the categories of h data with the smallest reconstruction error in the new cluster to determine the specific type of the phishing page in the new cluster, and then adding the newly trained single-class width learning system to the first integrated learning model to form a second integrated learning model.

[0011] S6: Dynamically adjust the weight of each single-class width learning system in the second ensemble learning model to balance its importance, thereby coping with changes in network attack methods;

[0012] S7: For the web page to be identified, preprocess it, extract its features, remove noise, and then pass it to the second ensemble learning model for identification. If the identification fails, it is passed to the buffer for storage. If the identification succeeds, the identification confidence of each single-class width learning system in the second ensemble learning model is calculated.

[0013] S8: According to the recognition confidence results of each single-class width learning system, the web page to be recognized is classified to determine whether it is a normal page or a phishing page.

[0014] Further, in step S1, the data formed after extracting features from a certain web page is {x, y}, where x represents data composed of multiple features, y represents the corresponding label, y=0 represents that the web page corresponding to x is a normal page, and y=1 represents that the web page corresponding to x is a phishing page. The initial data set formed by collecting multiple web pages is {X train ,Y train}, where X train Represents a set of multiple data, Y train Represents a set of labels corresponding to data.

[0015] Further, in step S2, for the initial data set {X train ,Y train}, respectively, the DBSCAN density clustering algorithm is performed on the normal page data and the abnormal page data to form different clusters, forming a cluster set C = {c1, c2, c3, ..., c i ,...,c u}, where c i Represents the i-th cluster in the cluster set.

[0016] Further, the specific operation steps of step S3 are as follows:

[0017] S31: For the i-th cluster c in the cluster set i , whose internal data set is X ci First, we use randomly generated parameters to construct the feature layer Z in the single-class width learning system n , where the lth node Z in the feature layer l Constructed according to the following formula:

[0018]

[0019] In the formula, φ e Represents the activation function of the feature layer node, represents the weight vector of the lth node in the feature layer, Represents the bias vector of the lth node of the feature layer; feature layer Z n It consists of n feature nodes, namely Z n =[Z1,Z2,Z3,...,Z n ];

[0020] S32: For feature layer Z n , using randomly generated weight vectors and bias vectors to map them to the enhanced feature representation space to obtain enhanced nodes. The specific formula is expressed as:

[0021]

[0022] In the formula, H j represents the jth enhancement node, φ h represents the activation function of the enhancement layer node, represents the weight vector of the jth node in the enhancement layer, represents the bias vector of the jth node in the enhancement layer; the enhancement layer H m It consists of m enhanced nodes, namely H m =[H1,H2,H3,...,H m ]; The feature layer is combined with the enhancement layer to construct a hidden layer A that extracts deep features of the data:

[0023] A=[Z1,Z2,Z3,...,Z n ,H1,H2,H3,...,H m ]=[Z n H m ]

[0024] S33: After achieving deep feature extraction, the training goal of the single-class width learning system is to ciReconstructed, so its optimization goal is:

[0025]

[0026] In the formula, W represents the weight matrix that maps the feature information of the hidden layer to the output layer, and is therefore also the solution target; T represents the weight matrix of the data set X. ci The matrix consists of label information; λ represents the regularization coefficient; the weight W can be solved by ridge regression theory:

[0027] W=(λI+A T A) -1 A T T

[0028] Where I is the identity matrix; therefore, the data set X ci The x of the qth data q The reconstruction error e is:

[0029] e=||A(x q )W=x q ||

[0030] In the formula, A(x q ) represents the data x q Hidden layer nodes calculated;

[0031] S34: After the weight W is obtained by ridge regression theory, the single-class width learning system is used in the data set X ci The average reconstruction error MES on is:

[0032]

[0033] In the formula, |X ci | represents the data set X ci The number of data in the ith cluster c i The corresponding single-class width learning system is learned and represented as B i ;

[0034] S35: Train a single-class width learning system for each of the u clusters in the cluster set C, obtain u single-class width learning systems, and construct a single-class width learning system set [B1, B2, B3, ..., B u ].

[0035] Further, in step S4, for a single class width learning system set [B1, B2, B3, ..., B u ], assign initial weights to each single-class width learning system, and the weight ω of the g-th single-class width learning system g for:

[0036]

[0037] Thus, the first integrated learning model is constructed.

[0038] Further, the specific steps of step S5 are as follows:

[0039] S51: Setting up a buffer Buffer for temporarily storing unrecognizable web page data;

[0040] S52: For a certain web page, extract its features to form data x t , pass it to the first integrated learning model to calculate the data x t The reconstruction error with each single-class width learning system, data x t The reconstruction error e calculated in the i-th single-class width learning system is i,t for:

[0041] e i,t =||A i (x t )W i =x t ||

[0042] In the formula, A i (x t ) represents the data x t The hidden layer nodes calculated in the i-th single-class width learning system, W i represents the weight of the i-th single-class width learning system;

[0043] S53: If the data x t The reconstruction error e calculated in the i-th single-class width learning system is i,t If the average reconstruction error MES of the i-th single-class width learning system is less than , then the data x t It is successfully recognized by the i-th single-class width learning system, otherwise it is considered to have failed, that is, it cannot be recognized;

[0044] S54: If the data x t If it cannot be successfully recognized by any single-class width learning system in the first ensemble learning model, it is stored in the buffer;

[0045] S55: Continuously monitor the data in the buffer Buffer. When it is found that the amount of data in the buffer Buffer is greater than a preset value, perform the DBSCAN density clustering algorithm on the data in the buffer Buffer to form a cluster set CL = {cl1, cl2, cl3, ..., cl v ,...,cl o}, where cl v represents the vth cluster;

[0046] S56: traverse each cluster in the cluster set CL, determine whether the amount of data in the cluster is greater than a preset value, and if so, add it to the new cluster set NCL, so that the new cluster set NCL = {ncl1, ncl2, ncl3, ..., ncl r ,...,ncl p}, where ncl r Represents the rth cluster whose internal data quantity is greater than the preset value;

[0047] S57: For each cluster in the new cluster set NCL, train a single-class width learning system. The rth cluster ncl in the new cluster set r The trained single-class width learning system is B nr ;

[0048] S58: for each cluster in the new cluster set, calculating the reconstruction error of each data inside the cluster on the corresponding single-class width learning system;

[0049] S59: For the rth cluster ncl in the new cluster set r , the number of internal data is |ncl r |, calculate each data in B nr The reconstruction error on the result is:

[0050]

[0051] In the formula, Err nr Represented by each data in B nr The list of reconstruction errors on nr,k Represents ncl r The kth data in the single-class width learning system B nr The reconstruction error on Err nr The reconstruction errors in are sorted from small to large, and the data corresponding to the h smallest reconstruction errors are selected [x min1 ,x min2 ,...,x minh ], where x min1 represents the data with the smallest reconstruction error, x min2 Represents the data with the second smallest reconstruction error, x minh Represents the data with the hth smallest reconstruction error;

[0052] S510: Execute the active learning strategy, and convert the data corresponding to the h minimum reconstruction errors [x min1 ,x min2 ,...,x minh] It is handed over to professionals to judge whether the web page corresponding to these h data is a normal page or a phishing page, and then a majority vote is conducted. If the final voting result is a normal page, then cluster ncl is considered r With B nr The corresponding data label is a normal page. If the final voting result is a phishing page, then cluster ncl r With B nr The corresponding data label is phishing page;

[0053] S511: Execute an active learning strategy for each cluster in the cluster set NCL to obtain data labels for each cluster and the corresponding single-class width learning system, and add the created single-class width learning system to the first integrated learning model to obtain a second integrated learning model.

[0054] Further, the specific operation steps of step S6 are as follows:

[0055] S61: In the second ensemble learning model, each single-class width learning system is assigned a timestamp, which records the time when the single-class width learning system last recognized successful data. The sth single-class width learning system B s The corresponding timestamp is t s , with weight ω s ;

[0056] S62: When the timestamp t of the s-th single-class width learning system s Satisfy t s ≤|T′-1000|, the weight of the s-th single-class width learning system needs to be lowered, where T′ is the current moment, so the weight adjustment mechanism is:

[0057]

[0058] Further, the specific operation steps of step S7 are as follows:

[0059] S71: For a web page to be identified, extract its features to form data x t , pass it to the second ensemble learning model and calculate x t The reconstruction error with each single-class width learning system, data x t The reconstruction error e calculated in the d-th single-class width learning system d,t for:

[0060] e d,t =||A d (x t )W d -x t ||

[0061] In the formula, A d (x t ) represents the data x t The hidden layer nodes calculated in the d-th single-class width learning system, W d represents the weight of the d-th single-class width learning system;

[0062] S72: If data x t The reconstruction error e calculated in the d-th single-class width learning system d,t If the average reconstruction error of the d-th single-class width learning system is less than , then the data x t It is successfully recognized by the d-th single-class width learning system, and the timestamp of the d-th single-class width learning system is updated to the current moment;

[0063] S73: If data x t If the data x cannot be successfully recognized by any single-class width learning system in the second ensemble learning model, then the data x t Pass it to the buffer Buffer;

[0064] S74: If data x t If the data x is successfully identified by one or more single-class width learning systems in the second ensemble learning model, it can be considered that the data x t The corresponding web page is of known type, and all single-class width learning systems in the second ensemble learning model are calculated in data x t The recognition confidence on the d-th single-class width learning system B in the second ensemble learning model d For data x t The recognition confidence calculation formula is:

[0065]

[0066] Where exp represents the exponential function, ω d Represents the d-th single-class width learning system B d The weight of d,t Represents data x t In the d-th single-class width learning system B d The reconstruction error on the second ensemble learning model is all single-class width learning systems in the data x t The recognition confidence list on is CNF=[cnf(B1,x t ),cnf(B2,x t ),...,cnf(B u ,x t )].

[0067] Further, the specific operation steps of step S8 are as follows:

[0068] S81: Select the single-class width learning system corresponding to the largest recognition confidence in the list CNF and obtain its data label y cnfmax ;

[0069] S82: According to y cnfmax For data x t The corresponding web page to be identified is classified. If y cnfmax =0 means that the final recognition result of the web page to be recognized is a normal page. cnfmax =1 represents that the final identification result of the web page to be identified is a phishing page.

[0070] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0071] 1. The present invention designs a phishing page identification method based on ensemble learning and active learning, which can effectively realize the identification of phishing pages.

[0072] 2. The present invention designs an active learning strategy based on reconstruction error, which significantly reduces the need for professionals to annotate web pages.

[0073] 3. The present invention designs a recognition confidence score by combining the weight of a single-class width learning system with the reconstruction error of the data, thereby improving the recognition accuracy of unknown web pages.

[0074] 4. The present invention temporarily stores the currently unrecognizable web page data by setting up a buffer, and then discovers the potential new web page types in the buffer through the DBSCAN density clustering algorithm, thereby ensuring the updating and continuous learning of the model and adapting to the changes in phishing methods.

[0075] 5. The present invention utilizes unlabeled web page data to train a single-class width learning system, thereby solving the dependence of traditional machine learning models on web page annotations.

[0076] In summary, the present invention learns the consistency of feature space learning based on clustering and single-class width learning systems, reduces the dependence on labeled web pages during training, and designs recognition confidence scores based on weights and reconstruction errors to efficiently identify unknown web pages. At the same time, a buffer is set up to save unrecognizable web pages, and new types of web pages are discovered and learned based on clustering algorithms and designed active learning mechanisms, thereby realizing the identification of phishing pages and timely detection and response to changes in phishing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is a framework diagram of the method of the present invention.

[0078] Figure 2Framework diagram of a single-class width learning system. DETAILED DESCRIPTION

[0079] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0080] like Figure 1 As shown, this embodiment discloses a phishing page identification method based on integrated learning and active learning, which includes the following steps:

[0081] 1) Initial dataset construction:

[0082] Collect relevant feature data of web pages, including subdomain level, path level, the number of special characters in URL, and other features, and perform necessary preprocessing operations, such as feature extraction and noise data removal, to form an initial data set suitable for model training, represented as {X train ,Y train}. Where X train Represents a set of multiple data, Y train Represents a set of labels corresponding to data. A single data can be represented as {x, y}, where x represents data composed of multiple features and y represents the corresponding label. y=0 means that the web page corresponding to x is a normal page, while y=1 means that the web page corresponding to x is a phishing page.

[0083] 2) Single-class clustering mechanism:

[0084] The initial data set {X train ,Y train} is divided into normal web page dataset {X0, Y0} and phishing page dataset {X1, Y1} according to different categories. X0 represents the train The set of data corresponding to all normal web pages in X train The set of labels of data corresponding to all normal web pages in X1. train The set of data corresponding to all phishing pages in X train A collection of labels for data corresponding to all phishing pages in .

[0085] The DBSCAN density clustering algorithm is performed on the normal web page dataset {X0, Y0} and the phishing page dataset {X1, Y1} respectively, and all the clusters found are constructed into a cluster set C = {c1, c2, c3, ..., c i ,...,c u}. Where C represents the cluster set, c i Represents the i-th cluster in the cluster set.

[0086] 3) Training a single-class width learning system:

[0087] For the cluster set C of training data, each cluster contains data from a specific area in the feature space. For each specific area, a single-class width learning system needs to be built to learn the feature pattern of the area, so as to identify whether the unknown web page belongs to this type and pattern in the subsequent process. The framework of the single-class width learning system is shown in Figure 2 The specific construction steps are as follows:

[0088] 3.1) For the i-th cluster c in the cluster set i , whose internal data set is X ci First, we use randomly generated parameters to construct the feature layer Z in the single-class width learning system. n . The lth node Z in the feature layer l Constructed according to the following formula:

[0089]

[0090] where φ e Represents the activation function of the feature layer node, represents the weight vector of the lth node in the feature layer, Represents the bias vector of the lth node in the feature layer. Feature layer Z n It consists of n feature nodes, namely Z n =[Z1,Z2,Z3,...,Z n ];

[0091] 3.2) For feature layer Z n , and also use the randomly generated weight vector and bias vector to map it to the enhanced feature representation space to obtain the enhanced node. Specifically:

[0092]

[0093] Among them, H j represents the jth enhancement node, φ h represents the activation function of the enhancement layer node, represents the weight vector of the jth node in the enhancement layer, Represents the bias vector of the jth node in the enhancement layer. m It consists of m enhanced nodes, namely H m =[H1,H2,H3,...,H m ]. The feature layer is combined with the enhancement layer to construct a hidden layer A that extracts deep features of the data:

[0094] A=[Z1,Z2,Z3,...,Z n ,H1,H2,H3,...,Hm ]=[Z n H m ]

[0095] 3.3) After achieving deep feature extraction, the training goal of the single-class width learning system is to ci Reconstructed, so its optimization goal is:

[0096]

[0097] Where W represents the weight matrix that maps the feature information of the hidden layer to the output layer, and is therefore the solution target. T represents the weight matrix of the data set X. ci The matrix is ​​composed of label information. λ represents the regularization coefficient. The weight W can be solved by ridge regression theory:

[0098] W=(λI+A T A) -1 A T T

[0099] Where I is the identity matrix. Therefore, the data set X ci The x of the qth data q The reconstruction error is:

[0100] e=||A(x q )Wx q ||

[0101] Among them, A(x q ) represents the data x q Hidden layer nodes for calculation.

[0102] 3.4) After the weight W is obtained by ridge regression theory, the single-class width learning system is used in the data set X ci The average reconstruction error MES on is:

[0103]

[0104] Among them, |X ci | represents the data set X ci The number of data in the i-th cluster c i The corresponding single-class width learning system is learned and represented as B i .

[0105] 3.5) Train a single-class width learning system for each of the u clusters in the cluster set C, and obtain u single-class width learning systems, which are then constructed into a set of single-class width learning systems [B1, B2, B3, ..., B u ].

[0106] 4) Weight allocation and construction of the first ensemble learning model:

[0107] For a single-class width learning system set [B1,B2,B3,...,B u ], assign initial weights to each single-class width learning system. The weight of the g-th single-class width learning system is:

[0108]

[0109] Thus, the first integrated learning model is constructed.

[0110] 5) Buffer mechanism and new network page type discovery mechanism:

[0111] Since the types of web phishing pages will change, criminals will innovate new types of phishing pages, so it is necessary to learn about new types of phishing pages. The specific steps are:

[0112] 5.1) Set up a buffer buffer to temporarily store unrecognizable web page data.

[0113] 5.2) For a certain web page, extract its features to form data x t , pass it to the first ensemble learning model, and calculate the reconstruction error between the data and each single-class width learning system. Data x t The reconstruction error e calculated in the i-th single-class width learning system is i,t for:

[0114] e i,t =||A i (x t )W i -x t ||

[0115] Among them A i (x t ) represents the data x t The hidden layer nodes calculated in the i-th single-class width learning system, W i Represents the weight of the i-th single-class width learning system.

[0116] 5.3) If the data x t The reconstruction error e calculated in the i-th single-class width learning system is i,t If the average reconstruction error MES of the i-th single-class width learning system is less than t Successfully recognized by the i-th single-class width learning system.

[0117] 5.4) If the sample x t If it cannot be successfully recognized by any single-category width learning system in the first integrated learning model, it will be stored in the buffer.

[0118] 5.5) Continuously monitor the data in the buffer. When the number of data in the buffer is greater than 50, perform the DBSCAN density clustering algorithm on the data in the buffer to form a cluster set CL = {cl1, cl2, cl3, ..., cl v ,...,cl o}, where cl v represents the vth cluster.

[0119] 5.6) Traverse each cluster in the cluster set CL and determine whether the number of data in the cluster is greater than 15. If so, add it to the new cluster set NCL. The new cluster set NCL is NCL = {ncl1, ncl2, ncl3, ..., ncl r ,...,ncl p}, where ncl r Represents the rth cluster whose internal data quantity is greater than 15.

[0120] 5.7) For each cluster in the new cluster set NCL, train a single-class width learning system. The rth cluster ncl in the new cluster set r The trained single-class width learning system is B nr .

[0121] 5.8) For each cluster in the new cluster set, calculate the reconstruction error of each sample within it on the corresponding single-class width learning system.

[0122] 5.9) For the rth cluster ncl in the new cluster set r , the number of internal data is |ncl r |, calculate each data in B nr The reconstruction error on the result is:

[0123]

[0124] In the formula, Err nr Represented by each data in B nr The list of reconstruction errors on nr,k Represents ncl r The kth data in the single-class width learning system B nr The reconstruction error on Err nr The reconstruction errors in are sorted from small to large, and the data corresponding to the three smallest reconstruction errors are selected [x min1 ,x min2 ,x min3 ], where x min1 represents the data with the smallest reconstruction error, x min2 Represents the data with the second smallest reconstruction error, xmin3 Represents the data with the third smallest reconstruction error.

[0125] 5.10) Execute the active learning strategy and convert the data corresponding to the three minimum reconstruction errors [x min1 ,x min2 ,x min3 ] It is handed over to professionals to judge whether the web pages corresponding to these three data are normal pages or phishing pages, and then a majority vote is conducted. If the final voting result is a normal page, then cluster ncl is considered r With B nr The corresponding data label is a normal page. If the final voting result is a phishing page, then cluster ncl r With B nr The corresponding data label is phishing page.

[0126] 5.11) Execute an active learning strategy for each cluster in the cluster set NCL to obtain the data label of each cluster and the corresponding single-class width learning system, and add the created single-class width learning system to the first integrated learning model to obtain the second integrated learning model.

[0127] 6) Weight adjustment mechanism:

[0128] Since phishing methods will change, the types of phishing web pages will also change constantly, which will cause some single-class width learning systems to no longer adapt, so their weights need to be adjusted to prevent them from making wrong judgments. The specific steps are:

[0129] 6.1) In the second ensemble learning model, each single-class width learning system is assigned a timestamp, which records the moment when the single-class width learning system last recognized successful data. s The corresponding timestamp is t s , with weight ω s .

[0130] 6.2) When the timestamp t of the s-th single-class width learning system s Satisfy t s ≤|T′-1000|, the weight of the s-th single-class width learning system needs to be lowered, where T is the current moment. Therefore, the weight adjustment mechanism is:

[0131]

[0132] 7) Confidence of identifying unknown web pages:

[0133] For unknown web pages to be identified, it is necessary to determine whether they can be identified by the second integrated learning model. If the identification is successful, it is considered to be an existing type of web page, otherwise it is considered to be a new type of web page. The specific operation steps are:

[0134] 7.1) For a web page to be identified, extract its features to form data x t , pass it to the second ensemble learning model and calculate x t Reconstruction error with each single-class width learning system. Data x t The reconstruction error e calculated in the d-th single-class width learning system d,t for:

[0135] e d,t =||A d (x t )W d -x t ||

[0136] Among them A d (x t ) represents the data x t The hidden layer nodes calculated in the d-th single-class width learning system, W d represents the weight of the d-th single-class width learning system.

[0137] 7.2) If the data x t The reconstruction error e calculated in the d-th single-class width learning system d,t If the average reconstruction error of the d-th single-class width learning system is less than , then the data x t It is successfully recognized by the d-th single-class width learning system, and the timestamp of the d-th single-class width learning system is updated to the current moment.

[0138] 7.3) If the data x t If the data x cannot be successfully recognized by any single-class width learning system in the second ensemble learning model, then the data x t Passed to the buffer Buffer.

[0139] 7.4) If the data x t If the data x is successfully identified by one or more single-class width learning systems in the second ensemble learning model, it can be considered that the data x t The corresponding web pages are of known types. Calculate the width of all single-class learning systems in the second ensemble learning model in the data x t The recognition confidence on the d-th single-class width learning system B in the second ensemble learning model d For data x t The recognition confidence calculation formula is:

[0140]

[0141] Among them, exp represents the exponential function, ω d Represents the d-th single-class width learning system B d The weight of d,t Represents data x t In the d-th single-class width learning system B d The reconstruction error on the second ensemble learning model is that all single-class width learning systems are in the data x t The recognition confidence list on is CNF=[cnf(B1,x t ),cnf(B2,x t ),...,cnf(B u ,x t )].

[0142] 8) Identify successful web page categories:

[0143] For the web pages that are successfully identified, it is necessary to classify them according to the identification situation within the second ensemble learning model to determine whether they are phishing pages. The specific steps are as follows:

[0144] 8.1) Select the single-class width learning system corresponding to the largest recognition confidence in the list CNF and obtain its data label y cnfmax .

[0145] 8.2) According to y cnfmax For sample x t The corresponding web page to be identified is classified. cnfmax =0 means that the final recognition result of the web page to be recognized is a normal page. cnfmax =1 represents that the final identification result of the web page to be identified is a phishing page.

[0146] The Phishing dataset contains 48 features of 5,000 phishing pages and 5,000 legitimate pages downloaded from January to June 2015 and May to June 2017, such as subdomain level, path level, URL length, the number of various symbols in the URL, etc., which is designed to help researchers analyze phishing features, quickly verify concepts or benchmark classification models. In this experiment, the first 500 data of the Phishing dataset are selected as the initial dataset for model initialization. Subsequent samples are passed to the model one by one for processing according to the online learning processing method, simulating the scenario of analyzing phishing pages one by one in real applications.

[0147] Accuracy is one of the common indicators used to evaluate the performance of classification models in machine learning. It is a simple and intuitive indicator used to measure the proportion of samples predicted correctly by the model to the total number of samples. This experiment uses the accuracy Acc as the evaluation indicator. The accuracy of model M is:

[0148]

[0149] Among them, TP is the number of samples correctly predicted as positive in the positive class; TN is the number of samples correctly predicted as negative in the negative class; FP is the number of samples incorrectly predicted as positive in the negative class; FN is the number of samples incorrectly predicted as negative in the positive class.

[0150] To verify the effectiveness of the method of the present invention, the accuracy rate is used as the evaluation index, and a comparative analysis is conducted with ALRandom, ALFixUn, ALVarUn, ALRandVarUn, ALSelSampling, ACLStream, and SmSCluster methods on the Phishing dataset containing multiple phishing pages and legitimate pages. The annotation budget of the comparison method is set to 10%. The experimental results are shown in Table 1:

[0151] Table 1 Experimental results

[0152] method Accuracy Request annotation ratio (%) ALRandom 0.88 10 ALFixUn 0.87 10 AL V Ar Un 0.90 10 ALRandVarUn 0.90 10 ALSelSampling 0.89 10 ACLStream 0.85 10 SmSCluster 0.51 10 Method of the present invention 0.929 1.3

[0153] The experimental results can fully demonstrate the effectiveness of the present invention. Through the consistency of the DBSCAN density clustering algorithm and the single-class width learning system, multiple regions of the feature space are independently learned, and the efficient and fast feature extraction ability of the single-class width learning system is used to deeply extract the data feature information in the region for learning and identification. Multiple single-class width learning systems are constructed into an integrated learning model, and a weight adjustment mechanism is designed to reasonably combine the capabilities of multiple single-class width learning systems to efficiently and correctly identify the categories of web pages. A buffer mechanism is designed to retain web pages of unknown types, and an active learning strategy is designed at the same time. When a new type of web page is discovered, a representative sample is selected to request professionals to identify it to determine whether it is a phishing page, and new types of web pages are identified and learned while minimizing the participation of professionals. Through the above steps, the feature information of various web pages can be fully mined, different types of web pages can be efficiently identified, and the success rate of identifying phishing pages can be improved.

[0154] Experimental conclusion: In view of the problem that existing algorithms are difficult to maintain recognition accuracy in scenarios where web page labels are scarce and phishing methods are frequently improved, the present invention proposes a phishing page recognition method based on ensemble learning and active learning. Experimental evaluation on a public Phishing dataset shows that the use of the method of the present invention can maintain a better accuracy than existing methods while having a lower annotation rate. Therefore, the method of the present invention can allow the model to learn adaptively and maintain the success rate of phishing page recognition while minimizing the participation of professionals in web page recognition. In subsequent research, more phishing method recognition problems will be explored, such as SMS and email phishing, social media phishing, etc. It has a good application prospect and is worth promoting.

[0155] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A phishing page recognition method based on ensemble learning and active learning, characterized in that: The steps include: S1: Collect web page data and perform preprocessing operations, including web page feature extraction and noise data removal, to obtain an initial data set of numerical features; S2: For each category of data in the initial data set, the DBSCAN density clustering algorithm is applied to perform independent clustering to form multiple different clusters to form a cluster set; S3: For each cluster in the cluster set, a single-class width learning system is trained respectively, and each single-class width learning system is responsible for identifying a specific class or part of the feature information of a class; S4: assigning the same weight to each single-class width learning system, combining all single-class width learning systems together, and constructing the first integrated learning model; S5: Setting up a buffer for storing web page data that cannot be recognized by the first integrated learning model, and continuously monitoring the data in the buffer. When the amount of data in the buffer exceeds a preset value, performing a DBSCAN density clustering algorithm on the data in the buffer. If a new cluster can be formed, using the data of the new cluster to train a new single-class width learning system, and requesting the categories of h data with the smallest reconstruction error in the new cluster to determine the specific type of the phishing page in the new cluster, and then adding the newly trained single-class width learning system to the first integrated learning model to form a second integrated learning model. S6: Dynamically adjust the weight of each single-class width learning system in the second ensemble learning model to balance its importance, thereby coping with changes in network attack methods; S7: For the web page to be identified, preprocess it, extract its features, remove noise, and then pass it to the second ensemble learning model for identification. If the identification fails, it is passed to the buffer for storage. If the identification succeeds, the identification confidence of each single-class width learning system in the second ensemble learning model is calculated; S8: According to the recognition confidence results of each single-class width learning system, the web page to be recognized is classified to determine whether it is a normal page or a phishing page.

2. The phishing page identification method based on ensemble learning and active learning according to claim 1 is characterized in that: In step S1, the data formed after extracting features from a certain web page is {x, y}, where x represents data composed of multiple features, y represents the corresponding label, y=0 represents that the web page corresponding to x is a normal page, and y=1 represents that the web page corresponding to x is a phishing page. The initial data set formed by collecting multiple web pages is {X train , Y train }, where X train Represents a set of multiple data, Y train Represents a set of labels corresponding to data.

3. The phishing page identification method based on integrated learning and active learning according to claim 2 is characterized in that: In step S2, for the initial data set {X train , Y train }, respectively perform DBSCAN density clustering algorithm on the normal page data and abnormal page data to form different clusters, forming a cluster set C = {c1, c2, c3, ..., c i , ..., c u }, where c i Represents the i-th cluster in the cluster set.

4. The phishing page identification method based on integrated learning and active learning according to claim 3 is characterized in that: The specific operation steps of step S3 are as follows: S31: For the i-th cluster c in the cluster set i , whose internal data set is X ci First, we use randomly generated parameters to construct the feature layer Z in the single-class width learning system n , where the lth node Z in the feature layer l Constructed according to the following formula: In the formula, φ e Represents the activation function of the feature layer node, represents the weight vector of the lth node in the feature layer, Represents the bias vector of the lth node of the feature layer; feature layer Z n It consists of n feature nodes, namely Z n =[Z1,Z2,Z3,…,Z n ]; S32: For feature layer Z n , using randomly generated weight vectors and bias vectors to map them to the enhanced feature representation space to obtain enhanced nodes. The specific formula is expressed as: In the formula, H j represents the jth enhancement node, φ h represents the activation function of the enhancement layer node, represents the weight vector of the jth node in the enhancement layer, represents the bias vector of the jth node in the enhancement layer; the enhancement layer H m It consists of m enhanced nodes, namely H m =[H1, H2, H3, ..., H m ]; The feature layer is combined with the enhancement layer to construct a hidden layer A that extracts deep features of the data: <h2 style=";text-align:left;direction:ltr">A=[Z1,Z2,Z3,...,Z<h2 style=";text-align:left;direction:ltr"> n <h2 style=";text-align:left;direction:ltr"> H1, H2, H3,..., H<h2 style=";text-align:left;direction:ltr"> m <h2 style=";text-align:left;direction:ltr"> ]=[Z<h2 style=";text-align:left;direction:ltr"> n <h2 style=";text-align:left;direction:ltr"> H<h2 style=";text-align:left;direction:ltr"> m <h2 style=";text-align:left;direction:ltr"> ] S33: After achieving deep feature extraction, the training goal of the single-class width learning system is to ci Reconstructed, so its optimization goal is: In the formula, W represents the weight matrix that maps the feature information of the hidden layer to the output layer, and is therefore also the solution target; T represents the weight matrix of the data set X. ci The matrix consists of label information; λ represents the regularization coefficient; the weight W can be solved by ridge regression theory: W=((λI+A T A) -1 A T T Where I is the identity matrix; therefore, the data set X ci The x of the qth data q The reconstruction error e is: e=||A(x q )W-x q || In the formula, A(x q ) represents the data x q Hidden layer nodes calculated; S34: After the weight W is obtained by ridge regression theory, the single-class width learning system is used in the data set X ci The average reconstruction error MES on is: In the formula, |X ci | represents the data set X ci The number of data in the ith cluster c i The corresponding single-class width learning system is learned and represented as B i ; S35: train a single-class width learning system for each of the u clusters in the cluster set C, obtain u single-class width learning systems, and construct a single-class width learning system set [B1, B2, B3, ..., B u ].

5. The phishing page identification method based on integrated learning and active learning according to claim 4 is characterized in that: In step S4, a set of single-class width learning systems [B1, B2, B3, ..., B u ], assign initial weights to each single-class width learning system, and the weight ω of the g-th single-class width learning system g for: Thus, the first integrated learning model is constructed.

6. The phishing page identification method based on integrated learning and active learning according to claim 5 is characterized in that: The specific steps of step S5 are as follows: S51: Setting up a buffer Buffer for temporarily storing unrecognizable web page data; S52: For a certain web page, extract its features to form data x t , pass it to the first integrated learning model to calculate the data x t The reconstruction error with each single-class width learning system, data x t The reconstruction error e calculated in the i-th single-class width learning system is i，t for: e i，t =||A i (x t )W i -x t || In the formula, A i (x t ) represents the data x t The hidden layer nodes calculated in the i-th single-class width learning system, W i represents the weight of the i-th single-class width learning system; S53: If the data x t The reconstruction error e calculated in the i-th single-class width learning system is i，t If the average reconstruction error MES of the i-th single-class width learning system is less than , then the data x t It is successfully recognized by the i-th single-class width learning system, otherwise it is considered to have failed, that is, it cannot be recognized; S54: If the data x t If it cannot be successfully recognized by any single-class width learning system in the first ensemble learning model, it is stored in the buffer; S55: Continuously monitor the data in the buffer Buffer. When it is found that the amount of data in the buffer Buffer is greater than a preset value, perform the DBSCAN density clustering algorithm on the data in the buffer Buffer to form a cluster set CL = {cl1, cl2, cl3, ..., cl v , ..., cl o }, where cl v represents the vth cluster; S56: traverse each cluster in the cluster set CL, determine whether the amount of data in the cluster is greater than a preset value, and if so, add it to the new cluster set NCL, so that the new cluster set NCL = {ncl1, ncl2, ncl3, ..., ncl r , ..., ncl p }, where ncl r Represents the rth cluster whose internal data quantity is greater than the preset value; S57: For each cluster in the new cluster set NCL, train a single-class width learning system. The rth cluster ncl in the new cluster set r The trained single-class width learning system is B nr ; S58: for each cluster in the new cluster set, calculating the reconstruction error of each data inside the cluster on the corresponding single-class width learning system; S59: For the rth cluster ncl in the new cluster set r , the number of internal data is |ncl r |, calculate each data in B nr The reconstruction error on the result is: In the formula, Err nr Represented by each data in B nr The list of reconstruction errors on nr，k Represents ncl r The kth data in the single-class width learning system B nr The reconstruction error on Err nr The reconstruction errors in are sorted from small to large, and the data corresponding to the h smallest reconstruction errors are selected [x min1 , x min2 , ..., x min h ], where x min1 represents the data with the smallest reconstruction error, x min2 Represents the data with the second smallest reconstruction error, x min h Represents the data with the hth smallest reconstruction error; S510: Execute the active learning strategy, and convert the data corresponding to the h minimum reconstruction errors [x min1 , x min2 , ..., x min h ] It is handed over to professionals to judge whether the web page corresponding to these h data is a normal page or a phishing page, and then a majority vote is conducted. If the final voting result is a normal page, then cluster ncl is considered r With B nr The corresponding data label is a normal page. If the final voting result is a phishing page, then cluster ncl r With B nr The corresponding data label is phishing page; S511: Execute an active learning strategy for each cluster in the cluster set NCL to obtain data labels for each cluster and the corresponding single-class width learning system, and add the created single-class width learning system to the first integrated learning model to obtain a second integrated learning model.

7. The phishing page identification method based on integrated learning and active learning according to claim 6 is characterized in that: The specific operation steps of step S6 are as follows: S61: In the second ensemble learning model, each single-class width learning system is assigned a timestamp, which records the moment when the single-class width learning system last recognized successful data. The sth single-class width learning system B s The corresponding timestamp is t s , with weight ω s ; S62: When the timestamp t of the s-th single-class width learning system s Satisfy t s ≤|T′-1000|, the weight of the s-th single-class width learning system needs to be lowered, where T′ is the current moment, so the weight adjustment mechanism is:

8. The phishing page identification method based on integrated learning and active learning according to claim 7 is characterized in that: The specific operation steps of step S7 are as follows: S71: For a web page to be identified, extract its features to form data x t , pass it to the second ensemble learning model and calculate x t The reconstruction error with each single-class width learning system, data x t The reconstruction error e calculated in the d-th single-class width learning system d，t for: e d，t =||A d (x t )W d -x t || In the formula, A d (x t ) represents the data x t The hidden layer nodes calculated in the d-th single-class width learning system, W d represents the weight of the d-th single-class width learning system; S72: If data x t The reconstruction error e calculated in the d-th single-class width learning system d，t If the average reconstruction error of the d-th single-class width learning system is less than , then the data x t It is successfully recognized by the d-th single-class width learning system, and the timestamp of the d-th single-class width learning system is updated to the current moment; S73: If data x t If the data x cannot be successfully recognized by any single-class width learning system in the second ensemble learning model, then the data x t Pass it to the buffer Buffer; S74: If data x t If the data x is successfully identified by one or more single-class width learning systems in the second ensemble learning model, it can be considered that the data x t The corresponding web page is of known type, and all single-class width learning systems in the second ensemble learning model are calculated in data x t The recognition confidence on the d-th single-class width learning system B in the second ensemble learning model d For data x t The recognition confidence calculation formula is: Where exp represents the exponential function, ω d Represents the d-th single-class width learning system B d The weight of d，t Represents data x t In the d-th single-class width learning system B d The reconstruction error on the second ensemble learning model is all single-class width learning systems in the data x t The recognition confidence list on is CNF = [cnf(B1, x t ),cnf(B2,x t ), ..., cnf(B u , x t )].

9. The phishing page identification method based on integrated learning and active learning according to claim 8, characterized in that: The specific operation steps of step S8 are as follows: S81: Select the single-class width learning system corresponding to the largest recognition confidence in the list CNF and obtain its data label y cnfmax ; S82: According to y cnfmax For data x t The corresponding web page to be identified is classified. If y cnfmax =0 means that the final recognition result of the web page to be recognized is a normal page. cnfmax =1 represents that the final identification result of the web page to be identified is a phishing page.

Images