Performance evaluation method and device for abnormal website classification model, equipment and medium

Abstract

The invention provides a performance evaluation method and device for abnormal website classification models, equipment and a medium. The method comprises the following steps: calculating a case-related website detection coverage rate of each abnormal website classification model; calculating the total category coverage rate of each abnormal website classification model according to the number of the identified sub-category abnormal websites and the total number of the real sub-category abnormal websites; determining a website sampling sample set of each subcategory; calculating the sub-category prediction accuracy of each sub-category so as to determine the prediction accuracy of all the sub-categories; determining a timeliness evaluation index according to the identification duration of each website to be evaluated; and determining a comprehensive evaluation score of the model according to any one or more of the involved website detection coverage rate, the total category coverage rate, the total sub-category prediction accuracy rate and the aging evaluation index. According to the method, different indexes are designed to evaluate the evaluation effect of the abnormal website classification model, multiple indexes are used for collaborative evaluation, and the comprehensiveness of the evaluation effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of information security technology, and in particular to a performance evaluation method, device, equipment and medium for an abnormal website classification model. Background Art

[0002] Telecommunication network information security is related to the harmony and stability of social life. However, the initiators of abnormal behaviors in telecommunications networks are constantly changing their technical means, and the types of abnormal behaviors are also complex and varied, which brings greater challenges to abnormal behavior detection.

[0003] Currently, artificial intelligence models based on machine learning have been used to automatically predict whether a web page is an abnormal website. However, the classification effect of the artificial intelligence model is mainly evaluated through a single common indicator such as accuracy or F1 value. Due to the particularity of abnormal telecommunications websites, for example, abnormal websites in this field are generally hidden, the current model evaluation indicators cannot accurately reflect the true and comprehensive evaluation effect in this field. Summary of the invention

[0004] The present invention provides a performance evaluation method, device, equipment and medium for an abnormal website classification model, so as to solve the defect of a single evaluation index for the abnormal website classification effect in the prior art, and to improve the comprehensiveness and accuracy of the evaluation of the abnormal website classification effect.

[0005] The present invention provides a performance evaluation method for an abnormal website classification model, comprising the following steps: According to the number of abnormal websites identified by each abnormal website classification model in the at least one abnormal website classification model for the website set to be evaluated and the total number of actual abnormal websites, the detection coverage rate of the websites involved in the case of each abnormal website classification model in the at least one abnormal website classification model is calculated; wherein each of the abnormal website classification models is used to classify the website set to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain a plurality of sub-categories of abnormal websites; the website coverage rate involved in the case is used to indicate the degree of consistency between the number of abnormal websites identified by the abnormal website classification model and the total number of abnormal websites that actually occurred; According to the number of abnormal sub-category websites identified by each abnormal website classification model and the total number of actual abnormal sub-category websites, the overall category coverage of each abnormal website classification model is calculated respectively; wherein the overall category coverage is the weighted coverage of each sub-category; Determine a sample set of websites for each subcategory; Calculate the probability that the samples in the website sampling sample set of each subcategory are classified accurately to obtain the subcategory prediction accuracy; determine the prediction accuracy of all subcategories according to the weighted prediction accuracy of each subcategory; Determine the timeliness evaluation index of each abnormal website classification model according to the statistical index of the recognition time of each website to be evaluated in the set of websites to be evaluated by each abnormal website classification model; The comprehensive model evaluation score of each abnormal website classification model is determined based on any one or more of the detection coverage of the URLs involved in the case, the overall category coverage, the prediction accuracy of all subcategories, and the timeliness evaluation indicators.

[0006] According to a performance evaluation method for an abnormal website classification model provided by the present invention, the number of abnormal websites identified includes the number of accessible model recognition coverage websites and the number of model recognition coverage websites; the total number of abnormal websites that actually occurred includes the total number of accessible involved websites and the total number of involved websites; the method of calculating the involved website coverage rate of each abnormal website classification model in at least one abnormal website classification model according to the number of abnormal websites identified in the to-be-evaluated website set and the total number of actually occurring abnormal websites by each abnormal website classification model in at least one abnormal website classification model includes: Based on the proportion of the number of accessible model-recognized covered websites in the total number of accessible websites involved in the case, the accessible website coverage rate is obtained; wherein the number of accessible model-recognized covered websites is the number of accessible websites among the abnormal websites correctly identified by each model; the total number of accessible websites involved in the case refers to the number of websites involved in the case that actually occurred and are accessible so far in the set of websites to be evaluated; The overall coverage rate is obtained based on the proportion of the number of websites covered by the model identification in the total number of websites involved in the case; wherein the number of websites covered by the model identification refers to the number of abnormal websites correctly identified by each model from the set of websites to be evaluated; and the total number of websites involved in the case refers to the websites involved in the case that actually occurred; The coverage rate of the websites involved in the case is obtained according to the accessible website coverage rate and the overall coverage rate.

[0007] According to a performance evaluation method for an abnormal website classification model provided by the present invention, the step of determining a sample set of websites for each subcategory includes: Calculating the complexity of the website structure of each website to be sampled in the set of websites to be sampled; wherein the complexity of the website structure is determined according to the number of tags in the hypertext markup language corresponding to each website; Determine multiple initial sampling sample sets for each subcategory; Performing multiple rounds of website elimination operations on the multiple initial sampling sample sets; In each round of website elimination operation, after eliminating a preset number of website samples from the multiple initial sample sets, the variance of the number of labels in each eliminated or added sample set is calculated; According to the variance of the number of labels in each sampled sample set after elimination, the error change rate of the variance before elimination and the variance after elimination is calculated; the preset elimination number is updated; and in each round of website elimination operation, after eliminating the preset elimination number of website samples from the multiple initial sampled sample sets, the step of calculating the variance of the number of labels in each sampled sample set after elimination is returned to; Selecting the initial sampling sample set corresponding to the minimum error change rate and the preset elimination quantity corresponding to the minimum error change rate; The website sampling sample set corresponding to the current subcategory is determined according to the initial sampling sample set corresponding to the minimum error change rate and the preset elimination quantity corresponding to the minimum error change rate.

[0008] According to a performance evaluation method for an abnormal website classification model provided by the present invention, the step of determining a sample set of websites for each subcategory includes: Calculating the complexity of the website structure of each website to be sampled in the set of websites to be sampled; wherein the complexity of the website structure is determined according to the number of tags in the hypertext markup language corresponding to each website; Determine multiple initial sampling sample sets for each subcategory; Performing multiple rounds of website elimination operations on the multiple initial sampling sample sets; In each round of website elimination operation, after adding a preset number of URL samples from the multiple initial sample sets, the variance of the number of labels in each added sample set is calculated; According to the variance of the number of labels in each sampled sample set after elimination, the error change rate of the variance before the increase and the variance after elimination is calculated; the preset increase number is updated; returning to the step of calculating the variance of the number of labels in each sampled sample set after adding the preset increase number of URL samples from the multiple initial sampled sample sets in each round of website elimination operation; Selecting an initial sampling sample set corresponding to the minimum error change rate and a preset increase number corresponding to the minimum error change rate; The website sampling sample set corresponding to the current subcategory is determined according to the initial sampling sample set corresponding to the minimum error change rate and the preset increase quantity corresponding to the minimum error change rate.

[0009] According to a performance evaluation method for an abnormal website classification model provided by the present invention, the step of determining a sample set of websites for each subcategory includes: According to the time when the website appears in the network data packet, a sampling set of websites for each subcategory is determined.

[0010] According to a performance evaluation method for an abnormal website classification model provided by the present invention, the method of calculating the probability that samples in a website sampling sample set of each subcategory are accurately classified to obtain the subcategory prediction accuracy includes: Based on the number of websites classified as abnormal websites by the abnormal website classification model in the website sampling sample set of each category and the number of abnormal websites actually occurring in the website sampling sample set, the conditional probability that each subcategory is accurately classified is obtained; According to the conditional probability that each subcategory is correctly classified, the subcategory prediction accuracy of each subcategory is obtained.

[0011] The present invention also provides a performance evaluation device for an abnormal website classification model, comprising the following modules: A module for calculating the detection coverage rate of websites involved in the case is used to calculate the detection coverage rate of websites involved in the case of each abnormal website classification model in at least one abnormal website classification model according to the number of abnormal websites identified in the website set to be evaluated and the total number of actual abnormal websites; wherein each abnormal website classification model is used to classify the website set to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain multiple sub-categories of abnormal websites; the website coverage rate involved in the case is used to indicate the degree of consistency between the number of abnormal websites identified by the abnormal website classification model and the total number of abnormal websites that actually occurred; The overall category coverage calculation module is used to calculate the overall category coverage of each abnormal website classification model according to the number of subcategory abnormal websites identified by each abnormal website classification model and the total number of actual subcategory abnormal websites; wherein the overall category coverage is the weighted coverage of each subcategory; A website sampling module, used to determine a sample set of websites for each subcategory; The prediction accuracy calculation module for all subcategories is used to calculate the probability that the samples in the website sampling sample set of each subcategory are accurately classified to obtain the prediction accuracy of the subcategory; the prediction accuracy of all subcategories is determined according to the weighting of the prediction accuracy of each subcategory; A timeliness evaluation index calculation module is used to determine the timeliness evaluation index of each abnormal website classification model according to the statistical index of the recognition time of each website to be evaluated in the set of websites to be evaluated by each abnormal website classification model; The model comprehensive evaluation module is used to determine the model comprehensive evaluation score of each abnormal website classification model based on any one or more of the detection coverage of the URLs involved in the case, the overall category coverage, the prediction accuracy of all subcategories and the timeliness evaluation indicators.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a performance evaluation method for an abnormal website classification model as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a performance evaluation method for an abnormal website classification model as described in any one of the above.

[0014] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the performance evaluation method of any of the abnormal website classification models described above.

[0015] The performance evaluation method, device, equipment and medium of the abnormal website classification model provided by the present invention calculate the detection coverage rate of the involved websites of each abnormal website classification model in at least one abnormal website classification model according to the number of abnormal websites identified in the website set to be evaluated and the total number of real abnormal websites in each abnormal website classification model in at least one abnormal website classification model; wherein each abnormal website classification model is used to classify the website set to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain multiple sub-category abnormal websites; according to the number of sub-category abnormal websites identified by each abnormal website classification model and the total number of real sub-category abnormal websites, the overall category coverage rate of each abnormal website classification model is calculated respectively; Among them, the overall category coverage is the weighted coverage of each subcategory; determine the sample set of website sampling for each subcategory; calculate the probability that the samples in the sample set of website sampling for each subcategory are accurately classified to obtain the subcategory prediction accuracy; determine the prediction accuracy of all subcategories according to the weighted prediction accuracy of each subcategory; determine the timeliness evaluation index of each abnormal website classification model according to the statistical index of the recognition time of each website to be evaluated in the set of websites to be evaluated by each abnormal website classification model; determine the model comprehensive evaluation score of each abnormal website classification model according to any one or more of the detection coverage of the website involved in the case, the overall category coverage, the prediction accuracy of all subcategories and the timeliness evaluation index. This method designs different indicators to evaluate the evaluation effect of the abnormal website classification model, and can use multiple indicators for collaborative evaluation, thereby improving the comprehensiveness of the evaluation effect. It not only considers the classification of normal and abnormal websites by the model, but also considers the classification effect of different subcategories in abnormal websites, thereby improving the accuracy of model evaluation. Furthermore, this application also uses a multi-URL collaborative sampling mechanism, which can effectively discover and solve the problem of differences in classification effect evaluation caused by reasons such as imbalanced sample categories, and significantly improve the credibility of the results. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the performance evaluation method of the abnormal website classification model provided by the present invention.

[0018] Figure 2 It is a structural schematic diagram of the performance evaluation system of the abnormal website classification model provided by the present invention.

[0019] Figure 3 It is a flow chart of the coverage fault tolerance mechanism provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the performance evaluation device of the abnormal website classification model provided by the present invention.

[0021] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Combine the following Figure 1-Figure 5 Describe the specific implementation process of the present invention.

[0024] Figure 1 It is a flow chart of the performance evaluation method of the abnormal website classification model provided by the present invention.

[0025] Specifically, combined Figure 2 The system can control and implement at least one abnormal website classification model (such as Figure 2 The testing and evaluation of Model 1, Model 2, ..., Model N in Figure 2As shown in , considering the two parts of real-time data source and offline data source (the real-time data source and offline data source are website data, such as website address URL (Uniform Resource Locator), website HTML (Hyper Text Markup Language) content, etc.), they are input into model 1, model 2, ..., model N in sequence. The specific operation of the performance evaluation system of the abnormal website classification model can be divided into the following steps.

[0026] Step 101, according to the number of abnormal websites identified by each abnormal website classification model in the at least one abnormal website classification model in the website set to be evaluated and the total number of actual abnormal URLs, calculate the detection coverage of the URLs involved in the case for each abnormal website classification model in the at least one abnormal website classification model; wherein each of the abnormal website classification models is used to classify the website set to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain multiple sub-categories of abnormal websites.

[0027] Among them, the abnormal website classification model is used to identify and classify the URLs and their contents received in real-time monitoring or offline, and identify whether they involve abnormal business, such as publishing abnormal financial information and publishing abnormal service information. The classification results can be normal websites, abnormal websites, and further subdividing the various subcategories in abnormal websites, such as further subdividing abnormal websites into abnormal financial websites, abnormal service websites, etc. The set of websites to be evaluated refers to the website data obtained from the above-mentioned real-time data source or offline data source, including the website address URL, and the HTML (HyperText Markup Language) web page information of the website. The set of websites to be evaluated in this application can be abnormal websites identified in the real-time data source within a certain period of time (for example, within one month). The meaning of the indicator "coverage rate of websites involved in the case" is explained below. First, the website involved in the case refers to the website that is actually identified as abnormal, that is, the abnormal website that actually occurred. "Coverage of websites involved in the case" means that the abnormal website identified by the model is the abnormal website that actually occurred. This requires satisfying any one of the following two conditions A and B to consider that the model identification is correct, that is, "coverage of websites involved in the case."

[0028] Condition A: The URL of the abnormal website identified by the model is consistent with the domain name of the website involved in the case, for example: http: / / 54315429.cc / xxx and https: / / 54315429.cc / yyyyy.

[0029] Condition B: The URL of the abnormal website identified by the model (URL 1) and the URL of the website involved in the case (URL 2) are passed through the BERT (Bidirectional Encoder Representations from Transformers) model to obtain two embedded vectors, which are respectively recorded as Embedding1 and Embedding2 (the penultimate word element corresponding to the [CLS] tag, i.e., token embedding). The cosine similarity is higher than 95%. The [CLS] tag is a special tag placed at the beginning of a sentence or document to indicate the central theme.

[0030] If condition A or condition B is met, it is considered that "the URLs involved are covered", or it is called "URL tolerance".

[0031] Through this coverage fault-tolerant mechanism that combines domain name matching and embedded vector similarity, accidental errors made by regulators when registering websites involved in the case are reduced, and the accuracy of coverage calculations is improved.

[0032] "Coverage rate of websites involved in the case" mainly refers to the proportion of abnormal websites correctly identified by the model among the total websites involved in the case.

[0033] Since abnormal websites have timeliness, that is, they may no longer exist after a period of time, which affects the evaluation of model technical indicators in the abnormal website detection scenario, we comprehensively consider "accessible website coverage" and "overall coverage".

[0034] Among them, the calculation method of "accessible website coverage rate" is: based on the proportion of the number of accessible model-recognized covered websites in the total number of accessible websites involved in the case, the accessible website coverage rate is obtained; among them, the number of accessible model-recognized covered websites is the number of accessible websites among the abnormal websites correctly identified by each model; the total number of accessible websites involved in the case refers to the number of websites involved in the case that have actually occurred and are still accessible in the set of websites to be evaluated.

[0035] The calculation method of "overall coverage rate" is: based on the proportion of the number of websites covered by model identification in the total number of websites involved in the case, the overall coverage rate is obtained; among which, the number of websites covered by model identification refers to the number of abnormal websites correctly identified by each model from the set of websites to be evaluated; the total number of websites involved in the case refers to the actual websites involved in the case.

[0036] Based on the accessible website coverage and the overall coverage, the coverage of the websites involved in the case is obtained.

[0037] In summary, we can calculate the coverage rate of the websites involved in the case. as follows.

[0038] ; in, and is a configurable weight parameter, =1; "Number of accessible model identification coverage websites" e is the number of accessible websites among the abnormal websites correctly identified by each model; "Total number of accessible websites involved in the case" E refers to the number of websites involved in the case that actually occurred and are accessible so far in the set of websites to be evaluated; "Number of model identification coverage websites" f refers to the number of abnormal websites correctly identified by each model from the set of websites to be evaluated, including accessible and inaccessible ones. "Total number of websites involved in the case" F refers to the websites involved in the case that actually occurred, including those that are accessible so far and those that are inaccessible so far.

[0039] Coverage of websites involved in the case according to each model Rank each model from high to low to get the coverage rate of the websites involved Ranking order , thus we get evaluation item 1 (i.e., the coverage rate of the websites involved in the case) ) Among them, the ranking normalization scoring function It can be defined as follows (this is just an example).

[0040] ; in, Indicates the coverage rate of the website involved in the case at the ith position in descending order .

[0041] In particular, for the current investigation of fraud cases, the coverage rate of the websites involved can also be calculated. as follows: ); Among them, m is the number of accessible model-identified surveys, M is the total number of accessible surveys; n is the number of model-identified surveys, N is the total number of surveys.

[0042] Step 102, according to the number of subcategory abnormal websites identified by each abnormal website classification model and the total number of actual subcategory abnormal websites, the overall category coverage of each abnormal website classification model is calculated respectively; wherein the overall category coverage is the weighted coverage of each subcategory.

[0043] This step considers the distribution of the recognition volume of each subcategory in the total number of recognized websites, which is reflected by the indicator of full category coverage (also called overall category coverage).

[0044] Specifically, for the coverage rate of abnormal websites in each subcategory identified by the model, starting from the business objectives, since the identification samples of abnormal financial websites, abnormal service websites and other abnormal websites have different importance, this application considers evaluating the proportion of the identification volume of the three subcategories of abnormal financial websites, abnormal service websites and other abnormal websites to the total number of identified websites, which are recorded as , , . The overall category coverage is calculated from this for.

[0045] ; in, 1, which respectively represents the weight parameter of the proportion of service-class anomaly subcategories and the weight parameter of the proportion of other anomaly categories (the weight of service-class anomalies and other anomaly categories are appropriately reduced). Based on the overall category coverage Rank each model from high to low to get the ranking order , thus obtaining the overall category coverage The normalized score of .

[0046] Step 103, determining a sample set of websites for each subcategory.

[0047] It should be noted that in the actual evaluation of the effect of URL classification, the use of sampling methods to select sample sets, and the use of the results of the sample sets to reflect the performance level of the entire model will cause the performance level of the model to be overly dependent on the selected samples. If the sample set is small, it is easy to cause the sample evaluation results to not represent the performance level of the model well, resulting in a large sampling error; if the sample set is large, the sampling error will be reduced, but more manual inspections will be required, resulting in a large amount of manual overhead, and repeated evaluations of highly similar samples are inefficient. Furthermore, we will sample different types of samples, but the existing sampling number allocation method will cause unreasonable allocation of manual resources, large total sampling errors, and other problems. Therefore, this application proposes a sampling method based on multi-URL collaboration.

[0048] The principles considered in this sampling method include: the more complex the structure of the abnormal website (the complexity is reflected by the TAG label in the website code), the more samples of this type should be; the later the abnormal website is identified (that is, the earlier the website first appears in DPI (Deep Packet Inspect)), the more samples of this type should be.

[0049] The target types of sampling are the original small categories (i.e., subcategories), including: abnormal websites for financial management, abnormal websites for academic services, abnormal websites for examination services, abnormal websites for identity forgery, abnormal websites for emotional services, and other n categories.

[0050] First, the initial samples of each subcategory are randomly given. For a small class, suppose the model recognizes abnormal websites, whose initial sampling function The definitions are as follows.

[0051] ; in, , is a settable constant, M is the number of abnormal websites identified by the model from this subcategory; is a percentage and can be set flexibly as needed; N is a constant and can be set flexibly as needed. This application uses different initial sampling functions for different categories. For example, the financial anomaly subcategory belonging to the financial anomaly adopts , the sub-class belonging to the service class exception adopts , the subclasses belonging to other abnormal classes adopt .

[0052] Step 104, calculate the probability that the samples in the website sampling sample set of each subcategory are correctly classified to obtain the subcategory prediction accuracy; and determine the prediction accuracy of all subcategory by weighting the prediction accuracy of each subcategory.

[0053] Next, the evaluation index is calculated for each type of sample sampled in step 103. Different from traditional indicators such as accuracy and recall, this application defines the prediction accuracy of all subcategories as As the evaluation index of the model classification effect in this scenario. Represents the total number of abnormal website samples that actually occurred in all subcategories in the total sample. Represents the final abnormal website of all subcategories combined, Represents the number of abnormal website samples that actually occurred in the i-th subcategory, Represents the number of samples predicted as abnormal websites in the ith subcategory; for the abnormal website classification model to be detected, the probability of correct classification in the ith subcategory for.

[0054] ; For all subcategories, the probability that the model classification result is consistent with the true label for: ; Based on the number of websites classified as abnormal websites by the abnormal website classification model in the website sampling sample set of each category and the number of abnormal websites that actually occurred in the website sampling sample set, the conditional probability that each subcategory is accurately classified is obtained; among them, represents the conditional probability that the i-th subcategory is correctly classified.

[0055] For the i-th category,

[0056] ; Then define the subcategory prediction accuracy of each subcategory for.

[0057] ; That is to say, according to the conditional probability of each subcategory being correctly classified, the subcategory prediction accuracy of each subcategory is obtained; among which, represents the conditional probability that the jth subcategory is correctly classified, represents the number of abnormal websites that actually occurred in the jth subcategory, represents the number of abnormal websites detected by the model in the jth subcategory, represents the number of samples correctly classified into the jth subcategory, w is the total number of samples, is the weight corresponding to the i-th subcategory, The closer it is to 1, the better the overall model classification effect is.

[0058] Based on the business objectives, we consider evaluating the accuracy of model recognition of only three categories of financial anomalies, service anomalies, and other anomalies (such as accurate, suspected, and inaccurate). Each tester customizes the mapping table from the original sub-category to the evaluation category. The accuracy rate (WMP) of the category (including at least one corresponding sub-category, such as service anomalies that also include academic anomalies and examination anomalies) is defined as follows.

[0059] ; in, Configure based on business relevance and importance.

[0060] Calculate the WMP value of financial management anomalies , abnormal WMP value of service class , other abnormal WMP values The overall category evaluation value is calculated from this for.

[0061] ; in, .in accordance with Rank the models from high to low and get , thus obtaining the normalized score of evaluation item 3 .

[0062] Step 105 , determining a time effectiveness evaluation index of each abnormal website classification model according to a statistical index of the recognition time of each abnormal website classification model for each website to be evaluated in the set of websites to be evaluated.

[0063] Specifically, the same amount of URL data is sent to each model 1-N through a real-time data source every day, and the model deployment environment is all intranet. The model timeliness performance is compared by comparing the model recognition time of the URL (= completion recognition time - start recognition time). Since the different recognition end time of each model may lead to different start times for subsequent evaluations, this application constructs a batch aligner, and the specific implementation steps are as follows.

[0064] For at least one batch of website data files sent to the abnormal website classification model, calculate the recognition time from data input to classification result output, and calculate the average recognition time , quartile , maximum value .

[0065] The timeliness evaluation index of the abnormal website classification model is calculated from this for.

[0066] ; in, , and is the normalization parameter, and .in accordance with Rank the models from low to high and get , thus obtaining the normalized scores of the timeliness evaluation indicators of each model .

[0067] ; in, is the normalized scoring function.

[0068] Step 106, determining the comprehensive model evaluation score of each abnormal website classification model according to any one or more of the detection coverage of the involved URLs, the overall category coverage, the prediction accuracy of all subcategories, and the timeliness evaluation index.

[0069] Final model comprehensive evaluation score (0-100).

[0070] ; In the above-mentioned embodiment, the detection coverage rate of the involved websites of each abnormal website classification model in at least one abnormal website classification model is calculated by according to the number of abnormal websites identified in the website set to be evaluated and the total number of real abnormal websites in each abnormal website classification model in at least one abnormal website classification model; wherein each abnormal website classification model is used to classify the website set to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain multiple sub-category abnormal websites; according to the number of sub-category abnormal websites identified by each abnormal website classification model and the total number of real sub-category abnormal websites, the overall category coverage rate of each abnormal website classification model is calculated respectively; wherein the overall category coverage rate is Weighting the coverage of each subcategory; determining the sample set of website samples for each subcategory; calculating the probability that the samples in the sample set of website samples for each subcategory are accurately classified to obtain the prediction accuracy of the subcategory; determining the prediction accuracy of all subcategories according to the weighting of the prediction accuracy of each subcategory; determining the timeliness evaluation index of each abnormal website classification model according to the statistical indicators of the recognition time of each website to be evaluated in the set of websites to be evaluated by each abnormal website classification model; determining the model comprehensive evaluation score of each abnormal website classification model according to any one or more of the detection coverage of the website involved in the case, the overall category coverage, the prediction accuracy of all subcategories and the timeliness evaluation index. This method designs different indicators to evaluate the evaluation effect of the abnormal website classification model, and can use multiple indicators for collaborative evaluation, thereby improving the comprehensiveness of the evaluation effect. It not only considers the classification of normal and abnormal websites by the model, but also considers the classification effect of different subcategories in abnormal websites, thereby improving the accuracy of model evaluation. Furthermore, this application also uses a multi-URL collaborative sampling mechanism, which can effectively discover and solve the problem of differences in classification effect evaluation caused by reasons such as imbalanced sample categories, and significantly improve the credibility of the results.

[0071] In one embodiment, the above step 103 includes: The number of websites in each subcategory in the sample set is determined based on the complexity of the website structure and / or the time it took to develop the website.

[0072] The website structure complexity is determined according to the number of tags in the hypertext markup language corresponding to each website; the website development time refers to the time from the moment when the website first appears in the network data packet to the moment when the website is classified.

[0073] Specifically, the following steps are included.

[0074] The website structure complexity of each website to be sampled in the set of websites to be sampled is calculated; wherein the website structure complexity is determined according to the number of tags in the hypertext markup language corresponding to each website.

[0075] Determine multiple initial sampling sample sets for each subcategory.

[0076] Multiple rounds of website sample quantity update operations are performed on the multiple initial sampling sample sets.

[0077] In each round of website elimination operation, after a first preset number of website address samples are updated from the multiple initial sampling sample sets respectively, the variance of the number of labels in each updated sampling sample set is calculated.

[0078] According to the variance of the number of labels in each updated sampling sample set, the error change rate of the variance before the update and the variance after the update is calculated; the first preset number is updated; returning to each round of website elimination operation, after updating the preset number of website samples from the multiple initial sampling sample sets respectively, the variance step of the number of labels in each updated sampling sample set is calculated.

[0079] An initial sampling sample set corresponding to the minimum error change rate and a first preset number corresponding to the minimum error change rate are selected.

[0080] The website sampling sample set corresponding to the current subcategory is determined according to the initial sampling sample set corresponding to the minimum error change rate and the first preset number corresponding to the minimum error change rate.

[0081] It should be noted that in order to distribute samples more reasonably, it is necessary to update the number of samples and sample sets of each type of website according to the characteristics of the anomaly type itself.

[0082] (1) For website types with simple structures (corresponding to small variance of TAG numbers), the number of samples can be reduced and some samples can be removed from the sample set.

[0083] Input: n types of URLs to be evaluated, number of samples of each type of URL , corresponding to the initial sampling sample set .

[0084] Output: URL type i that can reduce the number of samples, URL samples that need to be eliminated .

[0085] For i=1:n

[0086] Calculate the sample set of this type of URL The original variance of the number of tags

[0087] For j=1: From the URL sample collection Eliminate The new sample set obtained

[0088] Calculate the variance of the number of URL TAGs in the new sample set of this type

[0089] Calculate the error rate of change

[0090] End

[0091] From all (j=1,2,..., ) and the corresponding URL sample .

[0092] End

[0093] return The URL type serial number i corresponding to the minimum value, and the URL sample .

[0094] (2) For website types with complex structures (corresponding to large variance in the number of tags), it is necessary to increase the number of samples and select some samples from the remaining URLs to add to the sample set.

[0095] The calculation process is as follows.

[0096] Input: n types of URLs to be evaluated, number of samples of each type of URL , corresponding to the initial sampling sample set , and the sample set not included in the initial sampling .

[0097] Output: URL type i that needs to increase the number of samples, URL samples that need to be supplemented .

[0098] For i=1:n

[0099] Calculate the sample set of this type of URL The original variance of the number of tags

[0100] The set of samples that were never included in the initial sampling Randomly select T URL samples from (T is a constant and can be set).

[0101] For k=1:T

[0102] From the selected Sample collection of URLs Add new samples The new sample set obtained

[0103] Calculate the variance of the number of URL TAGs in the new sample set of this type

[0104] Calculate the error rate of change

[0105] End

[0106] From all (j=1,2,..., ) and the corresponding URL sample .

[0107] End

[0108] return The URL type number i corresponding to the maximum value, and the URL sample .

[0109] (3) For the types of websites that were developed early (corresponding to the early first appearance time in DPI), the number of website types sampled can be reduced and some samples can be removed from the sample set.

[0110] (4) For the types of websites with late development time (corresponding to the late first appearance time in DPI), it is necessary to increase the number of website types sampled and select some samples from the remaining websites to add to the sample set.

[0111] In the above embodiment, the sample set is updated in consideration of the characteristics of the abnormal type itself, which can improve the accuracy of the sampling result.

[0112] The performance evaluation device of the abnormal website classification model provided by the present invention is described below. The performance evaluation device of the abnormal website classification model described below and the performance evaluation method of the abnormal website classification model described above can be referred to each other.

[0113] Figure 4 : is a schematic diagram of a module of a performance evaluation device for an abnormal website classification model provided by the present invention, and the performance evaluation device for an abnormal website classification model includes the following modules: The module 401 for calculating the detection coverage rate of the websites involved in the case is used to calculate the detection coverage rate of the websites involved in the case of each abnormal website classification model in at least one abnormal website classification model according to the number of abnormal websites identified in the website set to be evaluated and the total number of actual abnormal websites; wherein each abnormal website classification model is used to classify the website set to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain multiple sub-categories of abnormal websites; the website coverage rate involved in the case is used to indicate the degree of consistency between the number of abnormal websites identified by the abnormal website classification model and the total number of abnormal websites that actually occurred; The overall category coverage calculation module 402 is used to calculate the overall category coverage of each abnormal website classification model according to the number of subcategory abnormal websites identified by each abnormal website classification model and the total number of actual subcategory abnormal websites; wherein the overall category coverage is the weighted coverage of each subcategory; A website sampling module 403 is used to determine a sample set of website samples for each subcategory; The prediction accuracy calculation module 404 for all subcategories is used to calculate the probability that the samples in the website sampling sample set of each subcategory are classified accurately, and obtain the prediction accuracy of the subcategory; and determine the prediction accuracy of all subcategories according to the weighting of the prediction accuracy of each subcategory; The timeliness evaluation index calculation module 405 is used to determine the timeliness evaluation index of each abnormal website classification model according to the statistical index of the recognition time of each website to be evaluated in the set of websites to be evaluated by each abnormal website classification model; The model comprehensive evaluation module 406 is used to determine the model comprehensive evaluation score of each abnormal website classification model based on any one or more of the detection coverage of the URLs involved in the case, the overall category coverage, the prediction accuracy of all subcategories and the timeliness evaluation indicators.

[0114] In one embodiment, the above-mentioned module 401 for calculating the detection coverage rate of websites involved in the case is further used to: obtain the accessible website coverage rate based on the proportion of the number of websites covered by the accessible model identification in the total number of accessible websites involved in the case; wherein, the number of websites covered by the accessible model identification is the number of accessible websites among the abnormal websites correctly identified by each model; the total number of accessible websites involved in the case refers to the number of websites involved in the case that actually occurred and are accessible so far in the set of websites to be evaluated; obtain the overall coverage rate based on the proportion of the number of websites covered by the model identification in the total number of websites involved in the case; wherein, the number of websites covered by the model identification refers to the number of abnormal websites correctly identified by each model from the set of websites to be evaluated; the total number of websites involved in the case refers to the websites involved in the case that actually occurred; according to the accessible website coverage rate and the overall coverage rate, obtain the website coverage rate involved in the case

[0115] In one embodiment, the website sampling module 403 is further used to: determine the number of websites in each subcategory according to the complexity of the website structure; and / or determine the number of websites in each subcategory according to the website development time; wherein the website structure complexity is determined according to the number of tags in the hypertext markup language corresponding to each website; the website development time refers to the time from the time when the website first appears in the network data packet to the time when the website is classified.

[0116] In one embodiment, the website sampling module 403 is further used to: calculate the website structure complexity of each website to be sampled in the website set to be sampled; wherein the website structure complexity is determined according to the number of tags in the hypertext markup language corresponding to each website; determine multiple initial sampling sample sets for each subcategory; perform multiple rounds of website sample quantity update operations on the multiple initial sampling sample sets; in each round of website elimination operation, after respectively updating a first preset number of website URL samples from the multiple initial sampling sample sets, calculate the variance of the number of tags in each updated sampling sample set; and calculate the variance of the number of tags in each updated sampling sample set according to the number of tags in each updated sampling sample set. The variance of the number of labels in the sample set is calculated, and the error change rate of the variance before the update and the variance after the update is calculated; the first preset number is updated; in each round of website elimination operation, after updating the preset number of website samples from the multiple initial sampling sample sets respectively, the variance step of calculating the number of labels in each updated sampling sample set is performed; the initial sampling sample set corresponding to the minimum error change rate and the first preset number corresponding to the minimum error change rate are selected; according to the initial sampling sample set corresponding to the minimum error change rate and the first preset number corresponding to the minimum error change rate, the website sampling sample set corresponding to the current subcategory is determined.

[0117] In one embodiment, the above-mentioned model comprehensive evaluation module 406 is further used to: The comprehensive model evaluation score of each abnormal website classification model is determined based on the weighting of any multiple indicators among the detection coverage of the URLs involved in the case, the overall category coverage, the prediction accuracy of all subcategories and the timeliness evaluation indicators.

[0118] In one embodiment, the above-mentioned all sub-category prediction accuracy calculation module 404 is further used to: Based on the number of websites classified as abnormal websites by the abnormal website classification model in the website sampling sample set of each category and the number of abnormal websites that actually occurred in the website sampling sample set, the conditional probability that each subcategory is accurately classified is obtained; based on the conditional probability that each subcategory is correctly classified, the subcategory prediction accuracy of each subcategory is obtained.

[0119] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the performance evaluation method of the abnormal website classification model, the method comprising: calculating the detection coverage of the involved websites of each abnormal website classification model in at least one abnormal website classification model according to the number of abnormal websites identified in the website set to be evaluated and the total number of real abnormal websites; wherein each abnormal website classification model is used to classify the website set to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain multiple sub-category abnormal websites; according to the number of sub-category abnormal websites identified by each abnormal website classification model and the total number of real sub-category abnormal websites, respectively calculate the detection coverage of each abnormal website classification model. The overall category coverage of the type; wherein the overall category coverage is the weighted coverage of each subcategory; determine the website sampling sample set of each subcategory; calculate the probability that the samples in the website sampling sample set of each subcategory are accurately classified to obtain the subcategory prediction accuracy; determine the prediction accuracy of all subcategories according to the weighted prediction accuracy of each subcategory; determine the timeliness evaluation index of each abnormal website classification model according to the statistical indicators of the recognition time of each to-be-evaluated website in the to-be-evaluated website set for each abnormal website classification model; determine the model comprehensive evaluation score of each abnormal website classification model according to any one or more of the detection coverage of the URLs involved in the case, the overall category coverage, the prediction accuracy of all subcategories and the timeliness evaluation index.

[0120] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes at least one instruction to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0121] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the performance evaluation method of the abnormal website classification model provided by the above-mentioned methods, the method comprising: according to each abnormal website classification model in at least one abnormal website classification model, for the number of abnormal websites identified in the website set to be evaluated and the total number of actual abnormal URLs, calculating the detection coverage of the URLs involved in the case for each abnormal website classification model in at least one abnormal website classification model; wherein each abnormal website classification model is used to classify the website set to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain multiple subcategories of abnormal websites; according to the subcategories identified by each abnormal website classification model The number of abnormal websites and the total number of real abnormal URLs in subcategories are used to calculate the overall category coverage of each abnormal website classification model; among which, the overall category coverage is the weighted coverage of each subcategory; determine the website sampling sample set of each subcategory; calculate the probability that the samples in the website sampling sample set of each subcategory are accurately classified to obtain the subcategory prediction accuracy; according to the weighted prediction accuracy of each subcategory, determine the prediction accuracy of all subcategories; according to the statistical indicators of the recognition time of each abnormal website classification model for each website to be evaluated in the set of websites to be evaluated, determine the timeliness evaluation index of each abnormal website classification model; according to any one or more of the detection coverage of the URLs involved, the overall category coverage, the prediction accuracy of all subcategories and the timeliness evaluation index, determine the model comprehensive evaluation score of each abnormal website classification model.

[0122] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the performance evaluation method of the abnormal website classification model provided by the above-mentioned methods, the method comprising: calculating the detection coverage of the involved websites of each abnormal website classification model in at least one abnormal website classification model according to the number of abnormal websites identified in the website set to be evaluated and the total number of actual abnormal websites in each abnormal website classification model in at least one abnormal website classification model; wherein each abnormal website classification model is used to classify the website set to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain multiple sub-categories of abnormal websites; and according to the number of sub-categories of abnormal websites identified by each abnormal website classification model and the total number of actual sub-categories of abnormal websites, the detection coverage of the involved websites of each abnormal website classification model is calculated. The total number of common URLs is used to calculate the overall category coverage of each abnormal website classification model; wherein the overall category coverage is the weighted coverage of each subcategory; a sampling sample set of websites for each subcategory is determined; the probability that the samples in the sampling sample set of websites for each subcategory are accurately classified is calculated to obtain the subcategory prediction accuracy; the prediction accuracy of all subcategories is determined according to the weighted prediction accuracy of each subcategory; the timeliness evaluation index of each abnormal website classification model is determined according to the statistical indicators of the recognition time of each website to be evaluated in the set of websites to be evaluated by each abnormal website classification model; the comprehensive model evaluation score of each abnormal website classification model is determined according to any one or more of the detection coverage of the URLs involved in the case, the overall category coverage, the prediction accuracy of all subcategories and the timeliness evaluation index.

[0123] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0124] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including at least one instruction to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A performance evaluation method for an abnormal website classification model, characterized in that: include: According to the number of abnormal websites identified by each abnormal website classification model in the at least one abnormal website classification model for the website set to be evaluated and the total number of abnormal websites that actually occurred, the website coverage rate involved in the case of each abnormal website classification model in the at least one abnormal website classification model is calculated; wherein each of the abnormal website classification models is used to classify the website set to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain a plurality of sub-categories of abnormal websites; the website coverage rate involved in the case is used to indicate the degree of consistency between the number of abnormal websites identified by the abnormal website classification model and the total number of abnormal websites that actually occurred; According to the number of abnormal sub-category websites identified by each abnormal website classification model and the total number of actual abnormal sub-category websites, the overall category coverage of each abnormal website classification model is calculated respectively; wherein the overall category coverage is the weighted coverage of each sub-category; Determine a sample set of websites for each subcategory; Calculate the probability that the samples in the website sampling sample set of each subcategory are classified accurately to obtain the subcategory prediction accuracy; determine the prediction accuracy of all subcategories according to the weighted prediction accuracy of each subcategory; Determine the timeliness evaluation index of each abnormal website classification model according to the statistical index of the recognition time of each website to be evaluated in the set of websites to be evaluated by each abnormal website classification model; The comprehensive model evaluation score of each abnormal website classification model is determined based on any one or more of the detection coverage of the URLs involved in the case, the overall category coverage, the prediction accuracy of all subcategories, and the timeliness evaluation indicators.

2. The performance evaluation method of the abnormal website classification model according to claim 1 is characterized in that: The number of identified abnormal websites includes the number of accessible model identification coverage websites and the number of model identification coverage websites; the total number of abnormal websites that actually occurred includes the total number of accessible involved websites and the total number of involved websites; the calculation of the involved website coverage rate of each abnormal website classification model in at least one abnormal website classification model according to the number of abnormal websites identified in the to-be-evaluated website set and the total number of abnormal websites that actually occurred, includes: Based on the proportion of the number of accessible model-recognized covered websites in the total number of accessible websites involved in the case, the accessible website coverage rate is obtained; wherein the number of accessible model-recognized covered websites is the number of accessible websites among the abnormal websites correctly identified by each model; the total number of accessible websites involved in the case refers to the number of websites involved in the case that actually occurred and are accessible so far in the set of websites to be evaluated; The overall coverage rate is obtained based on the proportion of the number of websites covered by the model identification in the total number of websites involved in the case; wherein the number of websites covered by the model identification refers to the number of abnormal websites correctly identified by each model from the set of websites to be evaluated; and the total number of websites involved in the case refers to the websites involved in the case that actually occurred; The coverage rate of the websites involved in the case is obtained according to the accessible website coverage rate and the overall coverage rate.

3. The performance evaluation method of the abnormal website classification model according to claim 1 is characterized in that: The website sampling sample set for each subcategory is determined, including: According to the complexity of the website structure, determine the number of websites in each subcategory in the sample set; and / or, Determine the number of websites in each subcategory to be sampled based on the time of website development; The website structure complexity is determined according to the number of tags in the hypertext markup language corresponding to each website; the website development time refers to the time from the moment when the website first appears in the network data packet to the moment when the website is classified.

4. The performance evaluation method of the abnormal website classification model according to claim 3 is characterized in that: Based on the complexity of the website structure, determine the number of websites in each subcategory in the sample set, including: Calculating the website structure complexity of each website to be sampled in the set of websites to be sampled; wherein the website structure complexity is determined according to the number of tags in the hypertext markup language corresponding to each website; Determine multiple initial sampling sample sets for each subcategory; Performing multiple rounds of website sample quantity update operations on the multiple initial sampling sample sets; In each round of website elimination operation, after respectively updating a first preset number of website samples from the plurality of initial sampling sample sets, the variance of the number of labels in each updated sampling sample set is calculated; According to the variance of the number of labels in each updated sampling sample set, calculate the error change rate of the variance before the update and the variance after the update; update the first preset number; return to the step of calculating the variance of the number of labels in each updated sampling sample set after updating the preset number of URL samples from the multiple initial sampling sample sets in each round of website elimination operation; Selecting an initial sampling sample set corresponding to the minimum error change rate and a first preset number corresponding to the minimum error change rate; The website sampling sample set corresponding to the current subcategory is determined according to the initial sampling sample set corresponding to the minimum error change rate and the first preset number corresponding to the minimum error change rate.

5. The performance evaluation method of the abnormal website classification model according to claim 1 is characterized in that: Determining the model comprehensive evaluation score of each abnormal website classification model according to any one or more of the detection coverage of the involved URLs, the overall category coverage, the prediction accuracy of all subcategories, and the timeliness evaluation index includes: The comprehensive model evaluation score of each abnormal website classification model is determined based on the weighting of any multiple indicators among the detection coverage of the URLs involved in the case, the overall category coverage, the prediction accuracy of all subcategories and the timeliness evaluation indicators.

6. The performance evaluation method of the abnormal website classification model according to claim 1 is characterized in that: The calculation of the probability that the samples in the website sampling sample set of each subcategory are accurately classified to obtain the subcategory prediction accuracy includes: Based on the number of websites classified as abnormal websites by the abnormal website classification model in the website sampling sample set of each category and the number of abnormal websites actually occurring in the website sampling sample set, the conditional probability that each subcategory is accurately classified is obtained; According to the conditional probability that each subcategory is correctly classified, the subcategory prediction accuracy of each subcategory is obtained.

7. A performance evaluation device for an abnormal website classification model, characterized in that: include: A module for calculating the detection coverage rate of websites involved in the case is used to calculate the detection coverage rate of websites involved in the case of each abnormal website classification model in at least one abnormal website classification model according to the number of abnormal websites identified in the website set to be evaluated and the total number of real abnormal websites; wherein each abnormal website classification model is used to classify the website set to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain multiple sub-categories of abnormal websites; The overall category coverage calculation module is used to calculate the overall category coverage of each abnormal website classification model according to the number of subcategory abnormal websites identified by each abnormal website classification model and the total number of actual subcategory abnormal websites; wherein the overall category coverage is the weighted coverage of each subcategory; A website sampling module, used to determine a sample set of websites for each subcategory; The prediction accuracy calculation module for all subcategories is used to calculate the probability that the samples in the website sampling sample set of each subcategory are accurately classified to obtain the prediction accuracy of the subcategory; the prediction accuracy of all subcategories is determined according to the weighting of the prediction accuracy of each subcategory; A timeliness evaluation index calculation module is used to determine the timeliness evaluation index of each abnormal website classification model according to the statistical index of the recognition time of each website to be evaluated in the set of websites to be evaluated by each abnormal website classification model; The model comprehensive evaluation module is used to determine the model comprehensive evaluation score of each abnormal website classification model based on any one or more of the detection coverage of the URLs involved in the case, the overall category coverage, the prediction accuracy of all subcategories and the timeliness evaluation indicators.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the performance evaluation method of the abnormal website classification model according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the performance evaluation method of the abnormal website classification model as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the performance evaluation method of the abnormal website classification model as described in any one of claims 1 to 6 is implemented.

Images