Performance evaluation methods, apparatus, equipment, and media for abnormal website classification models

Abstract

This invention provides a performance evaluation method, apparatus, device, and medium for anomaly website classification models. The method includes: calculating the detection coverage rate of involved websites for each anomaly website classification model; calculating the overall category coverage rate of each anomaly website classification model based on the number of identified sub-category anomaly websites and the total number of actual sub-category anomaly websites; determining the website sampling sample set for each sub-category; calculating the sub-category prediction accuracy rate for each sub-category, thereby determining the overall sub-category prediction accuracy rate; determining a timeliness evaluation index based on the identification time of each website to be evaluated; and determining the model's comprehensive evaluation score based on any one or more of the involved website detection coverage rate, overall category coverage rate, overall sub-category prediction accuracy rate, and timeliness evaluation index. This method designs different indicators to evaluate the performance of the anomaly website classification models and uses multiple indicators for collaborative evaluation, improving the comprehensiveness of the evaluation results.

Description

Technical Field

[0001] This invention relates to the field of information security technology, and in particular to a performance evaluation method, apparatus, device, and medium for an abnormal website classification model. Background Technology

[0002] Telecommunications network information security is crucial to the harmony and stability of social life. However, the perpetrators of abnormal telecommunications network behaviors are constantly changing their technical methods, and the types of abnormal behaviors are becoming increasingly complex and varied, posing greater challenges to the detection of abnormal behaviors.

[0003] Currently, machine learning-based artificial intelligence models are used to automatically predict whether a webpage is an abnormal website. However, the evaluation of the classification effect of artificial intelligence models is mainly based on common indicators such as single accuracy or F1 score. Due to the special nature of abnormal websites in the telecommunications industry, such as the fact that abnormal websites in this field are generally concealed, the current model evaluation indicators cannot accurately reflect the true and comprehensive evaluation effect in this field. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for evaluating the performance of anomaly website classification models, which addresses the shortcomings of existing technologies that rely on a single evaluation index for anomaly website classification effectiveness, thereby improving the comprehensiveness and accuracy of the evaluation of anomaly website classification effectiveness.

[0005] This invention provides a performance evaluation method for an abnormal website classification model, comprising the following steps:

[0006] Based on the number of abnormal websites identified by each of the at least one abnormal website classification models in the set of websites to be evaluated and the total number of actual abnormal URLs, the detection coverage rate of the involved URLs for each of the at least one abnormal website classification models is calculated; wherein, each of the abnormal website classification models is used to classify the set of websites to be evaluated into abnormal websites, and to classify the abnormal websites into multiple subcategories of abnormal websites; the involved website coverage rate is used to represent the degree of consistency between the number of abnormal websites identified by the abnormal website classification model and the total number of actual abnormal websites.

[0007] Based on the number of abnormal websites in each subcategory identified by each abnormal website classification model and the total number of actual abnormal URLs in each subcategory, the overall category coverage rate of each abnormal website classification model is calculated; wherein, the overall category coverage rate is a weighted average of the coverage rates of each subcategory.

[0008] Determine the website sampling set for each subcategory;

[0009] Calculate the probability that the samples in the website sampling set for each subcategory are correctly classified to obtain the subcategory prediction accuracy; determine the overall subcategory prediction accuracy by weighting the prediction accuracy of each subcategory.

[0010] Based on the statistical indicators of the identification time of each abnormal website classification model for each website in the set of websites to be evaluated, the timeliness evaluation indicators of each abnormal website classification model are determined.

[0011] The comprehensive evaluation score of each abnormal website classification model is determined based on any one or more of the following: the detection coverage rate of the website in question, the overall category coverage rate, the prediction accuracy rate of all subcategories, and the timeliness evaluation indicators.

[0012] According to a performance evaluation method for an abnormal website classification model provided by the present invention, the number of identified abnormal websites includes the number of accessible websites covered by the model and the total number of websites actually involved in the case; the total number of actually involved abnormal websites includes the total number of accessible websites involved in the case and the total number of websites actually involved in the case; the step of calculating the coverage rate of websites involved in the case for each of the at least one abnormal website classification models based on the number of abnormal websites identified and the total number of actually involved abnormal websites in the set of websites to be evaluated, according to the number of abnormal websites identified by each of the at least one abnormal website classification models, includes:

[0013] The accessible website coverage rate is obtained based on the proportion of the number of accessible model-identified covered websites in the total number of accessible involved websites; wherein, the number of accessible model-identified covered websites is the number of accessible websites among the abnormal websites correctly identified by each model; the total number of accessible involved websites refers to the number of involved websites in the set of websites to be evaluated that have actually occurred and are still accessible to date.

[0014] The overall coverage rate is obtained based on the proportion of the number of websites covered by the model in the total number of websites involved in the case; wherein, the number of websites covered by the model refers to the number of abnormal websites correctly identified by each model from the set of websites to be evaluated; and the total number of websites involved in the case refers to the websites that actually occurred in the case.

[0015] The coverage rate of the website in question is obtained based on the accessible website coverage rate and the overall coverage rate.

[0016] According to the performance evaluation method of the abnormal website classification model provided by the present invention, the step of determining the website sampling sample set for each sub-category includes:

[0017] Calculate the complexity of the website structure for each website in the set of websites to be sampled; wherein the complexity of the website structure is determined based on the number of tags in the Hypertext Markup Language corresponding to each website;

[0018] Determine multiple initial sample sets for each subcategory;

[0019] Multiple rounds of website removal operations are performed on the multiple initial sample sets;

[0020] In each round of website removal operation, after removing a preset number of website samples from the multiple initial sampling sample sets, the variance of the number of tags in each removed or added sampling sample set is calculated.

[0021] Based on the variance of the number of labels in each sampled set after removal, calculate the error rate of change between the variance before removal and the variance after removal; update the preset removal quantity; return to the step of calculating the variance of the number of labels in each sampled set after removal after removing the preset removal quantity of website samples from the multiple initial sampled sets in each round of website removal operation.

[0022] Select the initial sample set corresponding to the smallest error change rate and the preset number of samples to be removed corresponding to the smallest error change rate.

[0023] Based on the initial sample set corresponding to the minimum error change rate and the preset elimination quantity corresponding to the minimum error change rate, the website sample set corresponding to the current sub-category is determined.

[0024] According to the performance evaluation method of the abnormal website classification model provided by the present invention, the step of determining the website sampling sample set for each sub-category includes:

[0025] Calculate the complexity of the website structure for each website in the set of websites to be sampled; wherein the complexity of the website structure is determined based on the number of tags in the Hypertext Markup Language corresponding to each website;

[0026] Determine multiple initial sample sets for each subcategory;

[0027] Multiple rounds of website removal operations are performed on the multiple initial sample sets;

[0028] In each round of website elimination, after adding a preset number of URL samples from the multiple initial sampling sample sets, the variance of the number of tags in each additional sampling sample set is calculated.

[0029] Based on the variance of the number of labels in each removed sample set, calculate the error rate of change between the variance before addition and the variance after removal; update the preset addition quantity; return to the steps in each round of website removal operation, after adding the preset addition quantity of website samples from the multiple initial sample sets, and then calculating the variance of the number of labels in each added sample set.

[0030] Select the initial sample set corresponding to the smallest error change rate and the preset increase quantity corresponding to the smallest error change rate;

[0031] Based on the initial sample set corresponding to the minimum error change rate and the preset increase quantity corresponding to the minimum error change rate, the website sample set corresponding to the current sub-category is determined.

[0032] According to the performance evaluation method of the abnormal website classification model provided by the present invention, the step of determining the website sampling sample set for each sub-category includes:

[0033] Based on the time a website appears in network data packets, determine the sample set of websites for each subcategory.

[0034] According to the performance evaluation method of the abnormal website classification model provided by the present invention, the step of calculating the probability that a sample in the website sampling set of each sub-category is accurately classified to obtain the sub-category prediction accuracy includes:

[0035] Based on the number of websites in the website sampling set of each category that are classified as abnormal websites by the abnormal website classification model and the number of actual abnormal websites in the website sampling set, the conditional probability of each subcategory being accurately classified is obtained.

[0036] Based on the conditional probability of each subcategory being correctly classified, the subcategory prediction accuracy of each subcategory is obtained.

[0037] The present invention also provides a performance evaluation device for an abnormal website classification model, comprising the following modules:

[0038] The involved website URL detection coverage calculation module is used to calculate the involved website URL detection coverage of each of the at least one abnormal website classification models based on the number of abnormal websites identified by each abnormal website classification model in the website set to be evaluated and the total number of actual abnormal URLs; wherein, each of the abnormal website classification models is used to classify the website set to be evaluated into abnormal websites, and to classify the abnormal websites into multiple subcategories of abnormal websites; the involved website URL detection coverage is used to represent the degree of consistency between the number of abnormal websites identified by the abnormal website classification model and the total number of actual abnormal websites.

[0039] The overall category coverage calculation module is used to calculate the overall category coverage of each abnormal website classification model based on the number of sub-category abnormal websites identified by each abnormal website classification model and the total number of real sub-category abnormal URLs; wherein, the overall category coverage is a weighted sum of the coverage of each sub-category.

[0040] The website sampling module is used to determine the website sampling sample set for each subcategory;

[0041] The overall subcategory prediction accuracy calculation module is used to calculate the probability that the samples in the website sampling sample set of each subcategory are correctly classified, and obtain the subcategory prediction accuracy; the overall subcategory prediction accuracy is determined by weighting the prediction accuracy of each subcategory.

[0042] The timeliness assessment index calculation module is used to determine the timeliness assessment index of each abnormal website classification model based on the statistical index of the identification time of each website in the set of websites to be evaluated for each abnormal website classification model.

[0043] The model comprehensive evaluation module is used to determine the model comprehensive evaluation score of each abnormal website classification model based on any one or more of the following: the detection coverage rate of the website involved in the case, the overall category coverage rate, the prediction accuracy rate of all subcategories, and the timeliness evaluation indicators.

[0044] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a performance evaluation method for any of the above-described abnormal website classification models.

[0045] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a performance evaluation method for an abnormal website classification model as described above.

[0046] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a performance evaluation method for any of the above-described abnormal website classification models.

[0047] The present invention provides a performance evaluation method, apparatus, device, and medium for abnormal website classification models. This involves calculating the detection coverage rate of the involved URLs for each of at least one abnormal website classification models based on the number of abnormal websites identified by each model in a set of websites to be evaluated and the total number of actual abnormal URLs. Each of the abnormal website classification models is used to classify the set of websites to be evaluated into abnormal websites, and to classify the abnormal websites into multiple subcategories of abnormal websites. The overall category coverage rate of each abnormal website classification model is calculated based on the number of subcategories of abnormal websites identified by each model and the total number of actual subcategories of abnormal URLs. The method involves several key steps: First, the overall category coverage rate is a weighted average of the coverage rates of each subcategory. Second, a website sampling set is determined for each subcategory. Third, the probability that a sample in each subcategory's website sampling set is accurately classified is calculated, yielding the subcategory prediction accuracy. Fourth, the overall subcategory prediction accuracy is determined based on the weighted average of the subcategory prediction accuracy rates. Fifth, a timeliness evaluation indicator for each abnormal website classification model is determined based on the statistical indicator of the identification time for each website in the evaluation set. Sixth, the comprehensive evaluation score for each abnormal website classification model is determined based on any one or more of the following: the detection coverage rate of the involved website, the overall category coverage rate, the overall subcategory prediction accuracy, and the timeliness evaluation indicator. This method designs different indicators to evaluate the effectiveness of the abnormal website classification models and allows for the use of multiple indicators in a coordinated evaluation, improving the comprehensiveness of the evaluation. It considers not only the model's classification of normal and abnormal websites but also the classification effectiveness of different subcategories within abnormal websites, thus improving the accuracy of the model evaluation. Furthermore, this application also uses a multi-website collaborative sampling mechanism, which can effectively identify and resolve the discrepancies in classification performance evaluation caused by factors such as imbalanced sample classes, and significantly improve the reliability of the results. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating the performance evaluation method for the abnormal website classification model provided by this invention.

[0050] Figure 2 This is a schematic diagram of the performance evaluation system for the abnormal website classification model provided by the present invention.

[0051] Figure 3 This is a flowchart illustrating the coverage fault tolerance mechanism provided by the present invention.

[0052] Figure 4 This is a schematic diagram of the performance evaluation device for the abnormal website classification model provided by the present invention.

[0053] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0055] The following is combined Figures 1-5 The specific implementation process of the present invention is described.

[0056] Figure 1 This is a flowchart illustrating the performance evaluation method for the abnormal website classification model provided by this invention.

[0057] Specifically, in combination Figure 2 The performance evaluation system for the abnormal website classification model shown is explained. This system can control and implement at least one abnormal website classification model (such as...) by controlling data such as internet traffic collected by telecommunications operators. Figure 2 The testing and evaluation of Model 1, Model 2, ..., Model N in the model, such as... Figure 2 As shown, considering the two parts of real-time data source and offline data source (the real-time data source and offline data source contain website data, such as website address URL (Uniform Resource Locator), website HTML (Hyper Text Markup Language) content, etc.), which are sequentially input into Model 1, Model 2, ..., Model N, the specific operation of the performance evaluation system of this abnormal website classification model can be divided into the following steps.

[0058] Step 101: Based on the number of abnormal websites identified by each of the at least one abnormal website classification models in the set of websites to be evaluated and the total number of real abnormal URLs, calculate the detection coverage rate of the URLs involved in the case for each of the at least one abnormal website classification models; wherein, each of the abnormal website classification models is used to classify the set of websites to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain multiple subcategories of abnormal websites.

[0059] The abnormal website classification model is used to identify and classify URLs and their content received in real-time monitoring or offline, determining whether they involve abnormal business activities, such as publishing abnormal financial information or abnormal service information. The classification results can be normal websites, abnormal websites, or further subdivisions of abnormal websites, such as further subdividing abnormal websites into financial abnormal websites, service abnormal websites, etc. The set of websites to be evaluated refers to website data obtained from the aforementioned real-time or offline data sources, including website URLs and HTML (HyperText Markup Language) webpage information. In this application, the set of websites to be evaluated can be abnormal websites identified from real-time data sources within a certain period (e.g., within one month). The meaning of the indicator "coverage rate of websites involved in cases" is explained below. First, a website involved in a case refers to a website that has actually been identified as abnormal, i.e., a website that has actually experienced abnormality. "Coverage rate of websites involved in cases" means that the abnormal websites identified by the model are indeed the websites that have actually experienced abnormality. This requires either of the following two conditions, A and B, to be considered a correct identification by the model, i.e., "coverage rate of websites involved in cases".

[0060] Condition A: The URL of the abnormal website identified by the model is the same as the domain name of the website involved in the case, for example: http: / / 54315429.cc / xxx and https: / / 54315429.cc / yyyyy.

[0061] Condition B: The cosine similarity of the two embedded vectors obtained by the BERT (Bidirectional Encoder Representations from Transformers) model from the URLs of the abnormal website (URL 1) and the website involved in the case (URL 2) identified by the model is higher than 95%, denoted as Embedding1 and Embedding2 (taking the second-to-last word corresponding to the [CLS] token, i.e., token embedding). The [CLS] token is a special token placed at the beginning of a sentence or document to indicate the central theme.

[0062] If either condition A or condition B is met, it is considered that the "website in question is covered," or "website fault tolerance" is applied.

[0063] This coverage tolerance mechanism, which combines domain name matching and embedded vector similarity, reduces accidental errors that may occur when regulators register websites involved in cases, and improves the accuracy of coverage calculation.

[0064] "Case-related website coverage rate" mainly refers to the percentage of abnormal websites correctly identified by the model among the total number of cases-related websites.

[0065] Because abnormal websites are time-sensitive, meaning they may cease to exist after a period of time, this affects the evaluation of model technical indicators in abnormal website detection scenarios. Therefore, both "accessible website coverage" and "overall coverage" are considered.

[0066] The “accessible website coverage rate” is calculated as follows: the accessible website coverage rate is obtained by taking the proportion of the number of accessible model-identified covered websites to the total number of accessible involved websites; the number of accessible model-identified covered websites is the number of accessible websites among the abnormal websites correctly identified by each model; the total number of accessible involved websites refers to the number of involved websites in the set of websites to be evaluated that have actually occurred and are still accessible.

[0067] The “overall coverage rate” is calculated as follows: the overall coverage rate is obtained based on the proportion of the number of websites covered by the model in the total number of websites involved in the case; where the number of websites covered by the model refers to the number of abnormal websites correctly identified by each model from the set of websites to be evaluated; and the total number of websites involved in the case refers to the websites that actually occurred.

[0068] The coverage rate of the websites involved in the case was obtained based on the accessible website coverage rate and the overall coverage rate.

[0069] In summary, the calculation of the coverage rate of the websites involved in the case... as follows.

[0070] ;

[0071] in, and These are configurable weight parameters. =1; "Number of accessible websites covered by model identification" e refers to the number of accessible websites among the anomalous websites correctly identified by each model; "Total number of accessible websites involved in the case" E refers to the number of websites in the set of websites to be evaluated that have actually occurred and are still accessible; "Number of websites covered by model identification" f refers to the number of anomalous websites correctly identified by each model from the set of websites to be evaluated, including both accessible and inaccessible websites; "Total number of websites involved in the case" F refers to the websites involved in the case that actually occurred, including both accessible and inaccessible websites to date.

[0072] Based on the coverage of the websites involved in the case according to each model The models were ranked from highest to lowest to obtain the coverage rate of the websites involved in the case. Ranking order This leads to evaluation item 1 (i.e., the coverage of the websites involved in the case). Normalized score Among them, the ranking normalization scoring function It can be defined as follows (this is just an example).

[0073] ;

[0074] in, This represents the coverage rate of the i-th website in the case, ranked from highest to lowest. .

[0075] In particular, the coverage rate of websites involved in fraud cases can also be calculated for the websites used in on-site investigations. as follows:

[0076] );

[0077] Where m is the number of accessible model identification sites, M is the total number of accessible sites, n is the number of model identification coverage sites, and N is the total number of sites.

[0078] Step 102: Based on the number of sub-category abnormal websites identified by each abnormal website classification model and the total number of actual sub-category abnormal websites, calculate the overall category coverage rate of each abnormal website classification model; wherein, the overall category coverage rate is a weighted average of the coverage rates of each sub-category.

[0079] This step considers the distribution of recognition volume of each subcategory within the total number of recognized websites, and reflects this through the metric of full category coverage (also known as overall category coverage).

[0080] Specifically, regarding the coverage rate of abnormal websites identified by the model for each subcategory, from a business perspective, since the identified samples of financial abnormal websites, service abnormal websites, and other abnormal websites have different importance, this application considers evaluating the proportion of the identified number of financial abnormal websites, service abnormal websites, and other abnormal websites out of the total number of identified websites, denoted as follows: , , Therefore, the overall category coverage rate is calculated. for.

[0081] ;

[0082] in, 1. These represent the weight parameters for the proportion of service-type exception subcategories and the weight parameters for the proportion of other exception categories (with appropriate weight reduction for service-type exceptions and other exception categories). Based on the overall category coverage... Rank the models from highest to lowest to obtain the ranking order. This yields the overall category coverage rate. Normalized score .

[0083] Step 103: Determine the website sampling sample set for each subcategory.

[0084] It should be noted that in actual website classification performance evaluation, using sampling methods to select a sample set and relying on the results of that sample set to reflect the overall model performance can lead to an over-reliance on the selected samples. If the sample set is small, the evaluation results may not accurately represent the model's performance, resulting in significant sampling errors. Conversely, while a larger sample set reduces sampling errors, it requires more manual inspection, leading to substantial labor costs, and repeated evaluations of highly similar samples are inefficient. Consequently, we may sample different types of samples, but existing sampling allocation methods can result in unreasonable allocation of human resources and a large overall sampling error. Therefore, this application proposes a multi-website collaborative sampling method.

[0085] The principles considered in this sampling method include: the more complex the structure of the anomalous website (reflected by the TAG tags in the website code), the more samples of that type should be taken; the later the anomalous website is identified (i.e., the earlier the website first appears in DPI (Deep Packet Inspect), the more samples of that type should be taken).

[0086] The target types for sampling are the original subcategories (i.e., sub-categories), including: abnormal financial websites, abnormal academic service websites, abnormal examination service websites, abnormal identity fraud websites, abnormal relationship service websites, and n other categories.

[0087] First, initial samples are randomly given for each subclass. For a given subclass, assume the model identifies... An abnormal website, its initial sampling function The definition is as follows.

[0088] ;

[0089] in, , M is a configurable constant, representing the number of abnormal websites identified by the model from this subclass. The value is a percentage and can be flexibly set as needed; N is a constant and can be flexibly set as needed. This application uses different initial sampling functions for different major categories. For example, the subcategory of financial anomalies belonging to the financial category uses... Subcategories belonging to service-related exceptions adopt Subclasses belonging to other exception classes adopt .

[0090] Step 104: Calculate the probability that the samples in the website sampling set of each subcategory are correctly classified to obtain the subcategory prediction accuracy; determine the overall subcategory prediction accuracy based on the weighted sum of the prediction accuracy of each subcategory.

[0091] Next, evaluation metrics are calculated for the various types of samples sampled in step 103 above. Unlike traditional metrics such as accuracy and recall, this application defines the overall sub-category prediction accuracy. This serves as the evaluation metric for the model's classification performance in this scenario. This represents the total number of truly occurring anomalous website samples across all subcategories in the total sample. This represents the final abnormal website resulting from the combination of all subcategories. This represents the number of truly anomalous website samples in the i-th subcategory. The number of samples representing the predicted abnormal website for the i-th subcategory; for the abnormal website classification model to be detected, the probability of correctly classifying the i-th subcategory. for.

[0092] ;

[0093] For all subcategories, the probability that the model classification result matches the true label. for:

[0094] ;

[0095] Based on the number of websites classified as anomalous by the anomalous website classification model in the website sampling set for each category, and the actual number of anomalous websites in the website sampling set, the conditional probability of each subcategory being accurately classified is obtained; where... This represents the conditional probability that the i-th subcategory is correctly classified.

[0096] For the i-th category,

[0097] ;

[0098] Then, the subclass prediction accuracy of each subclass is defined. for.

[0099] ;

[0100] In other words, the subclass prediction accuracy of each subclass is obtained based on the conditional probability of each subclass being correctly classified; where, Let represent the conditional probability that the j-th subclass is correctly classified. This represents the number of truly abnormal websites in the j-th subcategory. This represents the number of abnormal websites detected by the model in the j-th subcategory. Let w represent the number of samples correctly classified into the j-th subclass, and w be the total number of samples. The weight corresponding to the i-th sub-category. The closer to 1, the better the overall model classification performance.

[0101] Based on business objectives, the evaluation focuses on the accuracy (e.g., accurate, suspected, inaccurate) of model identification across three main categories: financial anomalies, service anomalies, and other anomalies. Each testing party customizes a mapping table from the original subcategories to the evaluation categories. The Work Percentage (WMP) for each category (containing at least one corresponding subcategory, such as service anomalies including academic anomalies, exam anomalies, etc.) is defined as follows.

[0102] ;

[0103] in, Configure according to business relevance and importance.

[0104] Calculate the WMP value for financial management anomalies. WMP value of service-type exceptions Other abnormal WMP values Therefore, the overall category evaluation value is calculated. for.

[0105] ;

[0106] in, .in accordance with Rank the models from highest to lowest to obtain Thus, the normalized score of evaluation item 3 was obtained. .

[0107] Step 105: Based on the statistical indicators of the identification time of each abnormal website classification model for each website in the set of websites to be evaluated, determine the timeliness evaluation indicators for each abnormal website classification model.

[0108] Specifically, for each model 1-N, the same amount of data from the same URL is submitted daily through a real-time data source, and the model deployment environment is an intranet. The model's timeliness performance is compared by comparing the model recognition time at each URL (= completion recognition time - start recognition time). Since the different recognition end times of each model may lead to different start times for subsequent evaluations, this application constructs a batch aligner, and the specific implementation steps are as follows.

[0109] For each batch of website data files fed into the abnormal website classification model, calculate the recognition time from data input to classification result output, and calculate the average recognition time. quartile values Maximum value .

[0110] Therefore, the timeliness evaluation index of this abnormal website classification model is calculated. for.

[0111] ;

[0112] in, , and For normalization parameters, and .in accordance with Rank the models from low to high to obtain Thus, the normalized scores of the timeliness evaluation indicators for each model are obtained. .

[0113] ;

[0114] in, This is the normalized scoring function.

[0115] Step 106: Determine the comprehensive evaluation score of each abnormal website classification model based on any one or more of the following: the detection coverage rate of the website in question, the overall category coverage rate, the prediction accuracy rate of all subcategories, and the timeliness evaluation indicators.

[0116] Final model overall evaluation score (0-100) is...

[0117] ;

[0118] The above embodiments calculate the detection coverage rate of the involved URLs for each of the at least one abnormal website classification models based on the number of abnormal websites identified by each model in the set of websites to be evaluated and the total number of actual abnormal URLs. Each of the abnormal website classification models is used to classify the set of websites to be evaluated into abnormal websites, and to classify the abnormal websites into multiple subcategories of abnormal websites. The overall category coverage rate of each abnormal website classification model is calculated based on the number of subcategories of abnormal websites identified by each model and the total number of actual subcategories of abnormal URLs. The overall category coverage rate is... The method involves: weighting the coverage rates of each subcategory; determining the website sampling set for each subcategory; calculating the probability that samples in the website sampling set of each subcategory are accurately classified, thus obtaining the subcategory prediction accuracy; determining the overall subcategory prediction accuracy based on the weighted sum of the prediction accuracy rates of each subcategory; determining the timeliness evaluation index for each abnormal website classification model based on the statistical index of the identification time for each website in the set of websites to be evaluated by each abnormal website classification model; and determining the comprehensive evaluation score of each abnormal website classification model based on any one or more of the following: the detection coverage rate of the involved website, the overall category coverage rate, the overall subcategory prediction accuracy, and the timeliness evaluation index. This method designs different indicators to evaluate the evaluation effect of the abnormal website classification model and can use multiple indicators for collaborative evaluation, improving the comprehensiveness of the evaluation effect. It considers not only the model's classification of normal and abnormal websites but also the classification effect of different subcategories within abnormal websites, thus improving the accuracy of model evaluation. Furthermore, this application also uses a multi-website collaborative sampling mechanism, which can effectively identify and resolve the discrepancies in classification performance evaluation caused by factors such as imbalanced sample classes, and significantly improve the reliability of the results.

[0119] In one embodiment, step 103 above includes:

[0120] The number of websites in the sample set for each subcategory is determined based on the complexity of the website structure and / or the website development time.

[0121] The website structure complexity is determined based on the number of tags in the Hypertext Markup Language corresponding to each website; the website development time refers to the time elapsed between the moment the website first appears in network data packets and the time it is classified as a website.

[0122] Specifically, it includes the following steps.

[0123] Calculate the website structure complexity of each website in the set of websites to be sampled; wherein the website structure complexity is determined based on the number of tags in the Hypertext Markup Language corresponding to each website.

[0124] Determine multiple initial sample sets for each subcategory.

[0125] Multiple rounds of website sample quantity update operations are performed on the multiple initial sample sets.

[0126] In each round of website removal, after updating the first preset number of URL samples from the multiple initial sampling sample sets, the variance of the number of tags in each updated sampling sample set is calculated.

[0127] Based on the variance of the number of labels in each updated sample set, calculate the error rate of change between the variance before and after the update; update the first preset quantity; return to the step of calculating the variance of the number of labels in each updated sample set after updating the preset number of URL samples from the multiple initial sample sets in each round of website elimination operation.

[0128] Select the initial sample set corresponding to the smallest error change rate and the first preset quantity corresponding to the smallest error change rate.

[0129] Based on the initial sample set corresponding to the minimum error change rate and the first preset quantity corresponding to the minimum error change rate, determine the website sample set corresponding to the current sub-category.

[0130] It should be noted that, in order to allocate samples more reasonably, the number of samples and the sample set for each type of website need to be updated according to the characteristics of the anomaly type itself.

[0131] (1) For website types with simple structures (corresponding to small variance of TAG number), the number of samples can be reduced by removing some samples from the sample set.

[0132] Input: n classes of URLs to be evaluated, and the number of samples for each class. The corresponding initial sample set .

[0133] Output: URL type i that can reduce the sampling size, and the URL samples to be removed. .

[0134] For i=1:n

[0135] Calculate the sample set of this type of URL raw variance of TAG number

[0136] For j=1:

[0137] From the URL sample set Remove from The new sample set obtained later

[0138] Calculate the variance of the number of URL tags for this type of new sample set.

[0139] Calculate the rate of change of error

[0140] End

[0141] From all (j=1,2,..., Find the minimum value in the sample URLs and its corresponding URLs. .

[0142] End

[0143] return The minimum value corresponds to the URL type index i, and the URL sample. .

[0144] (2) For website types with complex structures (corresponding to large variance in the number of TAGs) that require an increased number of samples, select some samples from the remaining URLs and add them to the sample set.

[0145] The calculation process is as follows.

[0146] Input: n classes of URLs to be evaluated, and the number of samples for each class. The corresponding initial sample set and the set of samples not included in the initial sampling. .

[0147] Output: URL type i that needs increased sampling size, and the number of additional URL samples needed. .

[0148] For i=1:n

[0149] Calculate the sample set of this type of URL raw variance of TAG number

[0150] The sample set that was never in the initial sampling T URLs are randomly selected from the sample (T is a constant and can be set).

[0151] For k=1:T

[0152] From the selected Chinese URL sample set Supplementing new samples The new sample set obtained later

[0153] Calculate the variance of the number of URL tags for this type of new sample set.

[0154] Calculate the rate of change of error

[0155] End

[0156] From all (j=1,2,..., Find the maximum value in the sample URL and its corresponding URL. .

[0157] End

[0158] return The URL type index i corresponding to the maximum value, and the URL sample .

[0159] (3) For website types with earlier development time (corresponding to earlier first appearance time in DPI), the number of samples can be reduced by removing some samples from the sample set.

[0160] (4) For website types with late development time (corresponding to late first appearance time in DPI), the number of samples needs to be increased. Select some samples from the remaining websites and add them to the sample set.

[0161] The above embodiments, by updating the sample set considering the characteristics of the anomaly type itself, can improve the accuracy of the sampling results.

[0162] The performance evaluation device for the abnormal website classification model provided by the present invention is described below. The performance evaluation device for the abnormal website classification model described below can be referred to in correspondence with the performance evaluation method for the abnormal website classification model described above.

[0163] Figure 4 This is a schematic diagram of the modules of the performance evaluation device for the abnormal website classification model provided by the present invention. The performance evaluation device for the abnormal website classification model includes the following modules:

[0164] The involved website URL detection coverage calculation module 401 is used to calculate the involved website URL detection coverage of each of the at least one abnormal website classification models based on the number of abnormal websites identified by each abnormal website classification model in the website set to be evaluated and the total number of actual abnormal URLs; wherein, each of the abnormal website classification models is used to classify the website set to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain multiple subcategories of abnormal websites; the involved website URL detection coverage is used to represent the degree of consistency between the number of abnormal websites identified by the abnormal website classification model and the total number of actual abnormal websites.

[0165] The overall category coverage calculation module 402 is used to calculate the overall category coverage of each abnormal website classification model based on the number of sub-category abnormal websites identified by each abnormal website classification model and the total number of real sub-category abnormal URLs; wherein, the overall category coverage is a weighted sum of the coverage of each sub-category.

[0166] Website sampling module 403 is used to determine the website sampling sample set for each subcategory;

[0167] The overall subcategory prediction accuracy calculation module 404 is used to calculate the probability that the samples in the website sampling sample set of each subcategory are classified accurately, and obtain the subcategory prediction accuracy; based on the weighted sum of the prediction accuracy of each subcategory, the overall subcategory prediction accuracy is determined.

[0168] The timeliness assessment index calculation module 405 is used to determine the timeliness assessment index of each abnormal website classification model based on the statistical index of the identification time of each website in the set of websites to be evaluated for each abnormal website classification model.

[0169] The model comprehensive evaluation module 406 is used to determine the model comprehensive evaluation score of each abnormal website classification model based on any one or more of the following: the detection coverage rate of the website involved in the case, the overall category coverage rate, the prediction accuracy rate of all subcategories, and the timeliness evaluation indicators.

[0170] In one embodiment, the aforementioned website detection coverage calculation module 401 is further configured to: obtain an accessible website coverage rate based on the proportion of the number of accessible model-identified covered websites in the total number of accessible website-identified websites; wherein, the number of accessible model-identified covered websites is the number of accessible websites among the abnormal websites correctly identified by each model; the total number of accessible website-identified websites refers to the number of website-identified websites in the set of websites to be evaluated that have actually occurred and are still accessible; obtain an overall coverage rate based on the proportion of the number of model-identified covered websites in the total number of website-identified websites; wherein, the number of model-identified covered websites refers to the number of abnormal websites correctly identified by each model from the set of websites to be evaluated; the total number of website-identified websites refers to the actual website-identified websites; and obtain the website-identified coverage rate based on the accessible website coverage rate and the overall coverage rate.

[0171] In one embodiment, the website sampling module 403 is further configured to: determine the number of websites in the sample set for each subcategory based on the website structure complexity; and / or, determine the number of websites in the sample set for each subcategory based on the website development time; wherein the website structure complexity is determined based on the number of tags in the Hypertext Markup Language corresponding to each website; and the website development time refers to the time elapsed between the moment the website first appears in the network data packet and the time when the website is classified.

[0172] In one embodiment, the website sampling module 403 is further configured to: calculate the website structure complexity of each website in the website set to be sampled; wherein the website structure complexity is determined based on the number of tags in the Hypertext Markup Language corresponding to each website; determine multiple initial sampling sample sets for each subcategory; perform multiple rounds of website sample number update operations on the multiple initial sampling sample sets; in each round of website elimination operation, after updating a first preset number of URL samples from the multiple initial sampling sample sets, calculate the variance of the number of tags in each updated sampling sample set; and calculate the variance of the number of tags in each updated sampling sample set based on the number of tags in each updated sampling sample set. The variance of the number of labels in the sample set is calculated, and the error rate of change between the variance before and after the update is calculated. The first preset quantity is updated. In each round of website elimination operation, after updating the preset number of website samples from the multiple initial sample sets, the variance of the number of labels in each updated sample set is calculated. The initial sample set corresponding to the smallest error rate of change and the first preset quantity corresponding to the smallest error rate of change are selected. Based on the initial sample set corresponding to the smallest error rate of change and the first preset quantity corresponding to the smallest error rate of change, the website sample set corresponding to the current subcategory is determined.

[0173] In one embodiment, the above-mentioned model comprehensive evaluation module 406 is further used for:

[0174] The comprehensive evaluation score of each abnormal website classification model is determined by weighting any number of indicators among the following: the detection coverage rate of the website in question, the overall category coverage rate, the prediction accuracy rate of all subcategories, and the timeliness evaluation indicators.

[0175] In one embodiment, the above-mentioned all-sub-category prediction accuracy calculation module 404 is further configured to:

[0176] Based on the number of websites classified as abnormal by the abnormal website classification model in the website sampling set for each category and the actual number of abnormal websites in the website sampling set, the conditional probability of each subcategory being correctly classified is obtained; based on the conditional probability of each subcategory being correctly classified, the subcategory prediction accuracy of each subcategory is obtained.

[0177] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a performance evaluation method for an abnormal website classification model. This method includes: calculating the detection coverage rate of the involved URLs for each of the at least one abnormal website classification models based on the number of abnormal websites identified by each model in the set of websites to be evaluated and the total number of actual abnormal URLs; wherein each abnormal website classification model is used to classify the set of websites to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain multiple subcategories of abnormal websites; and calculating the detection coverage rate of each abnormal website classification model based on the number of subcategories of abnormal websites identified by each abnormal website classification model and the total number of actual subcategories of abnormal URLs. The overall category coverage rate is calculated as follows: the overall category coverage rate is the weighted average of the coverage rates of each subcategory; a sample set of websites for each subcategory is determined; the probability that samples in the sample set of websites for each subcategory are accurately classified is calculated to obtain the subcategory prediction accuracy rate; the overall subcategory prediction accuracy rate is determined based on the weighted average of the prediction accuracy rates of each subcategory; the timeliness evaluation index of each abnormal website classification model is determined based on the statistical index of the identification time of each website in the set of websites to be evaluated for each abnormal website classification model; and the comprehensive evaluation score of each abnormal website classification model is determined based on any one or more of the following: the detection coverage rate of the website in question, the overall category coverage rate, the overall subcategory prediction accuracy rate, and the timeliness evaluation index.

[0178] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes at least one instruction to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0179] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the performance evaluation method of the abnormal website classification model provided by the above methods. The method includes: calculating the coverage rate of the website detection of each of the at least one abnormal website classification models for the number of abnormal websites identified in the website set to be evaluated and the total number of real abnormal URLs, based on the number of abnormal websites identified by each of the at least one abnormal website classification models in the website set to be evaluated and the total number of real abnormal URLs; wherein, each abnormal website classification model is used to classify the website set to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain multiple subcategories of abnormal websites; based on the subcategories identified by each abnormal website classification model... The overall category coverage rate of each abnormal website classification model is calculated based on the number of abnormal websites and the total number of real abnormal URLs in each subcategory. The overall category coverage rate is a weighted average of the coverage rates of each subcategory. A website sampling set is determined for each subcategory. The probability that a sample in each subcategory's website sampling set is accurately classified is calculated to obtain the subcategory prediction accuracy rate. The overall subcategory prediction accuracy rate is determined based on the weighted average of the prediction accuracy rates of each subcategory. A timeliness evaluation index is determined for each abnormal website classification model based on the statistical indicator of the identification time for each website in the evaluation set. Finally, the comprehensive evaluation score of each abnormal website classification model is determined based on any one or more of the following: the detection coverage rate of the involved URLs, the overall category coverage rate, the overall subcategory prediction accuracy rate, and the timeliness evaluation index.

[0180] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements a performance evaluation method for the abnormal website classification models provided by the methods described above. This method includes: calculating the coverage rate of the URLs involved in a case for each of the at least one abnormal website classification models based on the number of abnormal websites identified by each model in a set of websites to be evaluated and the total number of actual abnormal URLs; wherein each abnormal website classification model is used to classify the set of websites to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain multiple subcategories of abnormal websites; based on the number of subcategories of abnormal websites identified by each model and the total number of actual subcategories of abnormal websites... The total number of regular URLs is used to calculate the overall category coverage rate of each abnormal website classification model; the overall category coverage rate is a weighted average of the coverage rates of each sub-category; a website sampling sample set is determined for each sub-category; the probability that samples in the website sampling sample set of each sub-category are accurately classified is calculated to obtain the sub-category prediction accuracy rate; the overall sub-category prediction accuracy rate is determined based on the weighted average of the prediction accuracy rates of each sub-category; the timeliness evaluation index of each abnormal website classification model is determined based on the statistical index of the identification time of each website to be evaluated in the set of websites to be evaluated for each abnormal website classification model; the comprehensive evaluation score of each abnormal website classification model is determined based on any one or more of the following: the detection coverage rate of the URLs involved in the case, the overall category coverage rate, the overall sub-category prediction accuracy rate, and the timeliness evaluation index.

[0181] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0182] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including at least one instruction to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A performance evaluation method for an abnormal website classification model, characterized in that, include: Based on the number of abnormal websites identified by each of the at least one abnormal website classification models in the set of websites to be evaluated and the total number of actual abnormal websites, the coverage rate of the websites involved in the case for each of the at least one abnormal website classification models is calculated; wherein, each of the abnormal website classification models is used to classify the set of websites to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain multiple subcategories of abnormal websites; the coverage rate of the websites involved in the case is used to represent the degree of consistency between the number of abnormal websites identified by the abnormal website classification model and the total number of actual abnormal websites. Based on the number of abnormal websites in each subcategory identified by each abnormal website classification model and the total number of actual abnormal websites in each subcategory, the overall category coverage rate of each abnormal website classification model is calculated; wherein, the overall category coverage rate is a weighted average of the coverage rates of each subcategory. Determine the website sampling set for each subcategory; Calculate the probability that the samples in the website sampling set for each subcategory are correctly classified to obtain the subcategory prediction accuracy; determine the overall subcategory prediction accuracy by weighting the prediction accuracy of each subcategory. Based on the statistical indicators of the identification time of each abnormal website classification model for each website in the set of websites to be evaluated, the timeliness evaluation indicators of each abnormal website classification model are determined. The comprehensive evaluation score of each abnormal website classification model is determined based on any one or more of the following: the detection coverage rate of the websites involved, the overall category coverage rate, the prediction accuracy rate of all subcategories, and the timeliness evaluation indicators.

2. The performance evaluation method for the abnormal website classification model according to claim 1, characterized in that, The number of identified abnormal websites includes the number of accessible websites covered by the model identification and the number of websites actually involved in the case; the total number of actually involved abnormal websites includes the total number of accessible websites involved in the case and the total number of websites actually involved in the case; the calculation of the coverage rate of websites involved in the case for each of the at least one abnormal website classification models for the number of abnormal websites identified and the total number of actually involved abnormal websites in the set of websites to be evaluated, based on the number of abnormal websites identified and the total number of actually involved abnormal websites in the set of websites to be evaluated, includes: The accessible website coverage rate is obtained based on the proportion of the number of accessible model-identified covered websites in the total number of accessible involved websites; wherein, the number of accessible model-identified covered websites is the number of accessible websites among the abnormal websites correctly identified by each model; the total number of accessible involved websites refers to the number of involved websites in the set of websites to be evaluated that have actually occurred and are still accessible to date. The overall coverage rate is obtained based on the proportion of the number of websites covered by the model in the total number of websites involved in the case; wherein, the number of websites covered by the model refers to the number of abnormal websites correctly identified by each model from the set of websites to be evaluated; and the total number of websites involved in the case refers to the websites that actually occurred in the case. The coverage rate of the website in question is obtained based on the accessible website coverage rate and the overall coverage rate.

3. The performance evaluation method for the abnormal website classification model according to claim 1, characterized in that, The determination of the website sampling set for each subcategory includes: Determine the number of websites in the sample set for each subcategory based on the complexity of the website structure; And / or, Determine the number of websites in the sample set for each subcategory based on their development time. The website structure complexity is determined based on the number of tags in the Hypertext Markup Language corresponding to each website; the website development time refers to the time elapsed between the moment the website first appears in network data packets and the time it is classified as a website.

4. The performance evaluation method for the abnormal website classification model according to claim 3, characterized in that, Based on the complexity of the website structure, determine the number of websites in the sample set for each subcategory, including: Calculate the website structure complexity of each website in the set of websites to be sampled; wherein the website structure complexity is determined based on the number of tags in the Hypertext Markup Language corresponding to each website; Determine multiple initial sample sets for each subcategory; Perform multiple rounds of website sample count update operations on the multiple initial sample sets; In each round of website elimination, after updating the first preset number of URL samples from the multiple initial sampling sample sets, the variance of the number of tags in each updated sampling sample set is calculated. Based on the variance of the number of labels in each updated sample set, calculate the error rate of change between the variance before and after the update; update the first preset quantity; return to the step of calculating the variance of the number of labels in each updated sample set after updating the preset number of URL samples from the multiple initial sample sets in each round of website elimination operation. Select the initial sample set corresponding to the smallest error change rate and the first preset quantity corresponding to the smallest error change rate; Based on the initial sample set corresponding to the minimum error change rate and the first preset quantity corresponding to the minimum error change rate, determine the website sample set corresponding to the current sub-category.

5. The performance evaluation method for the abnormal website classification model according to claim 1, characterized in that, The comprehensive evaluation score of each abnormal website classification model is determined based on any one or more of the following: the detection coverage rate of the websites involved, the overall category coverage rate, the prediction accuracy rate of all subcategories, and the timeliness evaluation indicators. The comprehensive evaluation score of each abnormal website classification model is determined by weighting any number of indicators among the website detection coverage rate, overall category coverage rate, prediction accuracy of all subcategories, and timeliness evaluation indicators.

6. The performance evaluation method for the abnormal website classification model according to claim 1, characterized in that, The calculation of the probability that samples in the website sampling set for each sub-category are correctly classified, to obtain the sub-category prediction accuracy, includes: Based on the number of websites in the website sampling set of each category that are classified as abnormal websites by the abnormal website classification model and the number of actual abnormal websites in the website sampling set, the conditional probability of each subcategory being accurately classified is obtained. Based on the conditional probability of each subcategory being correctly classified, the subcategory prediction accuracy of each subcategory is obtained.

7. A performance evaluation device for an abnormal website classification model, characterized in that, include: The module for calculating the coverage rate of the website addresses involved in the case is used to calculate the coverage rate of the website addresses involved in the case for each of the at least one abnormal website classification models based on the number of abnormal websites identified by each abnormal website classification model in the set of websites to be evaluated and the total number of real abnormal websites; wherein, each of the abnormal website classification models is used to classify the set of websites to be evaluated to obtain abnormal websites, and to classify the abnormal websites to obtain multiple subcategories of abnormal websites. The overall category coverage calculation module is used to calculate the overall category coverage of each abnormal website classification model based on the number of sub-category abnormal websites identified by each abnormal website classification model and the total number of real sub-category abnormal URLs; wherein, the overall category coverage is a weighted sum of the coverage of each sub-category. The website sampling module is used to determine the website sampling sample set for each subcategory; The overall subcategory prediction accuracy calculation module is used to calculate the probability that the samples in the website sampling sample set of each subcategory are correctly classified, and obtain the subcategory prediction accuracy; the overall subcategory prediction accuracy is determined by weighting the prediction accuracy of each subcategory. The timeliness assessment index calculation module is used to determine the timeliness assessment index of each abnormal website classification model based on the statistical index of the identification time of each website in the set of websites to be evaluated for each abnormal website classification model. The model comprehensive evaluation module is used to determine the model comprehensive evaluation score of each abnormal website classification model based on any one or more of the following: the detection coverage rate of the website involved in the case, the overall category coverage rate, the prediction accuracy rate of all subcategories, and the timeliness evaluation indicators.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the performance evaluation method for the abnormal website classification model as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the performance evaluation method for the abnormal website classification model as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the performance evaluation method for the abnormal website classification model as described in any one of claims 1 to 6.

Images