A malicious webpage classification model construction method and system, and a storage medium

Abstract

The application discloses a malicious webpage classification model construction method and system and a storage medium, wherein the method determines the category of a webpage through HTML content, thereby providing an accurate basis for subsequent classification; the average TF-IDF value of the text in the HTML label is calculated through a TF-IDF algorithm, thereby avoiding the interference of noise and irrelevant information; the HTML label and the text are combined, and the text is processed using a word segmentation algorithm, thereby providing a basis for subsequent representation and processing; the word vector is encoded and processed through a word embedding technology combined with an encoder, thereby realizing the deep representation and understanding of the text; finally, a malicious webpage classification model is constructed according to the text classification result, and the existing data of manually labeled webpage categories are used for training, thereby improving the accuracy of the malicious webpage classification model. The model constructed by the application does not need to be used in cooperation with other models, thereby reducing the required computing resources and reducing the complexity of model deployment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, in particular to a malicious webpage classification model construction method and system and a storage medium. BACKGROUND

[0002] Malicious webpages are webpages containing malicious codes, fraudulent content, false information or other information that can harm network users. Common types include phishing webpages, malicious download pages, fraudulent websites and malicious advertising webpages. Depending on the type of malicious webpage, the potential harm may vary: for example, attack-type malicious webpages such as phishing attacks, malicious advertising and botnets can cause large-scale spread, leading to a large number of confidential information leaks or losses and causing significant economic losses; for example, malicious webpages used to spread false information or engage in illegal inducement can easily lead to reputational or financial losses for vulnerable groups. Therefore, accurate classification of malicious webpages is an effective means of maintaining network order and user rights.

[0003] Invention patent application No. 2022115903674 discloses a malicious website identification method, the steps of which include:

[0004] Obtaining a to-be-identified website, obtaining an identification address location, website domain name information and webpage text content from the to-be-identified website;

[0005] Filtering the to-be-identified website through a preset whitelist website list to obtain a to-be-identified website;

[0006] Scoring the number of to-be-identified websites according to the identification address location and the website domain name information to obtain a number of website scores;

[0007] Identifying the webpage text content through a target multi-dimensional fingerprint model and the number of website scores, and determining whether the number of to-be-identified websites are malicious websites according to the identification result.

[0008] This malicious website identification method takes into account the identification address location, website domain name information and webpage text content of the to-be-identified website, and identifies the webpage text content from multiple dimensions, which is more comprehensive. However, it cannot accurately classify the types of malicious webpages.

[0009] Invention patent application No. 2022115903509 discloses a malicious website processing method, device, equipment and storage medium, which includes the steps of:

[0010] Obtaining webpage text, webpage label sequence and webpage screenshot from a to-be-processed malicious website;

[0011] extract feature information of the webpage screenshot, and identify the feature information through a target webpage image classification model to obtain a category of the webpage screenshot;

[0012] when the content of the webpage text is less than a preset text content threshold and the content of the webpage label sequence is less than a preset label sequence threshold, identify the webpage text through a target text classification model and identify the webpage label sequence through a target label sequence classification model;

[0013] determine a category of the malicious website to be processed according to the identification result and the category of the webpage screenshot;

[0014] determine a target website processing strategy according to the category of the malicious website to be processed, and process the malicious website to be processed according to the target website processing strategy.

[0015] The processing method of the malicious website disclosed in the application aims to divide the categories of malicious websites through webpage texts, webpage label sequences and webpage screenshots, and for this purpose, a target webpage image classification model, a target text classification model and a target label sequence classification model are constructed, and it is not difficult to understand that this judgment method not only requires a large amount of computing resources, but also has high complexity of model deployment.

[0016] It can be seen that the prior art still needs to be improved and developed. SUMMARY

[0017] In view of the above shortcomings of the prior art, the purpose of the application is to provide a malicious webpage classification model construction method, system and storage medium, aiming to solve the problem that the existing malicious website classification method not only requires a large amount of computing resources, but also has high complexity of model deployment.

[0018] The technical scheme of the application is as follows:

[0019] A malicious webpage classification model construction method comprises:

[0020] obtaining HTML content of a webpage to be identified through a web crawler, and determining a category of the webpage to be identified;

[0021] calculating average TF-IDF values of texts in each HTML label in the obtained HTML content through a TF-IDF algorithm, and deleting HTML labels and texts corresponding to the HTML labels with average TF-IDF values lower than a preset value;

[0022] combining the HTML labels with the texts corresponding to the HTML labels, performing word segmentation processing on the texts through a word segmentation algorithm, and constructing a word table according to a word segmentation result;

[0023] The segmented webpage text and HTML tags are converted into a word vector matrix through word embedding, and a Transformer encoder is used to encode the word vector for text classification.

[0024] A malicious webpage classification model is constructed according to the text classification result, and the constructed model is trained and updated through webpage text data with manually labeled webpage categories.

[0025] The effect of the above scheme is that the HTML content of the webpage to be recognized is obtained through the network crawler, and the category of the webpage is determined, thereby providing an accurate basis for subsequent classification; the average TF-IDF value of the text in the HTML tag is calculated through the TF-IDF algorithm, which can filter out important text information and avoid the interference of noise and irrelevant information; the HTML tag and the text are combined, and the text is processed using a segmentation algorithm, which can segment the text into independent words or phrases, and construct a word table, thereby providing a basis for subsequent representation and processing; the segmented webpage text and HTML tags are converted into a word vector matrix through word embedding technology, and the word vector is encoded through a Transformer encoder, thereby realizing the deep representation and understanding of the text; finally, a malicious webpage classification model is constructed according to the text classification result, and the model is trained through existing data with manually labeled webpage categories, thereby improving the accuracy of the malicious webpage classification model. Moreover, the malicious webpage classification model constructed by the present application does not need to be used with other models, thereby reducing the required computing resources and reducing the complexity of model deployment.

[0026] In a further preferred scheme, the step of obtaining the HTML content of the webpage to be recognized through the network crawler and determining the category of the webpage to be recognized specifically comprises:

[0027] The HTML content of the webpage to be recognized is obtained through the network crawler, and the HTML content includes page layout and page content.

[0028] The obtained HTML content is structured, the HTML tags are extracted, the text content in the webpage pictures is recognized through OCR technology, and the recognized text content and the directly extracted text content are associated with the extracted HTML tags, respectively.

[0029] The webpage category is matched in the sample library according to the HTML tags and the text content, and if the matching is unsuccessful, the webpage category is manually labeled.

[0030] The effect of the above scheme is that the crawling, structured processing, OCR and sample matching methods are comprehensively used, so that the determination of the webpage category is more accurate and efficient, and the precision and reliability of the malicious webpage classification model are improved. At the same time, the process of constructing the malicious webpage classification model is simplified, the demand for manual intervention is reduced, and the automation degree is improved, thereby reducing the cost and consumption of human resources.

[0031] In a further preferred scheme, the step of calculating the average TF-IDF value of the text in each HTML tag in the obtained HTML content by the TF-IDF algorithm, and deleting the HTML tags and the corresponding text whose average TF-IDF value is lower than a preset value, specifically comprises:

[0032] performing word segmentation processing on the text corresponding to each HTML tag using a word segmenter;

[0033] deleting the stop words and special symbols in the text after word segmentation processing, and calculating the TF-IDF value of each word remaining;

[0034] calculating the average TF-IDF value of all words in each HTML tag;

[0035] according to the calculation result, deleting the HTML tags and the corresponding text whose average TF-IDF value is lower than a preset value.

[0036] The effect of the above scheme is that the text in each HTML tag is processed by the word segmenter, which can effectively divide the text into independent words; the stop words and special symbols in the text after word segmentation are deleted, which can eliminate the content with less contribution to the meaning of the text; the TF-IDF value of each word is calculated, which can accurately measure the importance of the word in the webpage; the average TF-IDF value of the text in each HTML tag is calculated, which can obtain the overall importance measurement of the text in the tag; finally, according to the preset threshold, the HTML tags and the corresponding text whose average TF-IDF value is lower than the preset value are deleted, to ensure that the remaining content has certain keyness and information amount. After word segmentation, deletion of stop words and special symbols, calculation of the TF-IDF value of each word and the average TF-IDF value of the words in each HTML tag, and deletion of the HTML tags and the text whose average TF-IDF value is lower than the threshold, the accuracy and efficiency of the malicious webpage classification are improved, the noise and irrelevant information interference are reduced, and a more reliable data basis is provided for subsequent text processing and classification. At the same time, by finely screening the HTML tags and the corresponding text content, the performance and interpretability of the model are improved, which is not possessed by the existing malicious website classification method and model.

[0037] In a further preferred solution, the step of combining the HTML tags with the corresponding text, performing tokenization processing on the text using a tokenization algorithm, and constructing a word table according to the tokenization result specifically comprises:

[0038] combining the corresponding HTML tags with the text;

[0039] performing tokenization processing on the text in a character cutting manner, and retaining the token and HTML tags in the text tokenization result;

[0040] constructing a word table, and recording the token and HTML tags in the tokenization result into the constructed word table, and assigning a unique index number to the corresponding token and HTML tags in the word table.

[0041] The effect of the above solution is that the present application combines the HTML tags with the corresponding text, and performs tokenization processing on the text using a tokenization algorithm, thereby constructing a word table containing word units and HTML tags, which can effectively handle the special nature of malicious web page text, and solve the problem of poor effect of open source algorithm tokenizer when processing such text. Especially by performing tokenization in a character cutting manner and retaining the token information and HTML tags in the tokenization result, the key words and label information in the malicious web page can be accurately captured. The constructed word table can obviously provide rich vocabulary and label information, which will provide a useful basis for subsequent processing, representation and classification tasks. Therefore, the present application can improve the accuracy and efficiency of malicious web page classification, so as to effectively identify and classify malicious web pages.

[0042] In a further preferred solution, the step of converting the tokenized web page text and HTML tags into a word vector matrix through word embedding, and using a Transformer encoder to encode the word vector for text classification specifically comprises:

[0043] convert the token and HTML tags recorded in the word table into a word vector;

[0044] construct a word vector matrix from the converted word vector according to the index number in the word table, and input the word vector matrix with added position information of the word into the Transformer encoder;

[0045] the Transformer encoder encodes the word vector in the word vector matrix for text classification.

[0046] The effect of the above scheme is that the position encoding information of the word is added to the word vector matrix in order to consider the position information of the word, the word vector is encoded and processed by the multi-head self-attention mechanism and the feedforward neural network of the Transformer encoder, the relationship and context information between the words can be effectively captured, the structure information and text information of the web page text and the HTML tag are fused together, and better representation results are obtained, finally, the classification of the web page is realized by classifying the encoded word vector. Therefore, the advantages of the word embedding and the Transformer encoder can be fully utilized, and the accuracy and effect of the text classification are improved.

[0047] In a further preferred scheme, the step of constructing a vocabulary and recording the token in the segmentation result and the HTML tag in the constructed vocabulary, and assigning a unique index number to the corresponding token and HTML tag in the vocabulary specifically includes:

[0048] An empty vocabulary is created;

[0049] The token in the segmentation result is stored in the constructed vocabulary one by one, a unique index number is assigned to the token stored in the vocabulary, and the corresponding HTML tag of the token is recorded.

[0050] The effect of the above scheme is that by storing the segmentation result one by one, it can be ensured that each word item is accurately added to the vocabulary, effectively avoiding the problems of omission or repetition; at the same time, a unique index number is assigned when adding the word item, and the corresponding label is recorded, which can ensure that each word item has a unique identifier and preserves the corresponding relationship between the word item and the label; the accuracy and consistency of constructing the vocabulary are improved, and a more reliable foundation is provided for the subsequent text classification task.

[0051] In a further preferred scheme, the specific calculation method of the TF-IDF value in the step of deleting the stop words and special symbols in the text after the segmentation processing, and calculating the TF-IDF value of each word remaining includes:

[0052] The term frequency of each word in each label is calculated by a TF value calculation formula, and the TF value calculation formula is: TF = the number of occurrences of the word in the label / the total number of words in the label;

[0053] The inverse document frequency of each word is calculated by an IDF value calculation formula, and the IDF value calculation formula is: IDF = the total number of documents in the document set / the number of documents containing the word;

[0054] The TF-IDF value of each word is calculated by a TF-IDF value calculation formula: TF-IDF = TF value * IDF value.

[0055] The effect of the above scheme is that when a word frequently appears in the current label, the TF value of the word is relatively high, indicating that the word has high discrimination for the current label, and thus calculating the TF value of the word can evaluate the importance of the word in the current label; if a word appears in a small number of documents in the document set, the IDF value of the word is high, indicating that the word is rare in the entire document set and has high discrimination; integrating the TF value and the IDF value can comprehensively consider the importance of the word in the label and the rarity degree of the word in the entire document set, for example, stop words and special symbols usually have high frequency of occurrence and are generally present in the entire document set, and thus the TF-IDF value of the word is low, and the stop words and special symbols in the text processed by word segmentation can be effectively screened and filtered by calculating the TF-IDF value, so as to reduce noise and redundant information and improve the accuracy and effect of the subsequent text classification task.

[0056] In a further preferred scheme, the calculation method of the average TF-IDF value in the step of calculating the average TF-IDF value of all words in each HTML label is that the TF-IDF values of all words in each HTML label are added and then divided by the total number of words in the label.

[0057] The effect of the above scheme is that by calculating the average TF-IDF value of all words in each HTML label, the importance and discrimination of the label in the entire document set can be understood, and a higher average TF-IDF value indicates that the words in the label have high importance and rarity degree, which can be a key feature of web page classification; thus, calculating the average TF-IDF value helps to extract and identify key words and features related to text classification, further improving the accuracy and effect of the text classification task.

[0058] A system for implementing a malicious web page classification model construction method includes a memory and a processor, the memory is used to store a malicious web page classification model construction program, and the processor is used to run the malicious web page classification model construction program to implement the malicious web page classification model construction method as described above.

[0059] A storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the malicious web page classification model construction method as described above. The storage medium includes all the technical features of the malicious web page classification model construction method described above, and thus also has all the technical effects of the malicious web page classification model construction method described above, which will not be repeated here.

[0060] Compared with the prior art, the HTML content of the to-be-identified webpage is obtained by the network crawler, and the category of the webpage is determined, thereby providing an accurate basis for subsequent classification; the average TF-IDF value of the text in the HTML tag is calculated through the TF-IDF algorithm, important text information can be screened out, and the interference of noise and irrelevant information is avoided; the HTML tag and the text are combined, and the text is processed by using the word segmentation algorithm, the text can be segmented into independent words or phrases, and a word table is constructed, thereby providing a basis for subsequent representation and processing; the webpage text and the HTML tag after word segmentation are converted into a word vector matrix through word embedding technology, the word vector is encoded by using a Transformer encoder, and the deep representation and understanding of the text are realized; finally, a malicious webpage classification model is constructed according to the text classification result, and the existing manually annotated webpage category data is used for training, thereby improving the accuracy of the malicious webpage classification model. Moreover, the malicious webpage classification model constructed by the present application does not need to be used in cooperation with other models, thereby reducing the required computing resources and reducing the complexity of model deployment. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is a flowchart of a malicious webpage classification model construction method provided by the present application. DETAILED DESCRIPTION

[0062] The present application provides a malicious webpage classification model construction method, system and storage medium. In order to make the purpose, technical scheme and effect of the present application more clear and definite, the present application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0063] The present application provides a malicious webpage classification model construction method, as shown in Figure 1 , which comprises:

[0064] S100, the HTML content of the to-be-identified webpage is obtained by the network crawler, and the category of the to-be-identified webpage is determined. The network crawler is an automatic program which accesses the webpage on the Internet by simulating the behavior of the browser, and obtains the HTML source code of the webpage. While the HTML is a markup language which is used to describe the structure and content of the webpage.

[0065] S100 obtains the HTML content of the to-be-identified webpage through the web crawler, so that the subsequent webpage analysis and classification can be based on the structure and content of the webpage. Compared with the prior art technical method which only targets the webpage content, more information and features are provided for the classification and identification of malicious webpages. At the same time, by determining the category of the webpage, the to-be-identified webpages can be better organized and managed, and accurate label information is provided for subsequent model training and classification tasks, so as to construct an accurate and reliable malicious webpage classification model and improve the accuracy and effect of webpage classification. S100 is an indispensable step for the method for constructing the malicious webpage classification model, and is closely related to other steps and is an important basic condition for the implementation of the whole technical solution.

[0066] Specifically, when crawling the webpage, the target webpage is first selected, and the scope can be selected as free data and Internet data, which can be one target webpage or multiple target webpages. After selecting the target webpage, the web crawler program needs to be set, and the initial link and other related parameters such as crawling depth and crawling speed are specified. Then, the web crawler will simulate the behavior of the browser according to the set parameters and rules, send HTTP requests, and receive HTML responses returned by the server. After crawling the HTML content, the web crawler will parse the crawled content to obtain the key information such as the webpage title, the text content, and the link. Finally, the crawled webpages are classified, which can be matched with the sample library, manually identified and classified, or both used together, to set the feature guide for the malicious webpage classification when the malicious webpage classification model is finally constructed, or to make the finally constructed malicious webpage classification model able to directly identify and classify the malicious webpages. It can be understood that since the finally constructed malicious webpage classification model will also be trained by using the existing webpage text data manually annotated with webpage categories, the trained model can basically identify and classify most malicious webpages. Therefore, the malicious webpage classification model constructed by the present application can be used to identify malicious webpages and classify the types of malicious webpages.

[0067] In specific implementation, S100 specifically includes (for each step, examples will be used for illustration, and due to content sensitivity, the content of the malicious webpage is not listed as an example):

[0068] The HTML content of the to-be-identified webpage is crawled by a web crawler, and the HTML content includes a page layout and page content. If the to-be-identified webpage is a news webpage with a link of https: / / www.example-news.com / article123, the HTML content of the to-be-identified webpage crawled by the web crawler can include the overall layout of the webpage (such as a header, a footer, and a navigation bar, and these parts are usually marked by specific HTML tags, for example <header> 、 <footer>and <nav>), title (for describing the subject of the reported news, the tag is usually the title tag <h1>、< / h1> <h2>), abstract (a tag for summarizing the gist of news content, a tag is usually a paragraph tag ), the text content (the label usually has a paragraph label , the segmentation label , in addition the text content usually also contains line breaks or spaces, pictures (the tag usually is , in addition the picture usually also includes the src attribute, and hyperlinks (the form of the source or related article conference hyperlinks appears in the news, the tag usually is In addition, it usually also includes the href attribute); In addition, it can also include other elements, such as news author, publication date and label classification, etc., and these contents are also marked using different HTML tags.

[0069] The extracted HTML content is structured, the HTML tags are extracted, the text content in the webpage picture is recognized by the OCR technology, and the recognized text content and the directly extracted text content are respectively associated with the extracted HTML tags; by parsing the HTML document, each element of the webpage can be recognized, such as title, body, author and publication date, etc.; and if the picture is included, the OCR algorithm can be used to convert the text in the picture into recognizable text content. Since various different elements are usually identified using different HTML tags, the extracted text content can be directly associated with the HTML tags, such as the news title associated with the <h1>Tag, which associates the news body to The above-mentioned application places of each type of label are not listed one by one here.

[0070] According to the HTML label and the text content, the web page category is matched in the sample library, and if the matching is unsuccessful, manual marking of the web page category is performed. Using the existing sample library or classification standard, the extracted HTML label and text content are matched, and the news web page is attempted to be classified into a known news category, such as politics, sports, entertainment, etc. Specifically, the rule matching or machine learning algorithm can be used to realize the classification; but if the matching cannot be performed, manual marking is required, for example, if the web page is a news about environmental protection, but cannot be matched with the existing category, the web page can be manually marked as the "environment" category.

[0071] S200, the average TF-IDF value of the text in each HTML label in the obtained HTML content is calculated by using the TF-IDF algorithm, and the HTML label and the text corresponding to the HTML label whose average TF-IDF value is lower than a preset value are deleted. TF (Term Frequency) is the term frequency of each word in each label, and IDF (Inverse Document Frequency) is the inverse document frequency of each word. In this step, the text in each HTML label is processed by using a word segmentation tool, and the TF-IDF value of each word is calculated. Then, the average TF-IDF value of all words in each HTML label is calculated. If the average TF-IDF value is lower than the preset value, it indicates that the importance of the text in the label relative to other labels is low, and therefore the label and the corresponding text can be deleted. The text in the HTML content is selected by using the TF-IDF algorithm, the text information with high importance and rarity is retained, the text information with small influence on the classification task is deleted, the accuracy and effect of the subsequent text classification task are improved, the key text content is focused, and the interference of irrelevant information is reduced.

[0072] In a specific implementation, the S200 specifically includes:

[0073] S210, the text corresponding to each HTML label is processed by using a word segmentation tool; still taking the above-mentioned news web page as an example, it is assumed that the following HTML label and corresponding text content are obtained:

[0074] <title>Label: the news title is "Tech company releases latest smart phone".< / title>

[0075] Tag: news body paragraph 1 is "The phone uses advanced artificial intelligence technology, with powerful processing capabilities and high-definition screen."

[0076] Tag: News body paragraph 2 is "This smartphone will change the way users live, providing more convenience and innovative features."

[0077] Using a tokenizer to tokenize the text corresponding to each HTML tag, the following results are obtained:

[0078] <title>Label segmentation result: "Tech company releases latest smart phone";< / title>

[0079] First Tag word segmentation result: "The mobile phone adopts advanced artificial intelligence technology, has strong processing power and high-definition screen";

[0080] Second The tokenized result of the label is: "This smart phone will change the user's lifestyle, provide more convenience and innovative functions."

[0081] S220, delete the stop words and special symbols in the text after tokenization processing, and calculate the TF-IDF value of each word left;

[0082] Assuming we use a common stop word list, the filtered results are as follows:

[0083] <title>Label segmentation result: "Tech company releases latest smart phone";< / title>

[0084] The first Tag word segmentation result: "Mobile phone uses advanced artificial intelligence technology, has strong processing power and high-definition screen";

[0085] Second The tokenized result of the label is: "This smart phone changes the user's lifestyle and provides convenient and innovative functions."

[0086] S230, calculate the average TF-IDF value of all words in each HTML tag;

[0087] Specifically, the specific calculation method of TF-IDF value in S230 includes:

[0088] Calculate the term frequency of each word in each label by the TF value calculation formula: TF = the number of occurrences of the word in the label / the total number of words in the label;

[0089] Calculate the inverse document frequency of each word by the IDF value calculation formula: IDF = the total number of documents in the document set / the number of documents containing the word;

[0090] Calculate the TF-IDF value of each word by the TF-IDF value calculation formula: TF-IDF = TF value * IDF value;

[0091] Suppose we have a document set containing multiple documents, and the TF and IDF values of each word have been calculated as follows:

[0092] The TF value of the word "technology" is 1 / 6, and the IDF value is 2.5;

[0093] The TF value of the word "mobile phone" is 2 / 11, and the IDF value is 1.8;

[0094] The TF value of the word "smart" is 1 / 6, and the IDF value is 2.3;

[0095] The TF and IDF values of other words are calculated in the same way;

[0096] By multiplying the TF value and the IDF value of each word, we get the TF-IDF value of each word:

[0097] <title>The TF-IDF value of the word in the label is (1 / 6)*2.5=0.4167;< / title>

[0098] First The TF-IDF value of the word in the label is (2 / 11)*1.8=0.3273;

[0099] Second The TF-IDF value of the word in the tag is (1 / 6)*2.3=0.3833;

[0100] The TF-IDF values of other words are calculated in the same way, and thus the TF-IDF value of each word can be obtained.

[0101] In addition, the average TF-IDF value of the S230 is calculated by adding the TF-IDF values of all words in each HTML tag and then dividing the total number of words in the tag.

[0102] The average TF-IDF value of each word in the HTML tag calculated in the above example is:

[0103] <title>The average TF-IDF value of the word in the label is 0.4167;< / title>

[0104] The first The average TF-IDF value of the words in the label is 0.3273;

[0105] Second The average TF-IDF value of the words in the label is 0.3833.

[0106] S240, according to the calculation result, deleting HTML tags and the text corresponding to the HTML tags below the preset value.

[0107] Assuming that the preset value is 0.35, the calculation result is judged as follows:

[0108] <title>The average TF-IDF value of the label is 0.4167, which is higher than the preset value, and is retained;< / title>

[0109] The first The average TF-IDF value of the labels is 0.3273, which is lower than the preset value, and is deleted.

[0110] Second The average TF-IDF value of the label is 0.3833, which is higher than the preset value, and is reserved.

[0111] S300, combine the HTML tag with the text corresponding thereto, use a word segmentation algorithm to perform word segmentation processing on the text, and construct a word table according to the word segmentation result. The word segmentation algorithm is a process of cutting a continuous text sequence into meaningful words. Common word segmentation algorithms include rule-based methods and statistical-based methods, such as jieba, NLTK, etc., and the specific word segmentation algorithm used is not specifically limited in the present application; the word table is a commonly used tool in text analysis, used to represent the features and vocabulary of the text, and in the present application specifically refers to a list containing all the words after the text is processed by word segmentation. This step effectively associates the HTML tag with the text and performs word segmentation processing on the text, and a series of meaningful words can be obtained; and constructing a word table is helpful for subsequent feature extraction and text analysis tasks, such as text classification, information retrieval, and text generation, etc. By constructing a word table, we can better understand the content of the text, extract key information, and provide a basis for subsequent text processing and analysis tasks, so this step is also an important step of the technical solution of the present application.

[0112] In specific implementation, the S300 specifically includes:

[0113] Combine the corresponding HTML tag with the text; assume that we have a The label and its corresponding text content are "This is a news content", which are combined as This is a news content ”.

[0114] The text is segmented by cutting by character, and the token and HTML tag in the text segmentation result are retained; the combined text is segmented by using a segmenter; for example, for " This is a news content ”, the segmenter is used for cutting, and the segmentation result is [" ", "This is", "a", "news", "content", ”].

[0115] In the segmentation result, each word is retained, and the corresponding HTML tag is also retained, so the retained segmentation result is [(" ", "tag")", ("This is", "text"), ("a", "text"), ("news", "text"), ("content", "text"), ( ”, "tag”)].

[0116] A word table is constructed, and the token and HTML tag in the segmentation result are recorded in the constructed word table, and a unique index number is assigned to the corresponding token and HTML tag in the word table.

[0117] After constructing the blank word table, the words and tags in the retained segmentation result are recorded in the word table, and a unique index number is assigned to each word and tag; for example, the record in the word table can be: {" ": 0, "This is": 1, "a": 2, "news": 3, "content": 4, ”: 5}. In this way, the word table contains both the words in the text segmentation result and the corresponding HTML tags, and a unique index number is assigned to them for subsequent processing and analysis.

[0118] S400, convert the segmented webpage text and HTML tags into a word vector matrix through word embedding, and use a Transformer encoder to encode the word vectors for text classification. Word embedding: a technique that represents words as vectors in a continuous vector space, used to capture semantic and contextual relationships between words; in specific implementation, a pre-trained word embedding model (such as Word2Vec, GloVe or BERT) can be used to convert the segmented webpage text and HTML tags into word vectors; through word embedding, each word and tag is represented as a fixed-length vector, where the dimension of the vector is determined according to the selection and setting of the pre-trained model. Word vector matrix: refers to the construction of a word vector matrix according to the index number in the vocabulary, the number of rows of the word vector matrix is the same as the number of words in the vocabulary, and the number of columns is the same as the dimension of the word vector; the word vector matrix provides vector representation corresponding to each word and tag, which is used to input the model for further processing and analysis. Transformer encoder: a neural network model based on self-attention mechanism, widely used in natural language processing tasks such as text classification, machine translation, etc.; in this step, the invention uses a Transformer encoder to encode the word vector matrix to capture the relationship and context information between words and tags; specifically, the Transformer encoder can encode and model the word vector through multiple layers of self-attention and feedforward neural network layers, extract features and generate text representation.

[0119] The invention successfully converts the segmented webpage text and HTML tags into a word vector matrix through S400, and encodes the word vectors using a Transformer encoder to obtain a high-dimensional representation of the text, which contains semantic and contextual information of words and tags, can better represent webpage content, extract key features, and provide meaningful input for subsequent text classification tasks; by using the powerful modeling capability of the Transformer model, the invention can effectively classify webpage text, improving the accuracy and effectiveness of classification; it is not difficult to understand that S400 is also an essential step for the invention to build an accurate model.

[0120] In specific implementation, S400 specifically includes:

[0121] Convert both tokens and HTML tags recorded in the vocabulary into word vectors; assuming that Word2Vec model is used for word embedding, each token and HTML tag is converted into a 300-dimensional word vector, then the subsequent steps for processing the word vectors will be based on this;

[0122] The converted word vectors are constructed into a word vector matrix according to the index numbers in the vocabulary, and the constructed word vector matrix is input into the Transformer encoder after adding the position information of the words; assuming that there are 1000 tokens and 20 HTML tags in the vocabulary, the size of the word vector matrix is (1020, 300); in the word vector matrix, each row represents the word vector of a token or HTML tag; in each word vector in the word vector matrix, the position information of the word is also added, for example, the position information of the word is integrated into the word vector using the position encoding method; then the Transformer encoder can be used to encode the word vectors in the word vector matrix to obtain the representation of the text;

[0123] The Transformer encoder encodes the word vectors in the word vector matrix to perform text classification; specifically, the Transformer encoder is composed of multiple encoding layers (Encoder Layer), and each encoding layer contains a self-attention mechanism and a feedforward neural network; in each encoding layer, the word vectors in the word vector matrix are input into the self-attention mechanism for context modeling, and then passed through the feedforward neural network for feature extraction and conversion; after processing by multiple encoding layers, the word vectors in the word vector matrix are gradually encoded into text representations with rich semantic information.

[0124] S500, according to the text classification result, a malicious webpage classification model is constructed, and the constructed model is trained and updated by using the existing webpage text data manually labeled with webpage categories.

[0125] In specific implementation, when constructing the model, an appropriate machine learning or deep learning model can be selected to construct the malicious webpage classification model, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), or a Transformer model; before model training, a sufficient number of malicious webpage text data are collected as training data, which contain text content and corresponding malicious labels; then the prepared malicious webpage text data set can be used to train the constructed model, specifically, the feature vector is used as input and the corresponding malicious label is used as target output, and the weights and parameters of the model are optimized through the back propagation algorithm; thereafter, the trained model can be evaluated using an independent evaluation data set, and the classification accuracy, recall rate, precision and other indicators are calculated to evaluate the performance of the model; finally, according to the evaluation result, the model is optimized, the hyperparameters, network structure of the model are adjusted, or the regularization technique is used to improve the generalization ability and performance of the model.

[0126] The application further provides a system for implementing the malicious webpage classification model construction method, which comprises a memory and a processor, the memory is used for storing a model processing program for data analysis, and the processor is used for running the model processing program for data analysis to implement the malicious webpage classification model construction method.

[0127] The application further provides a storage medium having a computer program stored thereon, the computer program is executed by a processor to implement the steps of the malicious webpage classification model construction method.

[0128] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the computer program can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments of the application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (SyNchliNk) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0129] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems can be used with these teachings, depending on the implementation. Those skilled in the art will recognize that structures embodied by these descriptions can be synthesized from the description above and alternative structures embodying the application can be synthesized from application of the teachings. In addition, the present embodiments are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the application, and any references below to specific languages are provided for disclosure of enablement of the best mode of the application.

[0130] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order not to obscure the understanding of this description.

[0131] Similarly, it is to be understood that the embodiments of the application can be used in the exact form disclosed herein, or with minor modifications, and the present disclosure is not limited to the exact form disclosed herein. It is also to be understood that the embodiments of the application can be used in any combination of the disclosed embodiments.

[0132] Those skilled in the art will appreciate that the modules in the apparatuses in the embodiments can be adapted and placed in one or more apparatuses other than the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and further can be divided into multiple sub-modules or sub-units or sub-components. Any combination of all the features disclosed in the specification (including the accompanying claims, abstract and drawings), and any method or apparatus so disclosed, can be taken in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless explicitly stated otherwise, each feature disclosed in the specification (including the accompanying claims, abstract and drawings) can be replaced by alternative features that serve the same, equivalent or similar purpose.

[0133] Further, those skilled in the art will appreciate that, while certain embodiments described herein include some, but not other, features of the embodiments described in other embodiments, combinations of the features of the different embodiments are to be expected and are within the scope of the application and form different embodiments. For example, any of the claimed embodiments can be used in any combination.

[0134] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps not listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unitary claim, several devices, apparatuses or means can be listed, comprising means for carrying out a certain task. The use of the term'means' in a claim is intended to refer to a combination of devices, apparatuses or means for carrying out a task. The word 'first','second', 'third', etc. do not imply any order. The use of these terms is to be interpreted as names. Steps in the above-described embodiments, other than those in the claims, are not to be interpreted as limiting the order of the steps.< / h1> < / h2> < / nav> < / footer> < / header>

Claims

1. A method for constructing a malicious webpage classification model, characterized in that, The application relates to a method for identifying malicious webpages. HTML content of a webpage to be identified is crawled by a network crawler, and the category of the webpage to be identified is determined; TF-IDF algorithm is used to calculate the average TF-IDF value of text in each HTML tag in the obtained HTML content, and HTML tags with an average TF-IDF value lower than a preset value and the corresponding text are deleted; HTML tags and the corresponding text are combined, a word segmentation algorithm is used to perform word segmentation processing on the text, and a word table is constructed according to the word segmentation result; word embedding is used to convert the webpage text and the HTML tags after word segmentation into a word vector matrix, and a Transformer encoder is used to encode the word vector for text classification; a malicious webpage classification model is constructed according to the text classification result, and the constructed model is trained and updated by using webpage text data with manually labeled webpage categories; the step of combining HTML tags and the corresponding text, using a word segmentation algorithm to perform word segmentation processing on the text, and constructing a word table according to the word segmentation result specifically includes: combining the corresponding HTML tags and the text; performing word segmentation processing on the text in a character cutting manner and retaining tokens and HTML tags in the text word segmentation result; using a word segmenter to perform word segmentation processing on the combined text; retaining each word in the word segmentation result while retaining the corresponding HTML tags; constructing a word table and recording tokens and HTML tags in the word segmentation result into the constructed word table, and assigning unique index numbers to the corresponding tokens and HTML tags in the word table; the step of using word embedding to convert the webpage text and the HTML tags after word segmentation into a word vector matrix and using a Transformer encoder to encode the word vector for text classification specifically includes: converting tokens and HTML tags recorded in the word table into word vectors; constructing a word vector matrix from the converted word vectors according to the index numbers in the word table, and inputting the constructed word vector matrix into the Transformer encoder after adding position information of the words in the word vector matrix; the Transformer encoder encodes the word vectors in the word vector matrix for text classification. 2.The method of claim 1, wherein, the step of crawling HTML content of a webpage to be identified by a network crawler and determining the category of the webpage to be identified specifically includes: HTML content of a webpage to be identified is crawled by a network crawler, and the category of the webpage to be identified is determined; the HTML content includes page layout and page content; the obtained HTML content is structured, HTML tags are extracted, text content in webpage pictures is recognized by an OCR technology, and the recognized text content and directly extracted text content are associated with the extracted HTML tags; the HTML tags and the text content are matched with webpage categories in a sample library, and if the matching is unsuccessful, manual marking of webpage categories is performed. 3.The method of claim 2, wherein, The step of calculating the average TF-IDF value of the text in each HTML tag in the obtained HTML content by the TF-IDF algorithm and deleting the HTML tag and the corresponding text with the average TF-IDF value lower than the preset value specifically comprises: performing word segmentation on the text corresponding to each HTML tag by using a word segmenter; deleting the stop words and special symbols in the text after word segmentation and calculating the TF-IDF value of each word remaining; calculating the average TF-IDF value of all words in each HTML tag; according to the calculation result, deleting the HTML tag and the corresponding text with the average TF-IDF value lower than the preset value. 4.The method of claim 1, wherein, The step of constructing a word table and recording the token and HTML tag in the word segmentation result in the constructed word table and assigning a unique index number to the token and HTML tag in the word table specifically comprises: creating an empty word table; storing the token in the word segmentation result in the constructed word table one by one, assigning a unique index number to the token stored in the word table, and recording the HTML tag corresponding to the token. 5.The method of claim 3, wherein, The specific calculation method of the TF-IDF value in the step of deleting the stop words and special symbols in the text after word segmentation and calculating the TF-IDF value of each word remaining comprises: calculating the term frequency of each word in each tag by a TF value calculation formula, the TF value calculation formula being: TF = the number of occurrences of the word in the tag / the total number of words in the tag; calculating the inverse document frequency of each word by an IDF value calculation formula, the IDF value calculation formula being: IDF = the total number of documents in the document set / the number of documents containing the word; calculating the TF-IDF value of each word by a TF-IDF value calculation formula, the TF-IDF value calculation formula being: TF-IDF = TF value*IDF value. 6.The method of claim 1, wherein, The calculation method of the average TF-IDF value in the step of calculating the average TF-IDF value of all words in each HTML tag is: adding the TF-IDF values of all words in each HTML tag and then dividing by the total number of words in the tag. 7.A system for implementing a method for constructing a malicious webpage classification model, characterized in that, The computer program is executed by the processor to realize the steps of the malicious webpage classification model construction method according to any one of claims 1 to 6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the malicious webpage classification model construction method according to any one of claims 1 to 6.

Images