Website classification method and system based on multi-source fusion features

Through the website classification method with multi-source fusion characteristics, combined with URL, semantics and behavioral characteristics, and using deep learning technology, the problem of disguised website recognition in the existing technology is solved, and the classification accuracy and recall rate is achieved, which is suitable for Internet content. Management and personalized services.

CN119004169APending Publication Date: 2024-11-22SHANGHAI CRIMINAL SCI TECH RES INST +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411021244.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Existing website automation identification and classification technology is difficult to identify fake websites, especially for domain names that do not contain significant category characteristics, and the proportion of misidentification is relatively high.

Method used

Using a multi-source fusion feature website classification method, through the comprehensive application of URL features, static resource semantic features and dynamic network behavior features, combined with deep learning technology, a website classification recognition model is built, including website crawling module, URL feature extraction module, and semantics Feature extraction module and behavior feature extraction module, and fusion classification module classify websites.

Benefits of technology

It significantly improves the accuracy and recall rate of website identification classification, can more accurately identify unpopular websites and disguised websites, reduces the rate of misidentification, and is suitable for Internet content management and personalized service push and other fields.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119004169A_ABST
    Figure CN119004169A_ABST
Patent Text Reader

Abstract

According to the website classification method and system based on the multi-source fusion features, website features of websites are extracted through a website internal and external URL link weighted association method, semantic features of the websites are extracted through a mixed expert model website static resource analysis method, and behavior features of the websites are extracted through a website behavior analysis method based on code analysis; and carrying out website classification marking by using a deep learning technology. According to the method, the website classification and recognition model is constructed by comprehensively applying the website features, the static resource semantic features and the dynamic network behavior features of the website, the accuracy and recall rate of website recognition and classification are improved, technical support is provided for the fields of internet content management, personalized service pushing, accurate information retrieval and the like, and the method is suitable for popularization and application. And the healthy and ordered development of the network environment is promoted.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a technology in the field of information security, specifically a website classification method and system based on multi-source fusion features. Background Art

[0002] Web page classification is an important research topic in Internet space governance, and plays a key role in information search, recommendation systems, knowledge discovery, and network security management. At present, the automatic website recognition and classification technology usually relies on keyword matching of website text, images and other UI interface content. The defects of this type of technology are that it is difficult to apply to websites with complex structures, and the misrecognition rate is high. Summary of the invention

[0003] In view of the defects of the prior art that the inability to effectively identify disguised websites and the inability to classify domain names that do not contain significant category feature texts, the present invention proposes a website classification method and system based on multi-source fusion features. By comprehensively using the website's URL features, static resource semantic features and dynamic network behavior features, a website classification and recognition model is constructed to improve the accuracy and recall rate of website recognition and classification, provide technical support for the fields of Internet content management, personalized service push and accurate information retrieval, and promote the healthy and orderly development of the network environment.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a website classification method based on multi-source fusion features, which respectively extracts the website URL features of the website through a weighted association method of internal and external URL links of the website, extracts the website semantic features through a hybrid expert model website static resource analysis method, and extracts the website behavior features through a website behavior analysis method based on code analysis, and uses deep learning technology to perform website classification and labeling.

[0006] The present invention relates to a system for implementing the above method, comprising: a website crawling module for acquiring all static resources of a website to be identified, a website feature extraction module, a semantic feature extraction module, a behavior feature extraction module and a fusion classification module, wherein: the website crawling module pre-processes static resources, the website feature extraction module extracts features of all website addresses in the website, the semantic feature extraction module performs unified semantic understanding and feature extraction on collected text, images, videos, audio and other data, the behavior feature extraction module identifies dynamic behaviors such as request redirection and data packet hiding in JavaScript codes of the website, extracts behavior pattern features such as the way users interact with the website and the website background response, and the fusion classification module fuses and classifies the website's website features, semantic features and behavior features to determine the website type. Technical Effects

[0007] In the domain name feature extraction stage, the present invention weights and splices the main domain name link, subdomain name link and external link of the website respectively; in the static semantic feature extraction stage, through the fusion of the embedding output of the multimodal pre-trained model, the pre-trained model encoding network of multiple modalities extracts features of text, images and other resources in the website, and aligns the features of different modalities through the feature fusion layer; in the dynamic behavior feature extraction stage, through the website source code feature extraction, the interaction mode of the web page, network behavior and other factors are comprehensively analyzed, and the behavior characteristics of the website are included in the classification basis. Compared with the existing technology, the present invention can realize accurate website classification and solve the problem that the conventional method is insufficient in the identification and classification ability of unpopular websites and disguised websites. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a flow chart of the present invention;

[0009] Figure 2 It is a schematic diagram of the embodiment system;

[0010] Figure 3 This is a schematic diagram of the URL feature extraction process;

[0011] Figure 4 This is a schematic diagram of the semantic feature extraction process;

[0012] Figure 5 Schematic diagram of the behavior feature extraction process. DETAILED DESCRIPTION

[0013] like Figure 1 As shown, this embodiment relates to a website classification method based on multi-source fusion features, including:

[0014] Step 1) Figure 3 As shown, the URL features including the domain name and the jump link in the page are extracted, including:

[0015] 1.1) Extract all internal links of the website through the network feature extraction layer, that is, the main domain name of the website and the subdomains and external links consistent with the root directory, that is, the relationship between the website to be classified and the external sites.

[0016] The network feature extraction layer is an LSTM structure neural network with one hidden layer, whose input dimension is 256, hidden layer dimension is 128, output dimension is 32, and parameter weights of the LSTM structure neural network are obtained in advance through training.

[0017] 1.2) Perform weighted concatenation on the extracted internal links and external links to obtain URL features, including:

[0018] ① Weighted feature F of the primary domain name d d =fd r d ;

[0019] ② Sort the subdomains by their importance to the website, take the weighted average of the 10 most important subdomains, and then the weighted feature of the subdomain set Z is Where: Z = {z1,z2,…,z n};

[0020] ③According to the external link w i Credibility wi Rank them, take the weighted average of the top 100 external links in terms of credibility, and then the weighted feature of the external link set W is Where: W = {w1,w2,…,w n};

[0021] ④ Concatenate the above three features head to tail to obtain the URL feature F with a dimension of 96 addr =[F d ,F Z ,F W ], where: the feature of the primary domain name d is f d , subdomain z i The characteristic is f zi , external link i The characteristic is f wi , are 32-dimensional vectors, and the main domain reputation r d , its value is the inverse of Baidu search engine weight index, the importance of the subdomain to the website t zi =Depth(f zi )*Freq(f zi ), subdomain f zi The appearance level and frequency of occurrence in the website are Depth(f zi ) and Freq(f zi ).

[0022] Step 2) Use a multimodal hybrid expert model to extract semantic features from the text, image, video, and audio data on the website:

[0023] The input data for extracting text semantic features includes but is not limited to website title, text, and meta tags;

[0024] The input data for extracting image semantic features includes but is not limited to website logo images and content images;

[0025] The input data for extracting video semantic features includes but is not limited to website video clips and key picture frames;

[0026] The input data for extracting audio semantic features includes but is not limited to website background music and audio sound effects.

[0027] like Figure 4 As shown, the multimodal hybrid expert model includes: 1 routing layer, 4 expert networks and 1 semantic fusion network, wherein: the routing layer generates a routing selection function for selecting different expert networks according to input data in different formats; the 4 expert networks convert text into semantic vectors v respectively. t =f T (text), convert the image into a semantic vector v i =f I (image), convert the video into a semantic vector v v =f V (video), convert the audio into a semantic vector v a =f A (audio); The semantic fusion network aligns the representation scales of different expert networks and outputs the semantic feature F lang =g(v t ,v i ,v v ,v a ).

[0028] The routing function is R = f txt r txt +f img r img +f txt r txt +f aud r aud , where: text, image, video, and audio expert models are f txt 、f img 、f vid 、f aud For text, image, video and audio input, their routing weights are r txt 、r img 、r vid 、r aud , when the input is text, r txt =1, otherwise 0.

[0029] The expert network includes the embedding parts of ChatGLM-6B, InternViT-6B, VideoLLaMA2-7b and Qwen-Audio open source models.

[0030] The semantic fusion network is a double-layer feedforward neural network, and its weights are obtained by prior training.

[0031] Step 3) Through the code analysis model, based on the input website JavaScript source code, the behavioral features including active execution and triggered execution are extracted:

[0032] The proactively executed behavior features are actions that are automatically executed after the website is loaded, such as scheduled tasks, browser cache modifications, pop-ups, etc.

[0033] The behavior characteristics of triggering execution refer to: actions triggered by certain conditions, such as scrolling to the bottom to load more content, hovering the mouse to pop up a menu, clicking to submit a form, etc.

[0034] The code analysis model includes: an interactive mode analysis network for analyzing interactive modes such as event monitoring and DOM operations of web pages and their impact on page status, a response analysis network for analyzing the backend server's response to requests such as front-end API calls and asynchronous communications and their impact on page status, and a timed task analysis network for analyzing timed tasks such as timed refresh and polling in web pages.

[0035] The interaction mode I=f int (JSCode), where: JSCode includes code snippets related to user operation event listeners in the website source code, int The interactive mode analysis network is a JavaScript code pre-training model obtained by fine-tuning the CodeT5+embedding open source model.

[0036] The task goal of the fine-tuning is to match the code similarity of the JavaScript interactive code snippets, and the loss function of the fine-tuning is L=max(0, ||f int (A)-f int (P)|| 2 -||f int (A)-f int (N)|| 2 +α), where A is the reference code snippet, P is the code snippet with the same function as A, N is the code snippet with different functions from A, and α is a hyperparameter.

[0037] The request response B = f res (JSCode), where: JSCode includes network request and callback function code snippets in the website source code, F res The response analysis network is a JavaScript code pre-training model obtained by fine-tuning the CodeT5+embedding open source model.

[0038] The task goal of the fine-tuning is to perform code similarity matching on JavaScript request response code snippets, and the loss function of the fine-tuning is L=max(0,||f res (A)-f res (P)|| 2 -||f res (A)-f res (N)|| 2 +α), where A is the reference code snippet, P is the code snippet with the same function as A, N is the code snippet with different functions from A, and α is a hyperparameter.

[0039] The timed task T=f tim (JSCode), where: JSCode includes the scheduled task code snippet in the website source code, f tim It is a network for timing task analysis, that is, a JavaScript code pre-training model obtained by fine-tuning the CodeT5+embedding open source model.

[0040] The task goal of the fine-tuning is to perform code similarity matching on JavaScript request response code snippets, and the loss function of the fine-tuning is L=max(0,||f res (A)-f tim (P)|| 2 -||f tim (A)-f tim (N)|| 2 +α), where A is the reference code snippet, P is the code snippet with the same function as A, N is the code snippet with different functions from A, and α is a hyperparameter.

[0041] Step 4) After integrating the URL features, semantic features, and behavioral features obtained in steps 1-3 and establishing a feature vector, a deep learning model is used to classify the websites.

[0042] The eigenvector is F flat =[F addr ,F lang ,F act ], where: Behavioral features, i.e., the concatenated vector F of the output features of the code analysis model act =[I,B,T],F addr is the URL feature, F lang is a semantic feature.

[0043] The deep learning model is a 2-layer fully connected neural network, which calculates the specific type of the website and its probability based on the input feature vector.

[0044] The training loss function of the deep learning model is: Where: y nc is the label of the nth training website sample belonging to the cth category, p nc is the probability that the model predicts that the nth training website sample belongs to the cth category, N is the number of samples in the training set, and C is the number of categories in the training set.

[0045] like Figure 2 As shown, the system for implementing the above method in this embodiment includes: a website crawling module for acquiring all static resources of the website to be identified, a URL feature extraction module, a semantic feature extraction module, a behavioral feature extraction module and a fusion classification module.

[0046] The website crawling module includes: a request unit, a parsing unit and a storage unit, wherein: the request unit sends an HTTP request to the target website to obtain web page content, uses the HTTP protocol to send a website access request, and receives all responses; the parsing unit parses the web page content obtained from the target website, and the parsed content includes the website page URL link, text image and web page code, and the storage unit stores the parsed data in a local file system, a database or other storage medium.

[0047] The URL feature extraction module includes: a URL relationship analysis unit, a URL feature extraction unit and a URL feature splicing unit, wherein: the URL relationship analysis unit divides each URL link into internal links and external links according to the domain name suffix, jump relationship and search engine crowd index of each link in the website, and performs weighted processing; the URL feature extraction unit extracts features from the weighted internal links and external URLs to obtain internal link features and external link features; the URL feature splicing unit splices the internal link features and external link features to obtain the final website URL features.

[0048] The semantic feature extraction module includes: a routing unit, an expert model unit and a feature fusion unit, wherein: the routing unit switches different data channels according to different types of website static resources to ensure that the input data type matches the subsequent expert model; the expert model unit supports the processing of four types of data: text, image, video, and audio to obtain corresponding features; the feature fusion unit aligns the four features of text, image, video, and audio to obtain the final website static semantic features.

[0049] The behavior feature extraction module includes: a code extraction unit and a code analysis unit, wherein: the code extraction unit organizes and extracts JavaScript user interaction code, network request code and scheduled task code fragments in the web page source code, and the code analysis unit sends the user interaction code, network request code and scheduled task code fragments to the code pre-training model, converts them into embedding vectors, and splices them to obtain the final website dynamic behavior features.

[0050] The fusion classification module includes a multi-classification unit, which memorizes and classifies the website URL features, static semantic features and dynamic behavior features to obtain the category of the website.

[0051] The present invention verifies the effectiveness of the method on public datasets and self-built datasets respectively.

[0052] The effectiveness of the method of the present invention is verified on the labeled open source dataset Website Classification released on the Kaggle website in 2021. The dataset includes 1408 websites and category labels. The websites are divided into 15 categories. This test performs transfer learning on the parameters of the two fully connected layers in step 4) of the present invention. The model weights involved in steps 1) to 3) remain unchanged. The final test results are shown in Table 1. The accuracy and recall rates are both over 97%.

[0053] Table 1

[0054] The self-built test data set used in the present invention is Internet sites collected from Internet channels, and its labels are manually confirmed and reviewed. It contains five categories of websites: "dating, gambling, online games, e-commerce, and pornography". The test results are shown in Table 2. The accuracy and recall rates of each category are both over 90%.

[0055] Table 2

[0056] In summary, compared with the prior art, the present invention innovates feature extraction methods from multiple angles such as URL links, web page content, and code behavior to capture the multi-dimensional characteristics of websites, thereby classifying them more accurately. The complementarity of multi-source features helps to identify and utilize the advantages of different information sources and improve the overall performance of the classification model. A single data source may be affected by noise, missing values, or bias, while multi-source fusion can verify and complement each other, reduce the probability of misclassification, enhance the stability and reliability of the model, and contribute to the development of Internet services such as personalized recommendations, content filtering, search engine optimization, network monitoring, and security management.

[0057] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A website classification method based on multi-source fusion features, characterized in that: The website's URL features are extracted through the weighted association method of internal and external URL links, the website's semantic features are extracted through the hybrid expert model website static resource analysis method, and the website's behavioral features are extracted through the website behavior analysis method based on code analysis. Deep learning technology is used to classify and mark the website.

2. The website classification method based on multi-source fusion features according to claim 1 is characterized in that: The weighted association method for URL links inside and outside a website extracts URL features including its own domain name and the jump links in the page, specifically including: 1.1) Extract all internal links of the website through the network feature extraction layer, that is, the main domain name of the website and the subdomains and external links consistent with the root directory, that is, the relationship between the website to be classified and the external sites; 1.2) Perform weighted concatenation on the extracted internal links and external links to obtain URL features, including: ① Weighted feature F of the primary domain name d d =f d r d ; ② Sort the subdomains by their importance to the website, take the weighted average of the 10 most important subdomains, and then the weighted feature of the subdomain set Z is Where: Z = {z1,z2,…,z n }; ③According to the external link w i Credibility wi Rank them, take the weighted average of the top 100 external links in terms of credibility, and then the weighted feature of the external link set W is Where: W = {w1,w2,…,w n }; ④ Concatenate the above three features head to tail to obtain the URL feature F with a dimension of 96 addr =[F d ,F Z ,F W ], where: the feature of the primary domain name d is f d , subdomain z i The characteristic is f zi , external link i The characteristic is f wi , are 32-dimensional vectors, and the main domain reputation r d , its value is the inverse of Baidu search engine weight index, the importance of the subdomain to the website t zi =Depth(f zi )*Freq(f zi ), subdomain f zi The appearance level and frequency of occurrence in the website are Depth(f zi ) and Freq(f zi ).

3. The website classification method based on multi-source fusion features according to claim 2 is characterized in that: The network feature extraction layer is an LSTM structure neural network with one hidden layer, whose input dimension is 256, hidden layer dimension is 128, output dimension is 32, and parameter weights of the LSTM structure neural network are obtained in advance through training.

4. The website classification method based on multi-source fusion features according to claim 1 is characterized in that: The hybrid expert model website static resource analysis method adopts a multimodal hybrid expert model to extract semantic features including text features, image features, video features and audio features from input data of different formats of the website.

5. The website classification method based on multi-source fusion features according to claim 4 is characterized in that: The multimodal hybrid expert model includes: a routing layer, four expert networks and a semantic fusion network, wherein: the routing layer generates a routing selection function for selecting different expert networks according to input data in different formats; the four expert networks convert text into semantic vectors v respectively. t =f T (text), convert the image into a semantic vector v i =f I (image), convert the video into a semantic vector v v =f V (video), convert the audio into a semantic vector v a =f A (audio); The semantic fusion network aligns the representation scales of different expert networks and outputs the semantic feature F lang =g(v t , v i , v v , v a ); The routing function is R = f txt r txt +f img r img +f vid r vid +f aud r aud , where: text, image, video, and audio expert models are f txt 、f img 、f vid 、f aud For text, image, video and audio input, their routing weights are r txt 、r img 、r vid 、r aud , when the input is text, r txt =1, otherwise 0; The expert network includes: the embedding part of ChatGLM-6B, InternViT-6B, VideoLLaMA2-7b and Qwen-Audio open source models; The semantic fusion network is a double-layer feedforward neural network, and its weights are obtained by prior training.

6. The website classification method based on multi-source fusion features according to claim 1 is characterized in that: The website behavior analysis method based on code analysis extracts behavior features including active execution and triggered execution according to the input website JavaScript source code through the code analysis model; The active behavior characteristics mentioned above refer to: actions that are automatically executed after the website is loaded; The behavior characteristics that trigger execution refer to actions triggered by certain conditions.

7. The website classification method based on multi-source fusion features according to claim 6 is characterized in that: The code analysis model includes: an interactive mode analysis network for analyzing interactive modes such as event monitoring and DOM operations of web pages and their impact on page status, a response analysis network for analyzing the backend server's response to requests such as front-end API calls and asynchronous communications and their impact on page status, and a timed task analysis network for analyzing timed tasks such as timed refresh and polling in web pages; The interaction mode I=f int (JSCode), where: JSCode includes code snippets related to user operation event listeners in the website source code, int The interactive mode analysis network is a JavaScript code pre-training model fine-tuned on the CodeT5+embedding open source model; The request response B = f res (JSCode), where: JSCode includes network request and callback function code snippets in the website source code, res The response analysis network is a JavaScript code pre-trained model fine-tuned on the CodeT5+embedding open source model; The timed task T=f tim (JSCode), where: JSCode includes the scheduled task code snippet in the website source code, f tim It is a network for timing task analysis, that is, a JavaScript code pre-training model obtained by fine-tuning the CodeT5+embedding open source model.

8. The website classification method based on multi-source fusion features according to claim 1 is characterized in that: The deep learning technology mentioned above refers to: after integrating URL features, semantic features and behavioral features and establishing feature vectors, using a deep learning model to classify websites; The eigenvector is F flat =[F addr ,F lang ,F act ], where: Behavioral features, i.e., the concatenated vector F of the output features of the code analysis model act =[I,B,T],F addr is the URL feature, F lang is a semantic feature.

9. The website classification method based on multi-source fusion features according to claim 8 is characterized in that: The deep learning model is a 2-layer fully connected neural network, which calculates the specific type of the website and its probability based on the input feature vector; The training loss function of the deep learning model is: Where: y nc is the label of the nth training website sample belonging to the cth category, p nc is the probability that the model predicts that the nth training website sample belongs to the cth category, N is the number of samples in the training set, and C is the number of categories in the training set.

10. A website classification system based on multi-source fusion features for implementing any of the methods described in claims 1-9, characterized in that: include: The website crawling module, URL feature extraction module, semantic feature extraction module, behavioral feature extraction module and fusion classification module are used to obtain all static resources of the website to be identified, wherein: the website crawling module pre-processes the static resources, the URL feature extraction module extracts the features of all URLs in the website, the semantic feature extraction module performs unified semantic understanding and feature extraction on the collected data, the behavioral feature extraction module identifies the dynamic behavior in the JavaScript code of the website and extracts the behavioral pattern features, and the fusion classification module fuses and classifies the URL features, semantic features and behavioral features of the website to determine the type of website.

Citation Information

Cited By

  • Telecommunication network fraud damage mode analysis method

    CN120950692A

  • Illegal website identification and clue attribution method and device, equipment, medium and product

    CN120974031A