A method for identifying key source code fragments of template websites

Abstract

This invention relates to the field of template website discovery technology and discloses a method for identifying key source code fragments of template websites. The method includes obtaining the website's page source code based on both PC and mobile UserAgents, and acquiring external link resources such as images and files. After obtaining complete webpage source code information, HTML, JS, CSS, and IMG fragments or files are obtained by constructing extraction rules for source code files and fragments. An indicator system for evaluating the relevance and universality of the source code is formed based on manual analysis, and a batch of positive and negative samples are manually labeled. A key source code fragment judgment model is formed based on machine learning model training. The model analyzes real-time source code fragment data and outputs the judgment results. This invention extracts key source code fragments of websites by constructing a website relevance and universality dimension indicator system and based on a machine learning model. Based on this recognition result, it can support the rapid matching and identification of template websites for a large number of websites and source codes without possessing a large number of samples, greatly improving recognition efficiency and reducing computing power consumption.

Description

Technical Field

[0001] This invention relates to the field of template website discovery technology, specifically a method for identifying key source code fragments of template websites. Background Technology

[0002] Currently, most methods for identifying key source code fragments of template websites are based on content similarity, source code similarity, and feature extraction and matching. While these methods can effectively identify template websites to some extent, they also have some drawbacks, such as high resource consumption and slow computation.

[0003] With the continuous development of technology, new technologies and methods are constantly emerging. For example, deep learning technology can be used for the identification and analysis of template websites. By training on a large amount of template website data, a more accurate and efficient model can be obtained. Furthermore, some methods based on semantic understanding and natural language processing can also be used for template website analysis. Through semantic understanding and analysis of website content, the characteristics and patterns of template websites can be identified more accurately. The methods are typically as follows:

[0004] 1. Code structure: Template-based website building may have a larger file structure because it needs to adapt to various functional requirements, while the code of custom website building will be more concise.

[0005] 2. Webpage source files: Simple webpages with complex code, or strange or useless images in the website hosting space, may indicate a template-based website.

[0006] 3. Compare similar templates: If you see two websites that are identical in structure except for the content, lacking independence and showing strong similarity, they may have used the same template.

[0007] 4. Similar content: Template websites often have the same or largely similar content. By analyzing the text content, it is possible to identify whether two websites are template websites.

[0008] For example, application number "CN112989341A"; "An Analysis Method, System, and Process for Fraudulent Websites," relates to the field of internet domain name technology, specifically, to an analysis method for fraudulent websites. This method includes obtaining and saving the IP address of a domain name, extracting the IP addresses of fraudulent domain names from a list of fraudulent domain names, querying domain names with the same or similar IP addresses as the fraudulent domain names from the saved IP addresses, generating an analysis task, and analyzing the website and website content corresponding to the domain name based on set judgment rules to determine whether it belongs to a fraudulent website.

[0009] This technology is mainly used to analyze fraudulent websites. It may not be applicable to the analysis of other types of websites. When faced with a large number of domains and complex network environments, there may be challenges in terms of efficiency and accuracy. (1) It consumes a lot of resources: When acquiring and analyzing a large number of domains and website information, it may consume a lot of computing and storage resources.

[0010] (2) Slow calculation process: The analysis and judgment based on the set judgment rules may result in a slow analysis process, which cannot meet the needs of quickly identifying template websites.

[0011] (3) Limited adaptability: It mainly analyzes fraudulent websites, and its adaptability to other types of template websites may not be strong enough, and it cannot accurately identify different types of template websites.

[0012] (4) Misjudgment may occur: In a complex network environment, there may be some errors in the judgment of websites, which may lead to misjudgment.

[0013] Therefore, to address the above issues, a method for identifying key source code fragments of template websites is needed. Summary of the Invention

[0014] The purpose of this invention is to provide a method for identifying key source code fragments of template websites. This invention collects the webpage source code of known sample websites, constructs a system of website relevance and generality indicators, and extracts key source code fragments based on a machine learning model. Based on this identification result, it can support the rapid matching and identification of template websites across a large number of websites and their source code without requiring a large number of samples, greatly improving identification efficiency and reducing computational power consumption.

[0015] This invention is implemented as follows:

[0016] This invention provides a method for identifying key source code fragments of template websites, which is specifically implemented according to the following steps:

[0017] S1: Obtain the website's page source code based on both PC and mobile UserAgents, and retrieve external resources such as images and files; after obtaining the complete webpage source code information, extract HTML, JS, CSS, and IMG fragments or files by constructing source code file and source code fragment extraction rules; specifically, obtaining the website's page source code involves directly sending a request to the target website to obtain the page's source code, and the obtained source code is the original structure of the website's open source code;

[0018] Alternatively, you can access the website through browser rendering to obtain the complete source code structure after the website is fully loaded. This source code structure includes images and data content information. After the website source code is fully loaded, you can parse the website source code to obtain the resource file information associated with the website.

[0019] The specific rules for constructing source code files and source code fragment extraction include: summarizing and deduplicating the collected website source code; extracting specific tags such as html, meta, style, script, link, and img by parsing the page code; extracting and storing the current code fragment from the built-in content tags of html, meta, style, some scripts, and img; obtaining the src and href file link attributes of the link, some scripts, and img tags that are external links, and storing the corresponding files; and formatting and storing the collected website links, file links, code fragments, and code type information to form raw data that supports subsequent key source code analysis.

[0020] S2: Based on the analysis of the expert knowledge base, an indicator system for evaluating the relevance and universality of source code is formed, and a batch of positive and negative samples are manually labeled;

[0021] S3: Based on machine learning models, a model for analyzing key source code fragments of websites is formed;

[0022] Model building includes the construction and analysis of indicator dimensions; the specific steps are as follows:

[0023] S 3.1 The feature dimensions are constructed based on two directions: the universality and relevance of the source code. An analysis sample library is built based on the raw data from key source code analysis. The universality dimension identifies source code and content commonly used on the Internet, including common third-party libraries such as Baidu Statistics and Google Statistics. The uniqueness dimension is used to identify source code and content unique to the current template website, usually in combination with its business keywords and the data request behavior characteristics of the website's backend.

[0024] S 3.2 The data used for generality analysis specifically includes CDN domains, third-party libraries / SDKs, and simple general code. The relevance indicator analysis includes the obtained source code, images, and text. The source code includes data requests and initialization functions from the same domain on the website, the images include logos, and the text includes business keywords.

[0025] S4: Analyze real-time source code fragment data using a model and output the assessment results.

[0026] The existing dimensions of the system are labeled, and after the labeling is completed, the labels are reviewed by humans to form positive and negative samples. After completion, the machine learning decision tree model is used to train and form an analysis model; as shown in equations (1)-(2).

[0027] Equation (1)

[0028] in, P(c) represents the uncertainty of dataset D. i ) is category c i The proportion in dataset D;

[0029] Equation (2)

[0030] Where R(T) is the error of decision tree T on the training data, |T| is the number of leaf nodes in the tree, and α is the regularization parameter used to balance model complexity and error.

[0031] The indicator dimensions are shown in Table 1;

[0032] Indicator Name type illustrate Condition type external_link_address domain name Is this an external link address? cdn_domain Domain Names (Whitelist) Is it a CDN domain? Negative term sdk_name_hit SDK (whitelist) Does it hit the SDK? Negative term generic_minimal_code Code (whitelist rules) Is it minimalist code? Negative term business_word_hit_status text Keyword Hit Rate has_network_communication_behavior code Same-domain data request Has_initial_code code Initialization function tab_ico_icon image Is it a logo?

[0033] Table 1 Indicator Dimension Table

[0034] Furthermore, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a main controller, implements the method described in any of the above descriptions.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. By collecting the source code of known sample websites and constructing a system of website relevance and universality indicators, key source code fragments are extracted based on a machine learning model. Based on this identification result, it can support the rapid matching and identification of template websites across a large number of websites and their source code without possessing a large sample size. This significantly improves identification efficiency and reduces computational power consumption. It provides a fast and efficient matching and monitoring method in the field of template website discovery, especially in the identification of black and gray market template websites, greatly reducing resource consumption during the analysis process. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is a flowchart of the method of the present invention;

[0039] Figure 2 This is a flowchart of the indicator dimension construction and analysis of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to describe selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] Please see Figure 1-2 This invention provides a method for identifying key source code fragments of template websites, which is specifically implemented according to the following steps:

[0042] Follow these steps:

[0043] S1: Obtain the website's page source code based on both PC and mobile UserAgents, and retrieve external resources such as images and files; after obtaining the complete webpage source code information, extract HTML, JS, CSS, and IMG fragments or files by constructing source code file and source code fragment extraction rules; specifically, obtaining the website's page source code involves directly sending a request to the target website to obtain the page's source code, and the obtained source code is the original structure of the website's open source code;

[0044] Alternatively, you can access the website through browser rendering to obtain the complete source code structure after the website is fully loaded. This source code structure includes images and data content information. After the website source code is fully loaded, you can parse the website source code to obtain the resource file information associated with the website.

[0045] The specific rules for constructing source code files and source code fragment extraction include: summarizing and deduplicating the collected website source code; extracting specific tags such as html, meta, style, script, link, and img by parsing the page code; extracting and storing the current code fragment from the built-in content tags of html, meta, style, some scripts, and img; obtaining the src and href file link attributes of the link, some scripts, and img tags that are external links, and storing the corresponding files; and formatting and storing the collected website links, file links, code fragments, and code type information to form raw data that supports subsequent key source code analysis.

[0046] S2: Based on the analysis of the expert knowledge base, an indicator system for evaluating the relevance and universality of source code is formed, and a batch of positive and negative samples are manually labeled;

[0047] S3: Based on machine learning models, a model for analyzing key source code fragments of websites is formed;

[0048] Model building includes the construction and analysis of indicator dimensions; the specific steps are as follows:

[0049] S 3.1 The feature dimensions are constructed based on two directions: the universality and relevance of the source code. An analysis sample library is built based on the raw data from key source code analysis. The universality dimension identifies source code and content commonly used on the Internet, including common third-party libraries such as Baidu Statistics and Google Statistics. The uniqueness dimension is used to identify source code and content unique to the current template website, usually in combination with its business keywords and the data request behavior characteristics of the website's backend.

[0050] S 3.2 The data used for generality analysis specifically includes CDN domains, third-party libraries / SDKs, and simple general code. The relevance index analysis includes the obtained source code, images, and text. The source code includes data requests and initialization functions from the same domain on the website, the images include logos, and the text includes business keywords.

[0051] S4: Analyze real-time source code fragment data using a model and output the assessment results.

[0052] The existing dimensions of the system are labeled, and after the labeling is completed, the labels are reviewed by humans to form positive and negative samples. After completion, the machine learning decision tree model is used to train and form an analysis model; as shown in equations (1)-(2).

[0053] Equation (1)

[0054] in, P(c) represents the uncertainty of dataset D. i ) is category c i The proportion in dataset D;

[0055] Equation (2)

[0056] Where R(T) is the error of decision tree T on the training data, |T| is the number of leaf nodes in the tree, and α is the regularization parameter used to balance model complexity and error.

[0057] The indicator dimensions are shown in Table 1;

[0058] Table 1 Indicator Dimension Table

[0059] Indicator Name type illustrate Condition type external_link_address domain name Is this an external link address? cdn_domain domain name Is it a CDN domain? Negative term sdk_name_hit SDK Does it hit the SDK? Negative term generic_minimal_code code Is it minimalist code? Negative term business_word_hit_status text Keyword Hit Rate has_network_communication_behavior code Same-domain data request Has_initial_code code Initialization function tab_ico_icon image Is it a logo?

[0060] In this embodiment, raw URL data is prepared and collected. A batch of website source code files matching the expected business type are prepared through proactive discovery on the Internet, or the website source code files can be obtained through collection. Related resource collection involves parsing the webpage structure to obtain the website's associated external link files, including images, external JavaScript files, and external CSS files, and calculating downloads based on HTTP requests.

[0061] In this embodiment, website source code fragments are extracted;

[0062] Based on XML structure analysis, the website's content tags and code snippets are obtained and stored, such as html, meta, style, some scripts, and img. For tags that are external links, such as link, some scripts, and img, the file link attributes such as src and href need to be obtained, and the corresponding file needs to be stored.

[0063] In this embodiment, a method for constructing general and unique indicators is used for analysis.

[0064] Filter items:

[0065] A: CDN Domain Resources: Construct a CDN domain list to identify whether the loading source of external link resource files is a CDN service.

[0066] B: Third-party library / SDK identification: Construct a list of SDK matching rules, including name, HASH, etc., and detect resources such as JS / CSS / IMG.

[0067] C: General minimal code recognition: Reset, redirect, and other code in JavaScript code, such as window.locality="" and window.baseUrl="".

[0068] Identification items:

[0069] In terms of source code, it can identify data requests and initialization functions within the same domain, as implemented below:

[0070] D: Same-domain data request: Analyze JS files and code snippets, identify network request behavior by calling network request methods, obtain the host accessing the network, and detect whether it is a same-domain host or domain.

[0071] E: Initialization function: In some template websites for specific businesses, there may be code that loads configurations, initializes, etc., which can be identified based on the form and parameters of the code;

[0072] Develop business keywords and analyze the website source code based on these keywords.

[0073] F: Business Keyword Analysis: Analyze the content of a batch of business websites to identify trending and related keywords, and compile them into business keywords, which will be used as matching items when analyzing target websites.

[0074] In terms of images, it can recognize website logos, icons, and other image resources.

[0075] G: LOGO: Based on the alt, class, id and other attributes of the source code, analyze the surrounding reference content to identify whether the image is a LOGO.

[0076] In this embodiment, indicator data is labeled.

[0077] Based on expert review, the nature of each piece of source code is identified and assessed, and it is determined whether it is a key source code segment; thus forming sample data for model training.

[0078] In this embodiment, model analysis is performed;

[0079] Manually labeled and verified positive and negative samples are input into a machine learning model for learning, forming a website key source code fragment recognition model.

[0080] In this embodiment, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a main controller, implements the method described in any of the above descriptions.

[0081] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations will be apparent to those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying key source code fragments of a template website, characterized in that: Follow these steps: S1: Obtain the website's page source code based on both PC and mobile UserAgents, and acquire image and file external link resources; after obtaining complete webpage source code information, extract HTML, JS, CSS, and IMG fragments or files by constructing source code file and source code fragment extraction rules; the specific rules for constructing source code file and source code fragment extraction rules include summarizing and deduplicating the collected website source code, extracting specific tags such as html, meta, style, script, link, and img by parsing the page code, extracting the current code fragment from the built-in content tags of html, meta, style, some script, and img for storage; for tags of the nature of link, some script, and img external links, obtain the src and href file link attributes, and obtain the corresponding file for storage; format and store the collected website links, file links, code fragments, and code type information to form raw data that supports subsequent key source code analysis; S2: Based on the analysis of the expert knowledge base, an indicator system for evaluating the relevance and universality of source code is formed, and a batch of positive and negative samples are manually labeled; S3: Train a machine learning model to form a key source code fragment analysis model for websites; model construction includes indicator dimension construction and analysis; specifically, follow these steps: S 3.1 Feature dimensions are constructed based on the universality and relevance of the source code, and an analysis sample library is built based on the raw data from the key source code analysis. S 3.2 The data used for generality analysis specifically includes CDN domains, third-party libraries / SDKs, and simple general code. The relevance indicator analysis includes the obtained source code, images, and text. The source code includes data requests and initialization functions from the same domain on the website, the images include logos, and the text includes business keywords. S4: Analyze the real-time source code fragment data through the model and output the judgment results. Label the existing dimensions of the system. After the labeling is completed, hand it over to the manual review and labeling to form positive and negative samples. After completion, use the machine learning decision tree model to train and form the analysis model; as shown in equation (1)-equation (2). Equation (1) in, P(c) represents the uncertainty of dataset D. i ) is category c i The proportion in dataset D; Equation (2) Where R(T) is the error of decision tree T on the training data, |T| is the number of leaf nodes in the tree, and α is the regularization parameter used to balance model complexity and error.

2. The method for identifying key source code fragments of a template website according to claim 1, characterized in that, In step S1, obtaining the source code of a website's page specifically involves sending a request directly to the target website to obtain the source code of that page. The obtained source code is the original structure of the website's open source code. Alternatively, you can access the website through browser rendering to obtain the complete source code structure after the website is fully loaded. This source code structure includes images and data content information. After the website source code is fully loaded, you can parse the website source code to obtain the resource file information associated with the website.

3. The method for identifying key source code fragments of a template website according to claim 1, characterized in that: In step S 3.1 Among them, the generality dimension identifies source code and content that are commonly used on the Internet, including common third-party libraries such as Baidu Statistics and Google Statistics.

4. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the main controller, it implements the method as described in any one of claims 1-3 above.

Images