Similar domain name recommendation method based on website framework characteristics

Abstract

The invention relates to the technical field of asset management, and provides a similar domain name recommendation method based on website framework characteristics, which comprises the following steps of: uniformly converting URL (Uniform Resource Locator) data into lower case, removing URLs containing messy codes and abnormal punctuations, and removing associated white list domain names; a standardized cleaning method is applied to the preprocessed URL data, feature extraction is carried out on the part, except for the domain name, in the URL according to seven dimensions to construct URL rules, a white list and meaningless rule associated data are removed according to the standardized URL data to construct a domain name rule base, rule importance score calculation is carried out on the URL rules in the domain name rule base, and the URL data in the URL rule base are subjected to rule importance score calculation; constructing a similar domain name recognition model, and recommending similar domain names according to the input domain names; according to the method, the automation method is realized, the TOP N rules associated with a large number of similar domain names are automatically output in the similar domain name recommendation process, the cost of manual collection and induction is reduced, and the recommendation model can better adapt to a continuously changing network environment.

Description

Technical Field

[0001] The present invention relates to the technical field of asset management. By extracting rules that can uniquely identify a type of URL, similar domain names common to various types of domain name assets are recommended. Specifically, it is a similar domain name recommendation method based on website framework features. Background Art

[0002] With the explosive growth of Internet information, the importance and urgency of IT asset mapping have become increasingly prominent. The number of domain name assets on the Internet is huge and scattered, and effective technical means are urgently needed to discover the correlation between these assets and build an asset library for efficient management and maintenance.

[0003] In existing technical research, similar domain name identification technology mainly focuses on malicious domain name detection. These technologies mostly start from the character features, network features (such as WHOIS information) and resolution features (such as the number of times resolved, the number of CNAMEs, etc.) of the domain name, and use machine learning, deep learning and other algorithms to identify similar domain names. Although the research from the perspective of the domain name itself is relatively complete, they often require a lot of annotation costs and rely on a certain amount of prior knowledge.

[0004] In addition, these technologies mainly focus on discovering malicious domain names, and there are relatively few studies on the similarity of normal domain name assets such as OA and cameras. More importantly, the feature dimensions extracted by the malicious domain name recognition model, such as character features, may not be universal when distinguishing normal domain name assets.

[0005] There is a certain correlation or similarity between the structure and content of websites of the same type. Their information architecture and content classification patterns usually have specific rules, which are specifically manifested in the URL as similar site directory structures, the use of the same action instructions, query parameters, or access to the same resource files.

[0006] In summary, the existing background technology still has the following deficiencies:

[0007] 1. The existing technology faces bottlenecks in efficiency and applicability in the process of URL standardization. With the rapid development of the Internet, the diversity and magnitude of URLs have increased dramatically, making the complexity and management difficulty of URL data continue to rise. Due to the lack of a unified representation method, it is difficult to effectively compress and classify URL data, which not only reduces the efficiency of data query, but also limits the scalability and flexibility of applications.

[0008] 2. Existing similar domain name identification technologies are not universal enough for all domain names. On the one hand, existing technologies mostly focus on identifying malicious domain names, but the research on universal identification methods for domain name similarity is not sufficient. On the other hand, how to define the similarity between domain names and from which angles to distinguish the differences in similarity are challenges facing current technologies. These difficulties lead to the insufficiency of existing technologies in responding to the needs of different types of domain name assets.

[0009] 3. Existing technologies have limitations in the asset feature collection stage. Currently, the features collected and summarized through FOFA or manual work are limited, and it is difficult to obtain all the associated features of similar domain names, which limits the scope of asset management and monitoring by managers. In addition, these methods cannot adapt to the rapid changes in the network environment, and it is difficult to achieve real-time feature updates and expansions, making it impossible to keep the asset feature library up to date, affecting the timeliness and effectiveness of asset management. Summary of the invention

[0010] In order to solve the above technical problems, the present invention provides a similar domain name recommendation method based on website framework features. By targeting large-scale website request link data, a unified and standardized URL cleaning method is formulated, and the unique characterization features of the URL - URL framework rules are extracted to form a domain name framework rule library. On this basis, a similar domain name recognition model is constructed, and a domain name seed is input to output similar domain names of the same type to solve the problems in the prior art.

[0011] The technical solution of the present invention is as follows:

[0012] A method for recommending similar domain names based on website framework features comprises the following steps:

[0013] S1, URL preprocessing, converting URL data to lowercase, removing URLs containing garbled characters and abnormal punctuation, and removing URL data associated with whitelisted domain names including but not limited to domain names of major companies such as Tencent, Baidu, and Byte;

[0014] S2, URL standardization, applies standardized cleaning methods to the preprocessed URL data, including character processing related methods, URL structure parsing methods, and data cleaning and verification methods, and then extracts features from the parts of the URL except the domain name according to 7 dimensions to construct URL rules;

[0015] S3, build a domain name rule library. For the standardized URL data, remove the whitelist and meaningless rule-related data to build a domain name rule library;

[0016] S4, rule importance score calculation, performing rule importance score calculation on the URL data in the domain name rule library, wherein the rule importance score calculation includes rule complexity importance score calculation and rule content importance score calculation;

[0017] S5, build a similar domain name recognition model based on the rule importance score calculation result, and recommend similar domain names based on the input domain name.

[0018] Preferably, in step S2, the seven dimensional features are extracted respectively:

[0019] Feature name 1: Path details; Feature description: The path part of the URL excluding the domain name and query parameters;

[0020] Feature name 2: Path depth; Feature description: Number of directory levels of the URL;

[0021] Feature name 3: Action command; Feature description: After the last / and before the first ?, the path parameter corresponding to the URL action;

[0022] Feature name 4: resource file suffix; Feature description: the suffix of the resource file accessed in the URL;

[0023] Feature name 5: file name keyword; Feature description: extract file names containing given keywords;

[0024] Feature name 6: File name length including suffix; Feature description: The overall file name length of the resource file accessed in the URL;

[0025] Feature name 7: Query parameter; Feature description: The query parameter string after removing the query parameter value.

[0026] Preferably, the specific steps in step S3 are as follows:

[0027] S31, remove rules with length less than or equal to 15, including 6 separators '|';

[0028] S32, remove the rule whose last path parameter is empty. The rule format is "|2|||||id="; the original URL is "domain+ / ?id=". This type of rule has no action command and file name.

[0029] S33, remove the rules that are associated with only one domain name in the standardized results;

[0030] S34, treat the rules with the number of associated domain names being 2 or less and 2000 or more as whitelist rules, and remove the data associated with the whitelist rules;

[0031] S35, based on the dictionary of indiscriminative rules, remove indiscriminative rules.

[0032] Preferably, in step S35, based on the dictionary of rules with no discrimination, the rules with no discrimination are eliminated as follows:

[0033] Rule type 1: contains suffix name;

[0034] Rules: .js, .woff, .woff2, .css, .favicon, .ico, .ttf

[0035] Rule type 2: Include file name;

[0036] Rules: logo.png, webresourse.axd, iconfont.eot.

[0037] Preferably, the specific method for calculating the rule complexity importance score in step S4 is:

[0038] The rule complexity importance score calculation is based on the complexity of the rule length. The longer the rule, the higher the complexity and the higher the importance score in the similarity calculation.

[0039] Since different types of domain name asset association rules are different, the complexity of a domain name is relative, so the complexity score needs to be adjusted according to the different types of domain name association rules;

[0040] Therefore, we score the rules according to the differences in complexity of similar domain names. When recommending similar domain names, we obtain the associated rule set R of the input domain name, standardize the rule lengths in R (maximum and minimum standardization), and regard the standardized value as the rule complexity score.

[0041] Rule complexity calculation formula:

[0042]

[0043] Among them, diff_score represents the complexity weight of the rule, rule_len represents the length of the rule, and rule_len max Indicates the maximum value of the rules in the rule set R, rule_len min Indicates the minimum length of a rule in the rule set R.

[0044] Preferably, the specific method for calculating the rule importance score in step S4 is:

[0045] The importance is determined based on the action commands and file types associated with the rule. When the URL associated with the rule represents key actions such as management, login, and registration, the importance score is higher; when common resource file types appear in the rule, the importance score is lower. The specific method is:

[0046] Assume that the initial value of the importance score of each rule content is 0.5;

[0047] Determine whether the file type associated with the rule belongs to a common file type. If so, set the score to 0.2 and proceed to the next step. If not, proceed directly to the next step.

[0048] Determine whether the action command represented by the rule contains the following keywords: index, login, admin, phone, sys, system, home, list, member, server, manage, user, password, get, record. If it contains the keywords, the score is set to 1;

[0049] For rules containing common file types, if they contain management and login keywords, the score is determined based on the keywords included, and the score is 1;

[0050]

[0051] Among them, rule means rule, and info_score represents the importance score of the rule content.

[0052] Preferably, the file type associated with the judgment rule belongs to common file types, and common file types include mp4, flv, m4s, m3u8, mp3, ts, image, jpeg, png, gif, jpg, zip, apk, txt, log, 7z, webp and heif.

[0053] Preferably, the specific process of constructing the similar domain name identification model in step S5 is as follows:

[0054] S51, obtain the bridge rule set: input a batch of domain name seeds of the same type D orig {orig_d i}, get the domain name association rule set R{rule j}, where d represents the domain name, i represents the domain name index, orig represents the specific domain name seed, D represents the domain name set, and D orig That is, it is represented as a set of domain name seeds, R represents the rule set, and j is the index of the rule;

[0055] S52, obtain the associated domain name set: based on the bridge rule R{rule j}, combined with the domain name rule library, query the domain name data similarity D associated with the rule simi {simi_d i}, retain domain name, rule and URL sample data;

[0056] Among them, simi is the abbreviation of similar, D simi This means that there are multiple domain name sets with similar rules to the original domain name;

[0057] S53, remove whitelist rule data: The whitelist rule is based on the general template rule accumulated from the previous similar domain name recommendation effect, and the whitelist rule in the result is filtered twice to increase the concentration of domain name asset-specific rules in the result;

[0058] S54, dedicated rule recommendation: based on the number of domain name assets associated with the rule, recommend the rules and URL examples of the top N associated domain names; TOP represents a sorted selection method, N is a variable, usually representing a positive integer, and TOPN means selecting the top N associated domain names after sorting a series of associated domain names in descending order according to the number of associated domain names;

[0059] S55, domain name similarity calculation integrating importance scores: based on the intersection rule, obtain the pairwise correspondence between domain names; let A be the input domain name, B∈D simi {simi_d i} and B≠A, the intersection rule set between A and B is R inner {rule1,rule2,...,rule k}, k is the total number of rules;

[0060]

[0061] Among them, similarity(A,B) represents the similarity between domain names A and B, w diff Indicates the complexity weight of the rule. The default value is 0.4. info Indicates the content importance weight of the rule, the default value is 0.6; diff_score i The complexity importance score of the rule, info_score i Score the content importance of the rule;

[0062] S56, deduplication of similar domain name results: To avoid duplicate recommendations, deduplication of similar domain names in the results is performed, and they are sorted in descending order based on domain name similarity, and only the input domain name and URL sample of the recommended result that are most similar to the output similar domain name are retained.

[0063] Preferably, the similar domain name recommendation method based on website framework features supports the following types of recommended similar domain names: OA domain names, camera domain names, code receiving platform domain names, airport website domain names, pornography-related domain names, gambling-related domain names and pirated video domain names.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] 1. The present invention proposes a universal URL standardization method, deeply analyzes the parts of the URL except the domain name, extracts key information elements from seven important dimensions, and describes the content characteristics of the URL suffix in detail, laying the foundation for subsequent similar domain name identification.

[0066] 2. The present invention proposes a domain name similarity calculation method that integrates rule complexity and content importance: when making domain name similarity recommendations based on intersection rules, the influence of rule length and rule content on similarity calculation is comprehensively considered, and the similarity between domain names is calculated in the form of weighted scores, thereby improving the interpretability of domain name similarity and the accuracy of similar domain name identification.

[0067] 3. The present invention realizes automatic feature clustering and expands new features of similar assets. In the process of similar domain name recommendation, the top N rules with a large number of associated similar domain names are automatically output, thereby reducing the cost of manual collection and induction.

[0068] In summary, the present invention provides an efficient and real-time similar domain name identification method, which is applicable to various types of domain name assets and has high versatility and practicality. It supports the expansion of domain names and associated feature libraries of similar assets in an automated manner, so that the recommendation model can better adapt to the ever-changing network environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a schematic flow chart of the steps of the method of the present invention;

[0070] Figure 2 It is a flow chart of URL preprocessing and standardization of the present invention;

[0071] Figure 3 It is a schematic diagram of the process of constructing the domain name rule library of the present invention;

[0072] Figure 4 It is a schematic diagram of the process of recommending similar domain names based on the similar domain name identification model of the present invention;

[0073] Figure 5 It is a schematic diagram of the process flow of recommending special rules and similar domain name results of the present invention. DETAILED DESCRIPTION

[0074] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0075] The present invention provides a method for recommending similar domain names based on website framework features, comprising the following steps:

[0076] S1, URL preprocessing, converting URL data to lowercase, removing URLs containing garbled characters and abnormal punctuation, and removing URL data associated with whitelisted domain names including but not limited to domain names of major companies such as Tencent, Baidu, and Byte;

[0077] S2, URL standardization, applies standardized cleaning methods to the preprocessed URL data, including character processing related methods, URL structure parsing methods, and data cleaning and verification methods, and then extracts features from the parts of the URL except the domain name according to 7 dimensions to construct URL rules;

[0078] S3, build a domain name rule library. For the standardized URL data, remove the whitelist and meaningless rule-related data to build a domain name rule library;

[0079] S4, rule importance score calculation, performing rule importance score calculation on the URL data in the domain name rule library, wherein the rule importance score calculation includes rule complexity importance score calculation and rule content importance score calculation;

[0080] S5, build a similar domain name recognition model based on the rule importance score calculation result, and recommend similar domain names based on the input domain name.

[0081] Example:

[0082] like Figure 1-5 As shown, in this embodiment, an Internet enterprise has huge IT assets, including numerous OA systems, surveillance camera management systems, network service platforms, etc. These assets are accessed through different domain names. Enterprises generate large amounts of website request link data every day. With the continuous expansion of business and the increasing complexity of the system, these domain names need to be effectively managed in order to better protect IT asset security, optimize resource allocation, and prevent various risks (such as domain name risks related to illegal content such as pornography, gambling, and pirated videos).

[0083] Using crawler technology and FOFA tools, we collected initial seed data of different domain name assets. These data were used as input datasets for similar domain name recognition models to generate whitelist rules and determine thresholds.

[0084] First, we standardize and clean large-scale website request link data to extract framework features. Secondly, based on the standardization results, we set up a multi-layer data filtering strategy and output a domain name rule library on a daily basis to provide data support for the subsequent identification of similar domain names. Finally, we provide query services, input domain name seeds, recommend special rules for a category of domain name assets, and output the top N recommended domain names with similarity rankings to the input domain name based on the intersection rule weighted similarity calculation method.

[0085] The specific process steps are as follows:

[0086] S1, URL preprocessing: convert URL data to lowercase, remove URLs containing garbled characters and abnormal punctuation, and remove URL data associated with whitelist domain names including but not limited to domain names of major companies such as Tencent, Baidu, and Byte;

[0087] Enterprises collect massive amounts of website request link data from various business departments and network devices. First, all URL data is converted to lowercase, which helps to accurately match and compare in subsequent processing.

[0088] For example, "https: / / EXAMPLE.COM / page" and "https: / / example.com / page" will be treated as the same format and processed; then, these URLs are filtered to remove URLs containing garbled characters and abnormal punctuation. Garbled characters may be caused by data transmission errors or encoding problems, and abnormal punctuation may not conform to the standard grammatical structure of the URL.

[0089] For example, URLs like "https: / / example.com / pa%^&ge" will be excluded. In addition, since the domain names of major companies such as Tencent, Baidu and ByteDance have little relevance to internal IT asset management of enterprises, URL data related to these whitelisted domain names will be excluded; for example, any URL pointing to Baidu search pages or certain public services under Tencent will not be included in the subsequent analysis scope.

[0090] By unifying the case of large-scale URL data, removing garbled punctuation and irrelevant whitelist domain names, we ensure data quality and analysis accuracy. We also consider whitelist removal and garbled punctuation processing from the perspective of business logic. Compared with existing technologies, we pay more attention to business logic and improve processing efficiency.

[0091] S2, URL standardization, applies standardized cleaning methods to the preprocessed URL data, such as the URL structure parsing method at the character position, and then extracts features from the part of the URL except the domain name according to 7 dimensions to construct URL rules;

[0092] The characteristic dimensions of URL standardization rules are as follows:

[0093]

[0094]

[0095] These seven dimensions cover all key parts of the URL and can convert complex URL information into quantifiable and comparable data, laying the foundation for building URL rules and subsequent domain name analysis.

[0096] S3, build a domain name rule library. For the standardized URL data, remove the whitelist and meaningless rule-related data to build a domain name rule library;

[0097] S31: Check the standardized rules one by one, and remove the rules whose length is less than or equal to 15 and contain 6 separators "|". For example, some rules that are too simple and may not have enough information are excluded, such as some URL rules that are short and contain many meaningless separators generated in a temporary test environment.

[0098] S32: Identify and remove rules whose last path parameter is an empty value. For example, rules in the form of "|2|||||id=" (the original URL is "domain+ / ?id=") do not meet the requirements for building a rule base because they do not have valid information related to action commands and file names.

[0099] S33: Rules that are associated with only one domain name are eliminated because these rules may be specific to an isolated case and are not generally representative and cannot reflect the general relationship between domain names.

[0100] S34: rules with associated domain names of 2 or less and 2000 or more are considered whitelist rules, and data associated with whitelist rules are removed; these rules may be related to some large-scale common services within the enterprise and have little to do with specific IT asset management domain names.

[0101] S35: Eliminate based on a dictionary of rules with no discriminability. For example, in IT asset management, common resource file type-related rules such as ".js" (JavaScript file) will be eliminated if they do not have specific discriminability related to IT assets. Similarly, rules for file names such as "logo.png" that may appear in multiple different types of systems and have no special IT asset management significance will also be eliminated.

[0102] The dictionary of non-discriminative rules is as follows:

[0103] Rule Type rule Contains suffix .js, .woff, .woff2, .css, .favicon, .ico, .ttf Include file name logo.png, webresourse.axd, iconfont.eot

[0104] By removing the whitelist and meaningless rule-related data, the rule base can be purified to focus on the target domain name analysis-related rules, effectively purify data, focus on the target, improve the representativeness of the rule base, improve IT asset management efficiency, enhance risk prevention capabilities, and more accurately screen rules compared to existing technologies, with better adaptability and scalability.

[0105] S4, rule importance score calculation, performing rule importance score calculation on the URL data in the domain name rule library, wherein the rule importance score calculation includes rule complexity importance score calculation and rule content importance score calculation;

[0106] The specific method for calculating the rule complexity importance score is:

[0107] The complexity is determined based on the length of the rule. The longer the rule, the higher the complexity and the higher the importance score in the similarity calculation.

[0108] Since different types of domain name asset association rules are different, the complexity of a domain name is relative, so the complexity score needs to be adjusted according to the different types of domain name association rules;

[0109] Therefore, we score the rules according to the differences in complexity of similar domain names. When recommending similar domain names, we obtain the associated rule set R of the input domain name, standardize the rule lengths in R (maximum and minimum standardization), and regard the standardized value as the rule complexity score.

[0110] Rule complexity calculation formula:

[0111]

[0112] Among them, diff_score represents the complexity weight of the rule, rule_len represents the length of the rule, and rule_len max Indicates the maximum value of the rules in the rule set R, rule_len min Indicates the minimum length of a rule in the rule set R.

[0113] The specific method for calculating the rule importance score is:

[0114] The importance is determined based on the action commands and file types associated with the rule. When the URL associated with the rule represents key actions such as management, login, and registration, the importance score is higher; when common resource file types appear in the rule, the importance score is lower. The specific method is:

[0115] Assume that the initial value of the importance score of each rule content is 0.5;

[0116] Determine whether the file type associated with the rule belongs to a common file type. If so, set the score to 0.2 and proceed to the next step. If not, proceed directly to the next step.

[0117] Common file types are listed in the following table:

[0118]

[0119]

[0120] Determine whether the action command represented by the rule contains the following keywords: index, login, admin, phone, sys, system, home, list, member, server, manage, user, password, get, record. If it contains the keywords, the score is set to 1;

[0121] For rules containing common file types, if they contain management and login keywords, the score is determined based on the keywords included, and the score is 1;

[0122]

[0123] Among them, rule means rule, and info_score represents the importance score of the rule content.

[0124] By combining the dual-dimensional calculation method of rule complexity and content importance, detailed scoring rules are formulated for different types of action commands and file types in the calculation of rule content importance scores, accurately measuring rule contributions, flexibly adapting to different domain name assets, optimizing similar domain name identification results, and improving the rationality of resource allocation. Compared with existing technical rule evaluations, it is more comprehensive and more applicable across fields.

[0125] S5, build a similar domain name recognition model based on the rule importance score calculation result, and recommend similar domain names based on the input domain name.

[0126] S51, obtain the bridge rule set: input a batch of domain name seeds of the same type D orig {orig_d i}, get the domain name association rule set R{rule j}, where d represents the domain name, i represents the index of the domain name, orig represents the original specific input domain name seed, D represents the domain name set, and D orig That is, it is represented as a set of domain name seeds, R represents the rule set, and j is the index of the rule;

[0127] For example, for a typical OA domain name "oa-system.example.com", by analyzing its URL structure, path, action command, etc., a series of association rules are obtained, which constitute the bridge rule set for the OA domain name.

[0128] S52, obtain the associated domain name set: based on the bridge rule R{rule j}, combined with the domain name rule library, query the domain name data similarity D associated with the rule simi {simi_d i}, retain domain name, rule and URL sample data;

[0129] Among them, simi is the abbreviation of similar, D simi This means that there are multiple domain name sets with similar rules to the original domain name;

[0130] For example, in the analysis of OA domain names, find other domain names with similar rules to "oa-system.example.com" and retain the domain name, rule, and URL sample data. You may find that some other OA system-related domain names have similarities in path structure, action commands, etc., such as "new-oa-system.example.com".

[0131] S53, remove whitelist rule data: The whitelist rules used in this step are different from the whitelist rules used to build the domain name rule library. They are general template rules based on the accumulation of similar domain name recommendation effects in the previous period. They are secondary filtering of the whitelist rules in the results to increase the concentration of domain name asset-specific rules in the results.

[0132] For example, some rules related to general network services but unrelated to OA-specific business logic found in the previous OA domain name analysis were filtered out again to increase the concentration of OA domain name asset-specific rules in the results.

[0133] S54, dedicated rule recommendation: based on the number of domain name assets associated with the rule, recommend the rules and URL examples of the top N associated domain names; TOP represents a sorted selection method, N is a variable, usually representing a positive integer, and TOPN means selecting the top N associated domain names after sorting a series of associated domain names in descending order according to the number of associated domain names;

[0134] Assuming N=5, for the OA domain name, 5 rules closely related to OA system resource management, user authority management, etc. and their corresponding URL samples are recommended, which will help enterprises better manage and optimize IT assets related to the OA system.

[0135] S55, domain name similarity calculation integrating importance scores: based on the intersection rule, obtain the pairwise correspondence between domain names; let A be the input domain name, B∈D simi {simi_d i} and B≠A, the intersection rule set between A and B is R inner {rule1,rule2,...,rule k}, k is the total number of rules;

[0136]

[0137] Among them, similarity(A,B) represents the similarity between domain names A and B, w diff Indicates the complexity weight of the rule. The default value is 0.4. info Indicates the content importance weight of the rule, the default value is 0.6; diff_score i The complexity importance score of the rule, info_score i Score the content importance of the rule;

[0138] In this way, the similarity between OA domain names can be more accurately evaluated, so that more precise IT asset management decisions can be made.

[0139] S56, deduplication of similar domain name results: To avoid duplicate recommendations, deduplication of similar domain names in the results is performed, and they are sorted in descending order based on domain name similarity, and only the input domain name and URL sample of the recommended result that are most similar to the output similar domain name are retained.

[0140] For example, if multiple OA domain name analysis results all recommend the same similar OA domain name, this deduplication method can only retain the most relevant input domain name and its corresponding recommended URL sample, avoiding repeated recommendations that may cause confusion to IT asset management personnel.

[0141] By implementing a scoring strategy based on rule length and content features, the weights of rules with longer length and containing keywords in the intersection rules are increased, resulting in more accurate domain name similarity calculation.

[0142] The same method can also be applied to the management of camera domain names, code receiving platform domain names, airport website domain names, pornographic domain names, gambling domain names, and pirated video domain names. For example, in the management of pornographic domain names, the above steps can quickly identify other domain names similar to known pornographic domain names, so that timely measures can be taken to protect the security of the company's IT assets and comply with laws and regulations.

[0143] By accurately identifying similar domain names, enterprises can promptly discover potential security risks, such as the potential intrusion risks of domain names related to illegal activities such as pornography and gambling, or the copyright dispute risks that may be caused by pirated video domain names, thereby protecting the security of the enterprise's IT assets and reputation.

[0144] The embodiments of the present invention are provided for the purpose of illustration and description. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present invention. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for recommending similar domain names based on website framework features, characterized in that: The following steps are involved: S1, URL preprocessing, converting URL data to lowercase, removing URLs containing garbled characters and abnormal punctuation, and removing URL data associated with whitelisted domain names including but not limited to domain names of major companies such as Tencent, Baidu, and Byte; S2, URL standardization, applies standardized cleaning methods to the preprocessed URL data, including character processing related methods, URL structure parsing methods, and data cleaning and verification methods, and then extracts features from the parts of the URL except the domain name according to 7 dimensions to construct URL rules; S3, build a domain name rule library. For the standardized URL data, remove the whitelist and meaningless rule-related data to build a domain name rule library; S4, rule importance score calculation, performing rule importance score calculation on the URL rules in the domain name rule library, wherein the rule importance score calculation includes rule complexity importance score calculation and rule content importance score calculation; S5, build a similar domain name recognition model based on the rule importance score calculation result, and recommend similar domain names based on the input domain name.

2. A method for recommending similar domain names based on website framework features as claimed in claim 1, characterized in that: In step S2, the seven dimensional features are extracted respectively: Feature Name 1: Path Details; Feature Description: The path part of the URL except the domain name and query parameters; Feature name 2: Path depth; Feature description: Number of directory levels of the URL; Feature name 3: Action command; feature Note: After the last / and before the first ?, the path parameter corresponding to the URL action; Feature name 4: resource file suffix; Feature description: the suffix of the resource file accessed in the URL; Feature name 5: File name keyword; feature Description: Extract the file names containing the given keywords; Feature name 6: File name length including suffix; feature Description: The overall file name length of the resource file accessed in the URL; Feature name 7: query parameter; Feature Description: The query parameter string after removing the query parameter value.

3. A method for recommending similar domain names based on website framework features as claimed in claim 2, characterized in that: The specific steps in step S3 are as follows: S31, remove rules with length less than or equal to 15, including 6 separators '|'; S32, remove the rule whose last path parameter is empty, the rule form is "|2|||||id="; the original URL is "domain+ / ?id=", this type of rule has no action command and file name; S33, remove the rules that are associated with only one domain name in the standardized results; S34, treat the rules with the number of associated domain names being 2 or less and 2000 or more as whitelist rules, and remove the data associated with the whitelist rules; S35, based on the dictionary of indiscriminative rules, remove indiscriminative rules.

4. A method for recommending similar domain names based on website framework features as claimed in claim 3, characterized in that: In step S35, based on the non-discriminative rule dictionary, the rules without discrimination are eliminated as follows: Rule type 1: contains suffix name; Rules: .js, .woff, .woff2, .css, .favicon, .ico, .ttf Rule type 2: Include file name; Rules: logo.png, webresourse.axd, iconfont.eot.

5. A method for recommending similar domain names based on website framework features as claimed in claim 1, characterized in that: The specific method for calculating the rule complexity importance score in step S4 is: The rule complexity importance score calculation is based on the complexity of the rule length. The longer the rule, the higher the complexity and the higher the importance score in the similarity calculation. Since different types of domain name asset association rules are different, the complexity of a domain name is relative, so the complexity score needs to be adjusted according to the different types of domain name association rules; Therefore, we score the rules according to the differences in complexity of similar domain names. When recommending similar domain names, we obtain the associated rule set R of the input domain name, standardize the rule lengths in R (maximum and minimum standardization), and regard the standardized value as the rule complexity score. Rule complexity calculation formula: Among them, diff_score represents the complexity weight of the rule, rule_len represents the length of the rule, and rule_len max Indicates the maximum value of the rules in the rule set R, rule_len min Indicates the minimum length of a rule in the rule set R.

6. A method for recommending similar domain names based on website framework features as claimed in claim 1, characterized in that: The specific method for calculating the rule importance score in step S4 is: The importance is determined based on the action commands and file types associated with the rule. When the URL associated with the rule represents key actions such as management, login, and registration, the importance score is higher; when common resource file types appear in the rule, the importance score is lower. The specific method is: Assume that the initial value of the importance score of each rule content is 0.5; Determine whether the file type associated with the rule belongs to a common file type. If so, set the score to 0.2 and proceed to the next step. If not, proceed directly to the next step. Determine whether the action command represented by the rule contains the following keywords: index, login, admin, phone, sys, system, home, list, member, server, manage, user, password, get, record. If it contains the keywords, the score is set to 1; For rules containing common file types, if they contain management and login keywords, the score is determined based on the keywords included, and the score is 1; Among them, rule means rule, and info_score represents the importance score of the rule content.

7. A method for recommending similar domain names based on website framework features as claimed in claim 6, characterized in that: Whether the file type associated with the judgment rule belongs to common file types, common file types include mp4, flv, m4s, m3u8, mp3, ts, image, jpeg, png, gif, jpg, zip, apk, txt, log, 7z, webp and heif.

8. A method for recommending similar domain names based on website framework features as claimed in claim 1, characterized in that: The specific process of constructing the similar domain name identification model in step S5 is as follows: S51, obtain the bridge rule set: input a batch of domain name seeds of the same type D orig {orig_d i }, get the domain name association rule set R{rule j }, where d represents the domain name, i represents the domain name index, orig represents the specific domain name seed, D represents the domain name set, and D orig That is, it is represented as a set of original domain name seeds, R represents the rule set, and j is the index of the rule; S52, obtain the associated domain name set: based on the bridge rule R{rule j }, combined with the domain name rule library, query the domain name data similarity D associated with the rule simi {simi_d i }, retain domain name, rule and URL sample data; Among them, orig is the abbreviation of original, orig It represents the set of original input domain names; sim is the abbreviation of similar. simi This means that a new similar domain name data set is recommended based on the input domain name set combined with the bridge rule; S53, remove whitelist rule data: The whitelist rule is based on the general template rule accumulated from the previous similar domain name recommendation effect, and the whitelist rule in the result is filtered twice to increase the concentration of domain name asset-specific rules in the result; S54, dedicated rule recommendation: based on the number of domain name assets associated with the rule, recommend the rules and URL examples of the top N associated domain names; TOP represents a sorted selection method, N is a variable, usually representing a positive integer, and TOPN means selecting the top N associated domain names after sorting a series of associated domain names in descending order according to the number of associated domain names; S55, domain name similarity calculation integrating importance scores: based on the intersection rule, obtain the pairwise correspondence between domain names; let A be the input domain name, B∈D simi {simi_d i } and B≠A, the intersection rule set between A and B is R inner {rule1, rule2, ..., rule k }, k is the total number of rules; Among them, similarity(A,B) represents the similarity between domain names A and B, w diff Indicates the complexity weight of the rule. The default value is 0.

4. info Indicates the content importance weight of the rule, the default value is 0.6; diff_score i The info_score is the complexity importance score of the rule. i Score the content importance of the rule; S56, deduplication of similar domain name results: To avoid duplicate recommendations, deduplication of similar domain names in the results is performed, and they are sorted in descending order based on domain name similarity, and only the input domain name and URL sample of the recommended result that are most similar to the output similar domain name are retained.

9. A method for recommending similar domain names based on website framework features as claimed in claim 1, characterized in that: The similar domain name recommendation method based on website framework features supports the following types of recommended similar domain names: OA domain names, camera domain names, code receiving platform domain names, airport website domain names, pornography-related domain names, gambling-related domain names and pirated video domain names.

Images