A method and system for determining the homology of illegal websites

Abstract

This invention provides a method and system for determining the homology of illegal websites, relating to the field of network monitoring technology. The determination method includes: using a headless browser to realistically render the entry addresses of two websites to be determined, and recursively crawling all pages and link information within the corresponding two-level backlink range, starting from the entry address, to obtain the corresponding raw collected data; performing data normalization and cleaning on the raw collected data to obtain the set of second-level backlink domains and statistical data of the two websites to be determined; obtaining multi-feature data based on the set of second-level backlink domains and statistical data; and fusing the various multi-feature data to obtain homology probability data. This invention elevates homology determination from a single, superficial similarity judgment to a comprehensive quantitative assessment based on multiple evidence such as structure, attribution, and content, thereby significantly improving the accuracy of homology association for illegal websites disguised by methods such as domain switching, mirroring, and template reuse.

Description

Technical Field

[0001] This invention relates to the field of network monitoring technology, and more specifically, to a method and system for determining the origin of illegal websites. Background Technology

[0002] With the rapid development of internet technology and the widespread use of one-click website building tools, a large number of gray and black market businesses, such as online gambling, fraud, pornography, and illegal transactions, have migrated online. To evade regulation and legal traceability, these illegal websites commonly employ a series of anti-traceability and anti-detection techniques, such as using overseas servers for hosting, frequently changing domain names (domain name switching), using CDN or proxy services to hide their real IP addresses, reusing or slightly modifying website templates ("reskinning"), loading core content through dynamic scripts, and deploying anti-crawler mechanisms. These strategies present law enforcement agencies and security teams with severe challenges in tracking criminal gangs, linking related cases, and conducting large-scale governance, including difficulties in obtaining evidence, insufficient evidence for consolidation of cases, and low efficiency.

[0003] Currently, related technologies suffer from systemic shortcomings in determining the specific objective of homology, including target deviation, feature fragility, difficulty in scaling, and a lack of interpretable and quantifiable frameworks. Specifically: Target deviation: Most technologies aim to determine "whether it is malicious," rather than specifically serving the attribution and consolidation needs of "whether it is homology"; Feature fragility: The URLs, static templates, or visual features they rely on are easily tampered with or circumvented by attackers at low cost; Cost-scale contradiction: Methods that can obtain deep features (such as dynamic analysis) are costly and slow, making it difficult to cope with the batch comparison needs of the massive number of websites on the Internet.

[0004] Therefore, there is an urgent need for a new method that can overcome the above-mentioned shortcomings, is specifically designed for determining the origin of illegal websites, and is automated, robust, scalable, and auditable. Summary of the Invention

[0005] The problem solved by this invention is one or more of the aforementioned related technical problems.

[0006] To address the above problems, this invention provides a method and system for determining the origin of illegal websites.

[0007] In a first aspect, the present invention provides a method for determining the origin of illegal websites, comprising: For the entry addresses of the two websites to be judged, a headless browser is used to render them realistically. Starting from the entry address, all pages and link information within the corresponding two-level external link range are recursively crawled to obtain the corresponding raw data. The original collected data is normalized and cleaned to obtain the second-level backlink domain set and statistical data of the two websites to be judged. Based on the set of outer chain domains in the second layer and the statistical data, multi-feature data is obtained; The multi-feature data are fused to obtain homogeneous probability data.

[0008] Optionally, the multi-feature data includes the domain name structure similarity, server association data, second-level backlink duplication rate, and core page overlap between the two websites to be judged; the process of obtaining multi-feature data based on the second-level backlink domain set and the statistical data includes: The domain name structure similarity, server association data, and core page overlap are determined based on the statistical data. The repetition rate of the second-level backlinks is determined based on the two sets of second-level backlink domains.

[0009] Optionally, the statistical data includes registered domains; determining the domain name structural similarity based on the statistical data includes: According to the preset division rules, the registration domain of the website to be judged is segmented to obtain the corresponding fragment type sequence; Based on a preset algorithm, the domain name structure similarity is obtained according to the two fragment type sequences.

[0010] Optionally, the statistical data also includes domain names, and the server association data is determined based on the statistical data, including: Perform DNS resolution on the domain names of the two websites to be determined, and obtain the corresponding set of IP addresses; The corresponding attribution information is determined based on the corresponding set of IP addresses, and based on a set-based comparison strategy, it is determined whether there is any intersection between the two sets of IP addresses in any of the attribution information. If any of the attribution information exists such that at least one pair of IP addresses in the two sets of IP addresses are the same, then it is determined that the servers of the two websites to be determined belong to the same place, and the server association data is output as 1. If there are no identical IP addresses in all the attribution information, it is determined that the server attribution of the two websites to be judged is inconsistent, and the server association data is output as 0.

[0011] Optionally, the statistical data also includes all corresponding pages and the link relationships of each page, inbound link frequency, outbound link frequency, external link frequency, and business keyword hit data. The core page overlap is determined based on the statistical data, including: Based on all the corresponding pages and the link relationships between each page, an internal directed graph is constructed with each page as a node and the internal links between the pages as directed edges. Based on each page in the directed graph within the site, the corresponding core score is determined according to the inbound link degree, outbound link degree, external link degree, and business keyword hit data of the corresponding page. The core scores of all pages in the two websites to be judged are sorted and filtered to obtain the corresponding core page set; Based on the two sets of core pages, the number of pages with the same page identifier is determined, and the overlap of the core pages is determined based on the number of pages.

[0012] Optionally, the determination of the corresponding core score based on each page in the directed graph within the site, according to the corresponding inbound link degree, outbound link degree, external link degree, and business keyword hit data, includes: Formula 1 is used to determine the corresponding core score based on the inbound link degree, outbound link degree, external link degree, and business keyword hit data of the page. Wherein, Equation 1 is: ; in, The core score for page p. For the number of inbound links to page p, The number of inbound links to page p within the site. The external link degree of page p, The reward value for page p hitting the business keyword; These are the corresponding preset weighting coefficients.

[0013] Optionally, the method for determining the origin of illegal websites includes: When it is necessary to perform pairwise homology determination on multiple websites to be determined, the minimum hash algorithm is used to generate a corresponding fixed-length signature based on the set of second-level backlink domains of each website to be determined. Based on the locality-sensitive hashing method, candidate website pairs are obtained by filtering the signatures corresponding to all the websites to be judged; and the duplication rate of the second-level backlinks is calculated for the candidate website pairs.

[0014] Optionally, fusing the various multi-feature data to obtain homogeneous probability data includes: Formula 2 is used to fuse the domain name structure similarity, the server association data, the second-level backlink duplication rate, and the core page overlap to obtain the same-origin probability data. Wherein, Equation 2 is: ; Where P represents the homology probability data. The domain name structure similarity, For the server-related data, The repetition rate of the second-layer external links. The overlap of the core pages. These are the corresponding weight data.

[0015] Optionally, the step of fusing the various multi-feature data to obtain homogeneous probability data includes: Based on a preset threshold range, the homology determination level is determined according to the homology probability data.

[0016] Secondly, this invention provides a system for determining the origin of illegal websites, comprising: The collection unit is used to perform realistic rendering of the entry addresses of the two websites to be judged using a headless browser, and recursively crawl all pages and link information within the corresponding two-level external link range starting from the entry address to obtain the corresponding raw collection data. The processing unit is used to perform data normalization and cleaning on the original collected data to obtain the second-level backlink domain set and statistical data of the two websites to be judged. The processing unit is further configured to obtain multi-feature data based on the set of second-layer outer chain domains and the statistical data, and to fuse the multi-feature data to obtain homogeneous probability data.

[0017] The beneficial effects of the illegal website homology determination method and system of the present invention are: By constructing a complete technical process of "dynamic collection—cleaning and standardization—multi-feature extraction—probabilistic fusion," several beneficial effects have been achieved: Employing realistic rendering and two-layer recursive crawling, it effectively captures the deep external link structure and core pages of a website, overcoming the limitations of static analysis and the lag of blacklists; by standardizing and cleaning the raw data, interference from URL parameters, short links, CDN noise, etc., is eliminated, ensuring the accuracy and consistency of subsequent feature comparisons; based on the cleaned data, multi-dimensional features are extracted and fused, elevating the determination of homology from a single, superficial similarity judgment to a comprehensive quantitative assessment based on multiple evidence such as structure, attribution, and content. This significantly improves the accuracy, robustness, and interpretability of homology association for illegal websites disguised by methods such as domain switching, mirroring, and template reuse, providing an efficient and reliable technical means for network supervision and evidence collection. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a method for determining the homology of illegal websites according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of an illegal website homology determination system according to an embodiment of the present invention. Detailed Implementation

[0019] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0020] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0021] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0022] It should be noted that the terms "one" and "more" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0023] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0024] The relevant technical solutions and their shortcomings are mainly reflected in the following aspects: First, blacklist-based blocking mechanisms are widely deployed in various security gateways and browsers. This method matches and blocks malicious domains, URLs, or IP addresses by maintaining a list. While simple to implement and direct in blocking, it is essentially a passive response strategy, suffering from serious time lags and coverage blind spots. Blacklists cannot promptly include newly registered domains or rapidly generated mirror sites, and are easily bypassed by attackers through simple domain name changes or URL redirects, failing to meet the need for proactive discovery and association of new variants or homologous domains.

[0025] Secondly, detection methods based on static features have become a research focus in recent years. These methods typically do not dynamically execute page actions, but rather extract features based on URL lexical features, page HTML / DOM structure, JavaScript code style, visual rendering similarity, or text content, and then use machine learning models for classification. For example, Ma J et al. studied methods for identifying malicious websites using URL features. These methods achieve a degree of automated detection, but their core judgment is "whether it is illegal," rather than "whether it is from the same origin." More importantly, the static features they rely on, such as templates, styles, and text, are easily circumvented by "skinning" techniques. The same operator can generate websites with vastly different appearances but the same backend origin simply by changing the front-end template, resulting in poor robustness of such methods in same-origin determination.

[0026] Secondly, detection mechanisms based on dynamic behavior, such as high-interaction honeypot technology, determine malicious intent by simulating user access and observing dynamic page behaviors (such as malicious script execution and system calls). While this method can capture more conclusive behavioral evidence, it is costly to build and maintain, and carries certain security risks. More importantly, its design is intended for in-depth analysis of malicious behavior from a single individual, resulting in limited throughput and making it unsuitable for scenarios involving rapid comparison and correlation of massive numbers of websites. Furthermore, its output focuses on determining the nature of the behavior and does not inherently provide structured, co-originating evidence for joint investigations.

[0027] To address the problems existing in the aforementioned related technologies, embodiments of the present invention provide a method and system for determining the origin of illegal websites.

[0028] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for determining the origin of illegal websites, including: Step S100: For the entry addresses of the two websites to be judged, use a headless browser to perform realistic rendering, and starting from the entry address, recursively crawl all pages and link information within the corresponding two-level external link range to obtain the corresponding raw data.

[0029] Specifically, step S100 is the data acquisition foundation for the entire homology determination process. Its core objective is to simulate real user access behavior and automatically and comprehensively capture the structured data of the two websites to be determined, as well as their external connections.

[0030] The specific process is as follows: Initialization and target setting: Receive the entry addresses of two websites to be evaluated (e.g., https: / / site-a-example.com and https: / / site-b-sample.net). Create a separate data collection task for each website.

[0031] Launch a headless browser environment: Start a headless browser instance (such as Headless Chrome, Chromium, or Selenium) in the background. This browser has a complete rendering engine but no graphical interface, thus efficiently performing operations such as page loading, JavaScript parsing and execution, and CSS rendering, just like a real user opening a webpage.

[0032] Recursive crawling (depth=2): Starting from the given entry address (depth 0), it executes a breadth-first or depth-first recursive crawling strategy, but strictly limits the recursion depth to 2 levels.

[0033] First-level crawling: Access the browser's entry address and wait for the page to fully load (including content dynamically generated via AJAX / JavaScript). After loading, extract all outbound links within the current page (i.e., The URLs that the tags point to. These links point to pages that constitute the "first-level backlink" page set.

[0034] Second-level crawling: For each external link page URL crawled in the first level, a new browser tab or reused instance is launched to access, render, and load it. Then, all outbound links within these "first-level external link pages" are extracted. The pages these links point to constitute the "second-level external link" page set.

[0035] Recursion Termination: Once the crawling depth reaches 2, crawling to deeper levels will cease.

[0036] During the crawling of each page layer, the following raw data is recorded and stored synchronously, including: Page content: complete HTML source code and the constructed DOM tree structure. Network resources: all network requests and responses generated during page loading (which can be recorded in HAR format), which helps in analyzing resource dependencies and redirection chains. Visual snapshots: screenshots of the rendered page to solidify its visual appearance. Link graph: recording the correspondence between each page and its outbound links, forming preliminary link relationship data.

[0037] In some implementations, it is assumed that it is determined whether website A (example.com) and website B (sample.net) are from the same origin.

[0038] Crawl the site A: Access https: / / example.com in your browser. After the page loads, you'll find links pointing to https: / / example.com / login, https: / / example.com / pay, and an external ad link https: / / ad-network.com / offer123.

[0039] The browser will then continue to visit https: / / example.com / login and extract all the links on the login page (such as the forgot password page https: / / example.com / forgot and an external JS library link https: / / cdn.com / lib.js).

[0040] Similarly, visit https: / / example.com / pay and extract its link.

[0041] Note: The external link https: / / ad-network.com / offer123 will also be accessed and its second-level links will be extracted, but this is part of the exploration of "external link sites" and the purpose is to obtain the complete second-level external link diagram of site A.

[0042] Crawl Bilibili: Perform the exact same recursive crawling process on https: / / sample.net to obtain all pages and links within its two-level scope.

[0043] Ultimately, two sets of raw data packages containing webpage HTML, DOM, screenshots, and link relationships were obtained, corresponding to websites A and B respectively.

[0044] The adopted "two-layer recursive dynamic crawling based on a headless browser" strategy lays a solid and high-quality data foundation for subsequent source attribution determination. Its direct benefits include: First, by using real browser rendering, it ensures the complete capture of page content and links that rely on JavaScript dynamic loading or rendering, overcoming the limitations of traditional static crawlers in handling modern web technologies and greatly improving the completeness and authenticity of data collection. Second, explicitly limiting the recursion depth to two layers achieves the optimal engineering balance between "information coverage breadth" and "collection time / resource cost." A two-layer depth is sufficient to capture a website's core business pages (usually within one or two hops) and its main external traffic-driving links, while avoiding the exponential growth in cost and potential deadlock risks associated with infinite recursion. This makes the method feasible for large-scale implementation on massive targets. This step fundamentally guarantees the reliability and representativeness of the subsequently extracted features (especially external link structures and core pages).

[0045] Step S200: Perform data normalization and cleaning on the original collected data to obtain the set of second-level backlink domains and statistical data of the two websites to be judged.

[0046] Specifically, the messy and noisy raw data acquired in step S100 is transformed into clean, standardized, and directly usable structured data for quantitative comparison. Essentially, it is a data preprocessing and refining pipeline.

[0047] The specific process is as follows: URL normalization: For each URL (including entry address, page link, resource link, etc.) that is crawled, a series of rule-based operations are performed to convert it into a "standard form".

[0048] The operations include: unifying the protocol to HTTPS (or HTTP); unifying domain name case to lowercase; removing the default port (e.g., 80); standardizing the path encoding; and, most importantly, removing all tracking parameters and session identifiers, such as utm_source, gclid, sessionid, etc., as these parameters are meaningless for identifying the page itself and will cause duplication.

[0049] Short Links and Redirection Resolution: Identify and recursively access short link services (such as bit.ly / xxx) or 3xx status codes returned by the page (such as 301, 302 redirects) until the final, directly reachable "landing page" URL is obtained. This process records the complete redirect chain.

[0050] Registered Domain Extraction and On-Site / Off-Site Determination (eTLD+1 Extraction): Based on a list of common suffixes, extract the "valid top-level domain + first-level domain" from the normalized URL, i.e., the registered domain. For example, extract example.co.uk from https: / / sub.www.example.co.uk / path.

[0051] Based on this registered domain, determine whether a link is an "internal link" or an "external link". Links under the same registered domain are used to build the internal graph later; links from different registered domains are grouped into the "second-level external link domain set".

[0052] Public noise filtering: Maintain a list of known public infrastructure and service domain names (such as CDN: cloudflare.com, akamai.net; image hosting: imgur.com; statistics: google-analytics.com; font / icon libraries, etc.).

[0053] Compare all extracted backlink domains with this list and filter out these noisy domains directly. This is because they are shared by a large number of irrelevant websites and do not contain valid information for same-origin determination.

[0054] Data Structured Output: Second-Level External Link Domain Set: After all the above cleaning steps, this is the final, unique, and noise-removed set of external registered domains retained for each website.

[0055] Statistical Data: Page Granular Data: Records the inbound link count (how many other internal pages link to it), outbound link count (how many other internal pages it links to), and backlink count (how many different external registered domains it links to), as well as whether it hits preset business keywords (e.g., the URL path contains / register, / deposit). Link Relationship Table: The correspondence between the cleaned page URLs and their outbound link targets.

[0056] By thoroughly normalizing the original URLs (e.g., standardizing formats, removing variable parameters, and expanding redirects) and accurately extracting registered domains based on a list of common suffixes, this step eliminates "pseudo-differences" caused by differences in URL syntax, obfuscated tracking parameters, or short link redirects, ensuring the comparability of pages and links across different websites. Furthermore, by proactively filtering public noise domains such as CDNs and image hosting services, this step effectively removes the background "data noise" shared by most websites and lacking discriminative significance, thus significantly sharpening the "signals" that truly reflect the operator's intent and website associations—namely, those unique or shared third-party partner and traffic channel domains. This process transforms the chaotic raw data into clean, consistent feature "raw materials," directly determining the accuracy and stability of subsequent feature extraction, ensuring that same-origin determination is free from superficial technical details and directly addresses the essence of operations.

[0057] Step S300: Based on the set of second-level outer link domains and the statistical data, obtain multi-feature data.

[0058] Specifically, the core task of step S300 is to quantitatively extract multidimensional features from the standardized data produced in step S200 that can reveal the intrinsic relationship between the two websites. These features characterize website attributes from different technical dimensions, providing an evidentiary basis for subsequent comprehensive judgment.

[0059] The specific process may include: Determining the direction of feature extraction: Based on the analysis of similar operational behavior, features can be designed from four complementary dimensions: such as surface identifier relevance: analyzing the naming rules of the website domain itself; infrastructure relevance: analyzing the network server resources on which the website relies; external traffic relevance: analyzing which external sites the website actively links to; and internal structure relevance: analyzing which pages within the website undertake core business functions.

[0060] Feature quantification calculation: For two websites to be judged (denoted as site A and site B), parallel calculation is performed using the output of S200. For example, feature calculation based on domain names: the entry domain names of the two websites can be directly analyzed to quantify the similarity of their naming patterns.

[0061] Based on server information calculation characteristics: the IP addresses and associated information obtained from the domain name resolution of two websites can be used to determine whether their server resources (such as hosting providers and geographical locations) overlap.

[0062] Based on the characteristics of external link set calculation: the cleaned "second-level external link domain sets" of site A and site B can be compared, and a quantitative repetition ratio value can be obtained through set operation.

[0063] Based on page statistics, the following features can be calculated: By using the statistical data recorded for each page, such as "inbound link rate", "outbound link rate", "external link rate" and "business keyword hit rate", the most important core pages within each website can be identified, and then the overlap ratio of the core pages of the two websites can be calculated.

[0064] Feature normalization: The calculated feature values ​​are uniformly mapped to a scale between 0 and 1 using mathematical methods, or directly produce binary results of 0 or 1, so as to facilitate subsequent weighted fusion and comparison.

[0065] In some embodiments, such as website A (example.com) and website B (sample.net), data cleaning has been performed.

[0066] Calculate the surface identifier relevance features: Analyze the two strings example.com and sample.net and find that they both follow the pattern of "English word + .com / net" and have similar lengths. Therefore, calculate a score to represent the similarity of the naming patterns of the two.

[0067] Computational infrastructure relevance characteristics: Parsing example.com yields IP 1.1.1.1 (originating from Alibaba Cloud, Hangzhou), and parsing sample.net yields IP 2.2.2.2 (originating from Alibaba Cloud, Shanghai). Due to cloud service provider matching, the system determines this characteristic as "consistent" and outputs a value of 1.

[0068] Calculate the external link relevance characteristics: Website A's external link domain set is {'ad-partner.com', 'another-gambling-site.com'}, and Website B's external link domain set is {'ad-partner.com', 'payment-processor.com'}. The intersection of the two sets is {'ad-partner.com'}, and the union has three elements. The repetition rate (intersection-union ratio) is calculated to be 1 / 3 ≈ 0.33.

[0069] Calculate the internal structural relevance features: Based on inbound and outbound link data for each page, the core pages of website A are identified as { / login, / deposit, / register, / account}, and the core pages of website B are identified as { / signin, / deposit, / register, / wallet}. Comparison reveals overlap on the / deposit and / register pages, with an overlap ratio of 2 / 4 = 0.5.

[0070] In this embodiment, the duplication rate of second-level external links is defined as the Jaccard Similarity coefficient between the two sets: (Formula 1).

[0071] Explanation of the relationship: The repetition rate of second-level backlinks is the same as Jaccard's, and the two are equivalent, with no second metric.

[0072] Four quantified feature values ​​were obtained, which together constitute "multi-feature data" describing the relationship between the two websites.

[0073] The multi-dimensional feature calculations employed construct a multi-layered, three-dimensional evidence system for determining homology. Its core benefit lies in the simultaneous extraction of quantitative features from four technical levels: "surface identification," "infrastructure," "external traffic redirection," and "internal structure." This step overcomes the vulnerability of single-feature dimensions, which are easily circumvented or imitated. Specifically, the "external traffic redirection correlation" and "internal structure correlation" features deeply uncover stable behavioral traces left by operators when organizing website functions and traffic networks—traces that are difficult to disguise in the short term—becoming strong signals for determining homology. Meanwhile, the "surface identification" and "infrastructure" features serve as rapid and low-cost auxiliary verification, forming a feature combination with clear priorities and mutual corroboration. This multi-feature fusion strategy significantly improves the robustness and accuracy of the determination method against common adversarial tactics such as "skinning," changing domain names, and using different hosts, providing a solid quantitative basis for drawing reliable homology conclusions.

[0074] Step S400: The multi-feature data are fused to obtain homologous probability data.

[0075] Specifically, step S400 is the core of the decision-making process. Its task is to integrate the multiple quantitative features extracted in step S300, which reflect different dimensions, into a unified, easy-to-understand, and easy-to-use comprehensive judgment index, namely, homogeneous probability data. This process simulates the thinking of experts in making comprehensive judgments, but achieves objectivity, quantification, and standardization through algorithms.

[0076] For example, the feature values ​​calculated for the two websites in step S300 are combined into a feature vector in a fixed order (e.g., [domain similarity, server consistency, backlink duplication rate, core page overlap]). For example, the feature vectors of sites A and B may be represented as [0.2, 1.0, 0.33, 0.5].

[0077] The feature values ​​are then fused, for example, by using weighted linear fusion: a predefined fusion model is used to assign a weight to each feature in the feature vector to reflect the different contributions or reliability of different features to the conclusion of "homology". Subsequently, a total score is calculated by weighted summation (or other linear combination methods).

[0078] Probabilistic Output: The total score calculated in the previous step is transformed mathematically (e.g., by the Sigmoid function) or its range is directly limited to between 0 and 1, mapping it to the final homology probability data (P). The closer the P value is to 1, the higher the probability that the two websites are from the same source; the closer it is to 0, the lower the probability.

[0079] Alternatively, a grading suggestion can be selected: based on the obtained probability value P, it can be mapped to a more intuitive qualitative level. For example, a threshold can be set to label the results with P≥0.8 as "highly suspected homology", the results with 0.5≤P<0.8 as "moderately suspected homology", and the results with P<0.5 as "lowly suspected homology".

[0080] By assigning differentiated weights to features with varying reliability and discriminative power and then scientifically integrating them, this step constructs a comprehensive decision-making model that balances stability and sensitivity. It fully leverages the decisive role of strongly correlated signals such as backlinks and core page structure, while also reasonably incorporating the corroborating value of auxiliary signals like domain names and servers, thus deriving a quantified homology probability value between 0 and 1. This probability value not only provides an intuitive and clear basis for judgment, greatly improving decision-making efficiency, but more importantly, its clear feature contribution (item score) provides interpretability and traceability for the judgment result. When review or questioning is required, it is possible to clearly trace back which specific features led to the current probability value, meeting the stringent requirements of transparency and verifiability in technical judgment processes in judicial, auditing, and other scenarios. Ultimately, it outputs an authoritative conclusion that is objective, interpretable, and directly applicable to subsequent processes.

[0081] In this embodiment, by constructing a complete technical process of "dynamic acquisition—cleaning and standardization—multi-feature extraction—probabilistic fusion," several beneficial effects have been achieved: It employs realistic rendering and two-layer recursive crawling to effectively capture the deep external link structure and core pages of a website, overcoming the limitations of static analysis and the lag of blacklists; by standardizing and cleaning the raw data, interference from URL parameters, short links, CDN noise, etc., is eliminated, ensuring the accuracy and consistency of subsequent feature comparisons; based on the cleaned data, multi-dimensional features are extracted and fused, elevating the determination of homology from a single, superficial similarity judgment to a comprehensive quantitative assessment based on multiple evidence such as structure, attribution, and content. This significantly improves the accuracy, robustness, and interpretability of homology association for illegal websites disguised by methods such as domain switching, mirroring, and template reuse, providing an efficient and reliable technical means for network supervision and evidence collection.

[0082] Optionally, the multi-feature data includes the domain name structure similarity, server association data, second-level backlink duplication rate, and core page overlap between the two websites to be judged; the process of obtaining multi-feature data based on the second-level backlink domain set and the statistical data includes: The domain name structure similarity, server association data, and core page overlap are determined based on the statistical data. The repetition rate of the second-level backlinks is determined based on the two sets of second-level backlink domains.

[0083] Optionally, the statistical data includes registered domains; determining the domain name structural similarity based on the statistical data includes: According to the preset division rules, the registration domain of the website to be judged is segmented to obtain the corresponding fragment type sequence; Based on a preset algorithm, the domain name structure similarity is obtained according to the two fragment type sequences.

[0084] Specifically, after completing the data cleaning and obtaining the standardized dataset, this step defines four quantitative anchor points for homology determination and elaborates on the calculation logic of one of them.

[0085] The logic behind multi-feature data composition: This scheme clarifies that the "multi-feature data" used for comprehensive judgment consists of four features, each with its own data source and computational objective: Domain structure similarity, server association data, and core page overlap: the calculation of these three mainly relies on "statistical data". Here, "statistical data" is a cleaned, structured data set that includes not only the registered domain itself, but also more information derived from or related to the registered domain, such as IP address attribution, page link graph, keyword hit rate, etc.

[0086] Second-level backlink duplication rate: The calculation of this feature directly relies on the "second-level backlink domain set", that is, the consistency of two websites in terms of external traffic destination is quantified by set operations (such as the Jaccard similarity coefficient).

[0087] Example of domain name structure similarity calculation: This feature identifies traces of "naming habits" left behind when website operators register domain names in bulk. The calculation process is as follows: Input: The registered domains (eTLD+1) of the two websites, such as site12345abc.com and site67890xyz.net.

[0088] Segmentation (fragment type serialization): The system segments the domain name body (excluding common suffixes) into type sequences according to preset rules (such as alphabetic / numeric boundaries).

[0089] For site12345abc, it is divided into [letter: "site"], [number: "12345"], [letter: "abc"], resulting in the type sequence [L, D, L].

[0090] For site67890xyz: it is divided into [letter: "site"], [number: "67890"], [letter: "xyz"], which also yields the type sequence [L, D, L].

[0091] Similarity Calculation: Based on two types of sequences, a preset algorithm (such as edit distance, sequence alignment algorithm, or dynamic time warping) is used to calculate the similarity. In this example, the two sequences are completely identical ([L, D, L]), and the algorithm will give a very high base similarity value. At the same time, the algorithm will fine-tune and normalize by taking into account factors such as sequence length and character content similarity (such as whether the lengths of numeric segments are similar), and finally output a Sim_domain value between 0 and 1, for example, 0.85.

[0092] By defining "domain structure similarity" as a quantitative comparison of domain naming patterns (alphanumeric sequences) rather than simple fuzzy string matching, this method can effectively identify attackers using a "fixed root + variable number / suffix" batch domain registration strategy. This allows attackers to associate seemingly random different domains with the same registrant or automated script, enhancing the ability to discover homogeneous "domain families." Simultaneously, by clearly dividing the four features into two calculation paths—based on "statistical data" and based on "external link sets"—a clear logical framework is constructed from raw data to multi-dimensional evidence. This makes the data processing flow of the entire judgment system more modular and interpretable, and lays a clear foundation for the independent optimization and weight adjustment of each feature. This not only improves the accuracy of the judgment but also enhances the systematicity and maintainability of its engineering implementation.

[0093] Optionally, the statistical data also includes domain names, and the server association data is determined based on the statistical data, including: Perform DNS resolution on the domain names of the two websites to be determined, and obtain the corresponding set of IP addresses; The corresponding attribution information is determined based on the corresponding set of IP addresses, and based on a set-based comparison strategy, it is determined whether there is any intersection between the two sets of IP addresses in any of the attribution information. If any of the attribution information exists such that at least one pair of IP addresses in the two sets of IP addresses are the same, then it is determined that the servers of the two websites to be determined belong to the same place, and the server association data is output as 1. If there are no identical IP addresses in all the attribution information, it is determined that the server attribution of the two websites to be judged is inconsistent, and the server association data is output as 0.

[0094] Specifically, the calculation of server correlation data is an important supporting feature calculation step in the overall homology determination framework. Its goal is not to create complex, in-depth server profiles, but to quickly determine whether two websites share or are potentially related at the infrastructure level in an engineered and efficient manner.

[0095] The domain names of the two websites to be judged (from statistical data). For example, website A: site-one.com, website B: site-two.net.

[0096] DNS resolution: Performing standard DNS resolution concurrently on two domain names to obtain all corresponding IPv4 (A record) and IPv6 (AAAA record) addresses, forming a set of IP addresses. For example: site-one.com → Parsing yields {192.0.2.10, 192.0.2.11} site-two.net → Parsing yields {198.51.100.20, 203.0.113.30} Attribution information mapping: Utilizing local or remote IP information databases, multi-dimensional attribution information is queried for each IP address. Key dimensions typically include: cloud service provider / ISP (e.g., Alibaba Cloud, Amazon AWS, a local telecom operator), geographic location (e.g., country-province-city), Autonomous System Number (ASN) (e.g., AS15169 corresponds to Google), CIDR network segment, etc. For example, the mapping result of 192.0.2.10 might be {Cloud Merchant: XX, Location: Some Location, ASN:AS12345}.

[0097] Aggregate comparison strategy: The core of this strategy is to relax the matching criteria and pursue a high recall rate, aiming to discover correlation clues at any level. However, it is not a simple comparison of whether two IP sets have the same IP address (strict matching), but rather a comparison of their attribution information sets. For each attribution dimension (cloud provider, location, ASN), the values ​​of all IP addresses of the two websites in that dimension are extracted to form two "attribution value sets".

[0098] Judgment logic: As long as the "attribution value sets" of the two websites have an intersection (i.e., at least one of the same values) on any attribution dimension, it is determined that "the server attribution is consistent".

[0099] Output: 1 if they match, 0 otherwise.

[0100] For example, when determining whether an illegal website A (bet123.com) and an illegal website B (pic456.net) share the same origin, other characteristics (such as backlinks and core pages) may vary significantly depending on the type of business.

[0101] Performing this step will resolve the IP address of site A to {1.2.3.4}, which belongs to {Cloud Provider: "Rented from a certain city's IDC", ASN: "AS12345"}; and the IP address of site B to {5.6.7.8}, which belongs to {Cloud Provider: "Rented from a certain city's IDC", ASN: "AS67890"}.

[0102] In terms of "cloud service provider", both have the same set of values: {"rented from a certain city's IDC"}, and there is an overlap.

[0103] Therefore, the server correlation data output is 1. This result serves as a weak signal: two websites with different business types may be hosted by the same operator using the same batch of server resources, increasing the likelihood that they share the same origin and providing valuable auxiliary information for comprehensive judgment.

[0104] The server association calculation method defined in this embodiment provides a low-cost, high-efficiency infrastructure-level cross-validation for the entire origin determination system. Its significant advantages lie in the following: by employing a "collective comparison strategy" and "multi-dimensional attribution information matching," this method can effectively penetrate common spoofing techniques such as CDNs and cloud proxies, capturing shared traces of websites across server hosting providers, network autonomous systems, or physical regions. Even if attackers configure completely different IP addresses for different sites, as long as these IPs point to the same cloud service provider, data center, or network operator, the method can still identify their association, thus overcoming the shortcomings of simple IP comparison, which is easily bypassed and has incomplete coverage. This design allows this feature to maintain computational lightweight while possessing high practical value, serving as a stable circumstantial evidence that effectively complements other structural features, jointly enhancing the robustness and clue discovery capabilities of the overall determination system.

[0105] Optionally, the statistical data also includes all corresponding pages and the link relationships of each page, inbound link frequency, outbound link frequency, external link frequency, and business keyword hit data. The core page overlap is determined based on the statistical data, including: Based on all the corresponding pages and the link relationships between each page, an internal directed graph is constructed with each page as a node and the internal links between the pages as directed edges. Based on each page in the directed graph within the site, the corresponding core score is determined according to the inbound link degree, outbound link degree, external link degree, and business keyword hit data of the corresponding page. The core scores of all pages in the two websites to be judged are sorted and filtered to obtain the corresponding core page set; Based on the two sets of core pages, the number of pages with the same page identifier is determined, and the overlap of the core pages is determined based on the number of pages.

[0106] Optionally, the determination of the corresponding core score based on each page in the directed graph within the site, according to the corresponding inbound link degree, outbound link degree, external link degree, and business keyword hit data, includes: Formula 1 is used to determine the corresponding core score based on the inbound link degree, outbound link degree, external link degree, and business keyword hit data of the page. Wherein, Equation 1 is: ; in, The core score for page p. For the number of inbound links to page p, The number of inbound links to page p within the site. The external link degree of page p, The reward value for page p hitting the business keyword; These are the corresponding preset weighting coefficients.

[0107] Specifically, the calculation of the core feature of core page overlap essentially transforms the subjective judgment of "which pages are the core of the website" into a quantitative identification and comparison process based on objective data and clear rules. Its goal is to identify pages that carry key business logic by analyzing the website's internal link topology and business semantics, and to reveal the commonality of their operational logic by comparing the degree of reuse of such pages across different websites.

[0108] Constructing an internal directed graph: This involves creating a table of all page URLs belonging to the same registered domain, obtained after the S200 cleaning process, and their internal link relationships. Each page is treated as a node in the graph. If page A's HTML contains a link to page B (and B is on the same site), a directed edge is created from node A to node B. The goal is to transform the discrete set of pages into a structured network that reflects internal navigation and traffic flow, which forms the basis for all subsequent structural analyses.

[0109] Calculate the core score of the page: For each page node in the directed graph, obtain its quantitative indicators from the statistical data: inbound links (how many pages link to this page), outbound links (how many other pages in the site link to this page), external links (how many external domains link to this page), and business keyword hit (whether its URL path or title contains preset keywords such as / login, / deposit, etc.).

[0110] Scoring Logic: A scoring function is designed based on the principle that a page's core importance is positively correlated with its "widespread internal referencing" and "clear business functionality," and negatively correlated with its "distributed traffic to external pages." High inbound link density: This indicates that the page is a common destination for many other pages (e.g., homepage, unified login portal), serving as a structural hub. Low outbound link density: This indicates that the page is the "endpoint" of the business flow rather than a "redirecting station," and is more likely a core business landing page (e.g., payment completion page). Hitting business keywords: This directly identifies the page's function and provides strong semantic evidence.

[0111] Sort all pages of a website from highest to lowest according to their core score. Select the top K pages (e.g., K=4) to form the core page set of the website. Each page in the set is uniquely identified by its normalized "registration domain + path" (e.g., example.com / register).

[0112] Calculate core page overlap: Compare the core page sets of websites A and B. Count the number of pages with the same page identifier in both sets. The overlap calculation formula is: Overlap = (Number of identical pages) / K. K is a value between 0 and 1. A value of 1 indicates that the first K core pages of the two websites are completely identical; a value of 0 indicates that they are completely different.

[0113] In some embodiments, the definition and scope are as follows: starting from the entry page, construct a directed graph within the site within a maximum of two hops (Hop2). Under the same registered domain (eTLD+1), pages are represented by canonical URLs consisting of "registered domain + path"; the top-level URLs of each site are identified separately. Core Page Collection and The core page overlap is defined as: ;in, This refers to the overlap between core pages.

[0114] The process of obtaining the core page set includes: page normalization and scope determination: unifying protocol / case sensitivity / port / path, expanding short links and 3xx redirects, removing fragment identifiers and tracking parameters (such as utm_, gclid, fbclid, session, etc.), and extracting eTLD+1 based on the Public Suffix List (PSL); only retaining the entry graph of internal pages. .

[0115] In some embodiments, the process of obtaining the core page set is as follows: Page normalization and scope determination: standardize protocol / case sensitivity / port / path, expand short links and 3xx redirects, remove fragment identifiers and tracking parameters (such as utm_, gclid, fbclid, session, etc.), extract eTLD+1 based on the Public Suffix List (PSL); retain only the entry graph for internal pages. .

[0116] Structural metrics statistics: After the DOMContentLoaded and network idle conditions are met, all internal links are extracted and constructed. The site has directed edges within the site; external links are only used for "backlink density" statistics. For each page... Calculate: ① Inbound link degree within the station :point to ① The number of different source pages within the site; ② The number of outbound links within the site. : ③ Number of different target pages within the site; : The number of different external domains pointed to (deduplicated by external domain eTLD+1); ④ Business keyword hits : A value of 1 is recorded when the path or text matches / register, / login, / deposit, / checkout, etc.; otherwise, a value of 0 is recorded. Duplicate hrefs on the same page are removed before counting; each different page is counted once. Template-level full-width links (header / footer links appearing simultaneously on pages exceeding the threshold proportion) contribute 0 to the inbound link contribution or are counted as... Lowering the ranking; using "registered domain + path" for page keys usually ignores queries (unless it is added to the whitelist).

[0117] Core score and ranking: Calculated based on the "high inbound links / low outbound links" rule, that is, core score is performed using Formula 1.

[0118] Each original count can be done Alternatively, min-max normalization can be used. Empirical weights can be chosen. (Can be optimized according to business needs). By Before selecting descending order The core collection consists of several pages. To prevent navigation page "ranking manipulation", diversity constraints can be set for the same path prefix.

[0119] Integration of solidification and verifiability: Standardizing URLs, three-dimensional scores, and keyword hit rates for selected core pages. Evidence citations and parameters / thesaurus / PSL version, etc., are written into the evidence package to support auditing and replay.

[0120] DOMContentLoaded indicates that the browser has completed parsing the initial HTML and building the DOM tree, but this does not guarantee that stylesheets, images, or asynchronous scripts are fully loaded. This invention uses it as the first threshold for "structural extractability," ensuring that the main link structure within the page is essentially ready.

[0121] Network idle: Within a preset time window (e.g., 1-2 seconds), there are no new resource requests or only a very small number of long-lived connection requests, indicating relatively stable resource loading. Since modern websites often rely on asynchronous loading and front-end routing, this invention uses it as a second threshold to improve the complete capture rate of backlinks and redirect chains.

[0122] Dual threshold strategy: Extraction and statistics only begin when DOMContentLoaded is triggered and network idle is subsequently met (or the total timeout limit of 15-30 seconds is reached); if background requests continue to occur on the page, the longest waiting limit or the "inactive connection threshold" is used as a fallback condition to balance quality and throughput.

[0123] The core page collection contains the following data (each record includes at least the following): url_norm (registered domain + path); in_deg, out_deg_in, out_deg_ext; keyword_hit (0 / 1 or a list of hit words); corescore (including weight and normalized version); snap_ref (screenshot / DOM summary / HAR reference); ts (timestamp), run_id (task identifier), norm_version (PSL / noise library / parameter version).

[0124] For example: when As shown in Tables 1 and 2 Table 1. Collection of core pages for site A (example.com) )surface

[0125] Table 2 Core Set of Site B (sample.net) )

[0126] As can be seen from the two tables above, the set of overlapping paths is {register, login, deposit}, therefore .

[0127] The core page overlap calculation method contributes the most stable and difficult-to-spoof structural evidence of homology in the overall solution. Its core beneficial effect lies in the following: by abstracting the website into a directed graph and quantifying page coreness based on the "inbound-outbound" topology and business keywords, this method can penetrate the differences in surface style and layout, directly targeting the business logic skeleton of website operation, and identifying the "cornerstone" pages that carry key functions such as login, recharge, and transactions. Even if attackers "reskin" the website or adjust the structure of secondary pages, these core business pages and their pivotal position in the internal link network usually remain stable, making the "overlap" feature based on this extremely strong anti-circumvention capability and robustness. This feature, as a high-weight strong signal, forms internal and external corroboration with the "second-level backlink repetition rate," together constituting a solid technical pillar for determining homology, greatly reducing the risk of misjudgment caused by surface changes, and improving the reliability and accuracy of the judgment conclusion.

[0128] Optionally, the method for determining the origin of illegal websites further includes: When it is necessary to perform pairwise homology determination on multiple websites to be determined, the minimum hash algorithm is used to generate a corresponding fixed-length signature based on the set of second-level backlink domains of each website to be determined. Based on the locality-sensitive hashing method, candidate website pairs are obtained by filtering the signatures corresponding to all the websites to be judged; and the duplication rate of the second-level backlinks is calculated for the candidate website pairs.

[0129] Specifically, key performance optimization strategies are implemented for large-scale judgment scenarios. When pairwise homology comparisons of massive websites (e.g., thousands to hundreds of thousands) are required, efficient approximate retrieval techniques are introduced to solve the previously infeasible quadratic computational complexity problem, ensuring the overall solution has the capability for large-scale deployment.

[0130] In some implementations, there are N websites, and comparing each pair requires calculating the backlink duplication rate C(N,2) times (precise Jaccard similarity calculation). When N is very large (e.g., N=10,000 requires approximately 50 million calculations), the computational load increases exponentially, which becomes unsustainable.

[0131] We need to design a method that can recall all highly similar website pairs (i.e., high Jaccard values, usually corresponding to the same source) with high probability, while reducing the number of website pairs that need to be accurately calculated from O(N²) to close to O(N).

[0132] Core operation process: Step 1: Generate MinHash signature: The set of second-level backlink domains for each website after being cleaned by the S200 steps (for example: the set for website X is {"ad-net.com", "payment-xyz.com", "tracker-abc.com"}).

[0133] Use k independent and uniform hash functions (e.g., k=128). For each set of websites, calculate the hash value of all domain elements under each hash function and record the minimum value under each hash function. Each website ultimately obtains a MinHash signature vector of length k (composed of k minimum values). The probability that the hash values ​​at the same position in the MinHash signature vectors of two sets are equal is exactly equal to the Jaccard similarity between the two sets.

[0134] Step 2: LSH Bucketing and Candidate Pair Generation: For the MinHash signature vectors of all N websites, divide each signature vector of length k into b "bands" on average, with each band containing r values ​​(satisfying k = b × r). For example, if k = 100, we can set b = 10 and r = 10.

[0135] Hash bucketing: For the i-th band (i=1 to b), concatenate the r values ​​of that band in the website signature into a string, calculate its hash value, and use it as the "bucket key" for the website in the i-th hash table. Place the website ID into the corresponding bucket.

[0136] Generating candidate pairs: Core rule: For any hash table (i.e. any "band"), all websites that fall into the same bucket form a "candidate pair" for each pair.

[0137] Principle: The probability that r values ​​from two websites are exactly the same in the same band is J. r (J is the true Jaccard value). The probability that they are consistent in at least one of the b bands is 1 - (1 - J) r ) b By adjusting b and r, an S-shaped curve can be designed so that websites with high Jaccard values ​​(e.g., >0.7) become candidates with a high probability, while websites with low Jaccard values ​​are filtered out with a high probability.

[0138] Step 3: Perform precise calculations on candidate pairs: For the list of candidate website pairs and their original set of second-level backlink domains, perform precise Jaccard calculations (using Equation (1)) only on the candidate pairs in the list. This calculation result will be used as the "second-level backlink duplication rate" feature value of the website pair and input into the subsequent fusion determination step.

[0139] For website pairs that did not become candidate pairs, The default value is 0 or marked as "no calculation required" because they are highly unlikely to be similar.

[0140] By introducing the MinHash and LSH filtering mechanisms, the overall homology determination scheme is endowed with crucial scalability and engineering practicality. Its core benefits lie in transforming the high-dimensional set similarity comparison problem into a fast hashing and bucketing operation of low-dimensional signature vectors. This step achieves a revolutionary reduction in computational complexity from quadratic to near linear when performing fully connected homology analysis on massive websites, thus making large-scale homology association analysis on databases of tens of thousands or even hundreds of thousands of websites a theoretically feasible but practically practical reality. Simultaneously, its probability-based filtering design (by adjusting parameters b and r) ensures that highly similar website pairs are retained as candidates with extremely high recall rates. This method achieves significant performance improvements while almost preserving the ability to discover suspected homology website pairs, perfectly resolving the inherent contradiction between "efficiency" and "detection rate" in large-scale data analysis. This makes this technical solution not only suitable for case-by-case analysis but also capable of supporting large-scale cyberspace governance, black market graph construction, and clue mining tasks at the platform, industry, and even national levels, greatly expanding its application boundaries and practical value.

[0141] Optionally, fusing the various multi-feature data to obtain homogeneous probability data includes: Formula 2 is used to fuse the domain name structure similarity, the server association data, the second-level backlink duplication rate, and the core page overlap to obtain the same-origin probability data. Wherein, Equation 2 is: ; Where P represents the homology probability data. The domain name structure similarity, For the server-related data, The repetition rate of the second-layer external links. The overlap of the core pages. These are the corresponding weight data.

[0142] Optionally, the step of fusing the various multi-feature data to obtain homogeneous probability data includes: Based on a preset threshold range, the homology determination level is determined according to the homology probability data.

[0143] Specifically, this step follows immediately after step S400 (multi-feature fusion to obtain homologous probability data P) and serves as the final output encapsulation of the entire method.

[0144] The homology probability data P generated by step S400 is a value between 0 and 1 (e.g., P = 0.73).

[0145] One or more thresholds are predefined to divide the continuous range of P-values ​​into several discrete, semantically defined judgment level intervals. This is a set of configurable business rules.

[0146] Level determination process: The calculated P value is compared with the preset threshold and classified into the corresponding level range.

[0147] For example: Threshold setting (can be adjusted according to business sensitivity): Strong homology (or "highly probable homology"): P ≥ 0.80 Suspected homology (or "moderately suspected homology"): 0.50 ≤ P < 0.80 Low homology (or "low degree of suspected homology / dissimilarity"): P < 0.50 Judgment operation: This is a simple logical judgment process. For example, if P=0.73, since 0.50 ≤ 0.73<0.80, the judgment level is "suspected homology".

[0148] In some embodiments, a complete same-origin determination process was performed, for example, on the websites abc-bet.com and xyz-gamble.net.

[0149] After feature extraction and fusion, the homology probability P = 0.87 between the two was calculated.

[0150] Load the preset threshold rules (strong: 0.8, suspected: 0.5, low: below 0.5).

[0151] Since 0.87 ≥ 0.80, the determination level is automatically marked as "strong homology".

[0152] The final output not only outputs the value P=0.87, but also the conclusion of "strong homology", and usually includes key sub-characteristic values ​​as evidence (such as: external link duplication rate 0.9, core page overlap 0.75, server consistency, etc.).

[0153] This conclusion of "strong common origin" can directly indicate to law enforcement or risk control personnel that the two websites are very likely operated by the same group and should be investigated together or dealt with in a related manner.

[0154] By mapping continuous probability values ​​to discrete levels such as "strong," "suspicious," and "low," which have clear action targets, this method transforms complex algorithm outputs into intuitive conclusions that non-technical users, such as relevant personnel and risk control analysts, can quickly understand without secondary interpretation, significantly lowering the application threshold of the technology. Simultaneously, the tiered approach allows for differentiated handling strategies. For example, "strongly related" sites can be urgently blocked and investigated together, while "suspiciously related" sites can be included in a key monitoring list for continuous surveillance, thereby achieving optimal resource allocation and improving the accuracy and efficiency of overall governance actions. This design makes this patented solution not merely a technical algorithm, but a complete decision support tool tailored to practical business needs.

[0155] like Figure 2 As shown, an embodiment of the present invention provides an illegal website homology determination system, comprising: The collection unit is used to perform realistic rendering of the entry addresses of the two websites to be judged using a headless browser, and recursively crawl all pages and link information within the corresponding two-level external link range starting from the entry address to obtain the corresponding raw collection data. The processing unit is used to perform data normalization and cleaning on the original collected data to obtain the second-level backlink domain set and statistical data of the two websites to be judged. The processing unit is further configured to obtain multi-feature data based on the set of second-layer outer chain domains and the statistical data, and to fuse the multi-feature data to obtain homogeneous probability data.

[0156] An embodiment of the present invention provides an illegal website homology determination device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to implement the illegal website homology determination method as described above when the computer program is executed.

[0157] This invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the illegal website homology determination method described above.

[0158] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A method for determining the origin of illegal websites, characterized in that, include: For the entry addresses of the two websites to be judged, a headless browser is used to render them realistically. Starting from the entry address, all pages and link information within the corresponding two-level external link range are recursively crawled to obtain the corresponding raw data. The original collected data is normalized and cleaned to obtain the second-level backlink domain set and statistical data of the two websites to be judged. Based on the set of outer chain domains in the second layer and the statistical data, multi-feature data is obtained; The multi-feature data are fused to obtain homogeneous probability data.

2. The method for determining the origin of illegal websites according to claim 1, characterized in that, The multi-feature data includes the domain name structure similarity between the two websites to be judged, server association data, second-level backlink duplication rate, and core page overlap; the multi-feature data obtained based on the second-level backlink domain set and the statistical data includes: The domain name structure similarity, server association data, and core page overlap are determined based on the statistical data. The repetition rate of the second-level backlinks is determined based on the two sets of second-level backlink domains.

3. The method for determining the origin of illegal websites according to claim 2, characterized in that, The statistical data includes registered domains; determining the domain name structural similarity based on the statistical data includes: According to the preset division rules, the registration domain of the website to be judged is segmented to obtain the corresponding fragment type sequence; Based on a preset algorithm, the domain name structure similarity is obtained according to the two fragment type sequences.

4. The method for determining the origin of illegal websites according to claim 3, characterized in that, The statistical data also includes domain names, and the server association data is determined based on the statistical data, including: Perform DNS resolution on the domain names of the two websites to be determined, and obtain the corresponding set of IP addresses; The corresponding attribution information is determined based on the corresponding set of IP addresses, and based on a set-based comparison strategy, it is determined whether there is any intersection between the two sets of IP addresses in any of the attribution information. If any of the attribution information exists such that at least one pair of IP addresses in the two sets of IP addresses are the same, then it is determined that the servers of the two websites to be determined belong to the same place, and the server association data is output as 1. If there are no identical IP addresses in all the attribution information, it is determined that the server attribution of the two websites to be judged is inconsistent, and the server association data is output as 0.

5. The method for determining the origin of illegal websites according to claim 4, characterized in that, The statistical data also includes all corresponding pages and the link relationships of each page, inbound links within the page, outbound links within the site, external links, and business keyword hit data. The overlap of the core pages is determined based on the statistical data, including: Based on all the corresponding pages and the link relationships between each page, an internal directed graph is constructed with each page as a node and the internal links between the pages as directed edges. Based on each page in the directed graph within the site, the corresponding core score is determined according to the inbound link degree, outbound link degree, external link degree, and business keyword hit data of the corresponding page. The core scores of all pages in the two websites to be judged are sorted and filtered to obtain the corresponding core page set; Based on the two sets of core pages, the number of pages with the same page identifier is determined, and the overlap of the core pages is determined based on the number of pages.

6. The method for determining the homology of illegal websites according to claim 5, characterized in that, The core score is determined for each page in the directed graph within the site based on its inbound link degree, outbound link degree, external link degree, and business keyword hit data, including: Formula 1 is used to determine the corresponding core score based on the inbound link degree, outbound link degree, external link degree, and business keyword hit data of the page. Wherein, Equation 1 is: ； in, The core score for page p. For the number of inbound links to page p, The number of inbound links to page p within the site. The external link degree of page p, This is the reward value for page p to hit the business keyword; These are the corresponding preset weighting coefficients.

7. The method for determining the origin of illegal websites according to claim 2, characterized in that, The method for determining the origin of illegal websites also includes: When it is necessary to perform pairwise homology determination on multiple websites to be determined, the minimum hash algorithm is used to generate a corresponding fixed-length signature based on the set of second-level backlink domains of each website to be determined. Based on the locality-sensitive hashing method, candidate website pairs are obtained by filtering the signatures corresponding to all the websites to be judged; and the duplication rate of the second-level backlinks is calculated for the candidate website pairs.

8. The method for determining the origin of illegal websites according to claim 2, characterized in that, The process of fusing the various multi-feature data to obtain homogeneous probability data includes: Formula 2 is used to fuse the domain name structure similarity, the server association data, the second-level backlink duplication rate, and the core page overlap to obtain the same-origin probability data. Wherein, Equation 2 is: ； Where P represents the homology probability data. The domain name structure similarity, For the server-related data, The repetition rate of the second-layer external links. The overlap of the core pages. These are the corresponding weight data.

9. The method for determining the origin of illegal websites according to claim 1, characterized in that, The step of fusing the various multi-feature data to obtain the homogeneity probability data includes: Based on a preset threshold range, the homology determination level is determined according to the homology probability data.

10. A system for determining the origin of illegal websites, characterized in that, include: The collection unit is used to perform realistic rendering of the entry addresses of the two websites to be judged using a headless browser, and recursively crawl all pages and link information within the corresponding two-level external link range starting from the entry address to obtain the corresponding raw collection data. The processing unit is used to perform data normalization and cleaning on the original collected data to obtain the second-level backlink domain set and statistical data of the two websites to be judged. The processing unit is further configured to obtain multi-feature data based on the set of second-layer outer chain domains and the statistical data, and to fuse the multi-feature data to obtain homogeneous probability data.

Images