Processing device, processing method, and processing program
The processing device addresses the issue of mislabeling 'unpopular but legitimate' sites by generating a permission list through web crawling and machine learning, improving the accuracy of malicious site identification and reducing false positives.
Patent Information
- Application Number
- PCT/JP2024/019773
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-05-29
- Publication Date
- 2024-12-05
AI Technical Summary
Existing methods for identifying malicious websites often mislabel 'unpopular but legitimate' sites as malicious due to reliance on popularity-based top lists, leading to false positives and difficulties in distinguishing between legitimate and phishing sites.
A processing device and method that selects reliable seed sites, performs web crawling to extract external links and DNS records, and uses machine learning to generate a permission list that includes 'unpopular but legitimate' sites, reducing false positives by focusing on link relationships and network structure.
Improves the accuracy of identifying malicious sites and controlling access by correctly indicating 'unpopular but legitimate' sites, reducing false positives and enhancing precision in malicious site identification.
Smart Images

Figure JP2024019773_05122024_PF_FP_ABST
Abstract
Description
Processing device, processing method, and processing program
[0001] The present invention relates to a processing device, a processing method, and a processing program.
[0002] An allowlist is a list of sites that have been pre-determined to be trustworthy based on certain criteria. Sites on the allowlist are excluded from malicious judgments and are considered to be non-malicious sites (legitimate sites, benign sites). An allowlist is also sometimes called a safelist or whitelist.
[0003] Allowlists are commonly used in web security to mitigate the problem of "false positives," where non-malicious websites are mistakenly identified as malicious.
[0004] However, the web is extremely vast and dynamically changing, making it difficult to manually select a comprehensive list of trusted sites and continually update and maintain an allow list.
[0005] As a result, "top lists" have become widely used in both industry and academia as de facto allow lists, based on the implicit assumption that "popularity is likely to be safe."
[0006] A top list is a ranking list of websites or domain names in order of popularity based on specific criteria. There are multiple top lists, and each one has significantly different definitions of the data used, definitions of "popularity," and calculation methods.
[0007] A representative and widely used top list is the Tranco Top Million List (Non-Patent Document 1). The Tranco Top Million List is a combination of the Alexa Top 1 Million, Cisco Umbrella 1 Million, and Majestic Million top lists. Alexa Top 1 Million is one of the most commonly used top lists and is determined based on the average number of daily visits and page views to a website. Cisco Umbrella 1 Million is a list of the most popular domain names ranked in order based on DNS observations in the DNS (Domain Name System) service provided by Cisco Umbrella (registered trademark). Majestic Million is a list of website popularity calculated based on web links provided by Majestic SEO (registered trademark).
[0008] The Secrank List (Non-Patent Document 2) is a new top list processing method that uses a voting algorithm based on the DNS request volume of each domain name. There is also the Chrome User Experience Report (CrUX), which ranks the popularity of websites based on data collected from users of the Google Chrome web browser.
[0009] Victor Le Pochat et al., “TRANCO: A Research-Oriented Top Sites Ranking Hardened Against Manipulation”, In: Proc. NDSS (2019).Qinge Xie, et al., “Building an Open, Robust, and Stable Voting-Based Domain Top List”, In: Proc. USENIX Security (2022).
[0010] There are two main examples of using top lists as de facto allow lists:
[0011] For example, let's look at the first example, which is used to remove false positives. To prevent false positives, one example is to use domain names from the Majestic top list as an allow list. Additionally, the open source threat intelligence platform MISP (Malware Information Sharing Platform) uses Alexa Top1k, Umbrella Top 20k, Majestic Top1k, and Tranco Top 10k as options for removing false positives.
[0012] Next, we will explain a second example of its use with benign labeling. Top lists are widely used when using machine learning in the security field, for example, to label domain names as "benign" or to exclude some domain names from analysis as "benign" in the analysis process. Furthermore, when detecting phishing sites, domain names listed at the top of the top list are often selected as "genuine" domains that are likely to be imitated.
[0013] However, because the Top List is a system that lists "popular" sites in order, there is a fundamental problem that "unpopular but good sites" are difficult to list. For example, sites of new companies, sites of new services of existing companies, and sites of services provided to specific regions are unlikely to be listed on the Top List.
[0014] If such sites are not listed, false positives will occur when identifying malicious sites or controlling (filtering) access to them, resulting in legitimate sites being mistakenly identified as malicious, or in the inability to distinguish between less popular legitimate sites and phishing sites.
[0015] The present invention has been made in consideration of the above, and aims to provide a processing device, processing method, and processing program that can generate an allowance list that correctly indicates ``unpopular but legitimate sites,'' thereby identifying malicious sites and improving the accuracy of access control to malicious sites.
[0016] In order to solve the above-mentioned problems and achieve the objectives, the processing device is characterized by having a selection unit that selects trusted sites as seeds, a first extraction unit that performs web crawls multiple times to extract external links from the sites selected as seeds, a second extraction unit that performs DNS crawls to extract DNS records based on the sites selected as seeds and the domain names of the external links extracted by the first extraction unit, and a generation unit that identifies trusted sites based on characteristics of the information extracted by the web crawl and / or characteristics of the information extracted by the DNS crawl, and generates an allow list indicating the trusted sites.
[0017] According to the present invention, it is possible to generate an allow list that correctly indicates "unpopular but legitimate sites," thereby improving the accuracy of identifying malicious sites and controlling access to malicious sites.
[0018] FIG. 1 is a diagram illustrating an example of the configuration of a processing device according to an embodiment. FIG. 2 is a diagram illustrating an overview of collection targets and processing during web crawling. FIG. 3 is a diagram illustrating first to fifth feature vectors. FIG. 4 is a diagram illustrating first to fifth feature vectors. FIG. 5 is a flowchart illustrating the processing procedure of processing executed by the processing device illustrated in FIG. 1. FIG. 6 is a flowchart illustrating the processing procedure of the generation processing illustrated in FIG. 5. FIG. 7 is a flowchart illustrating the processing procedure of the re-execution algorithm executed by the determination unit. FIG. 8 is a diagram illustrating a hierarchical ring model. FIG. 9 is a flowchart illustrating the processing procedure of labeling processing. FIG. 10 is a flowchart illustrating the processing procedure of the first determination processing illustrated in FIG. 9. FIG. 11 is a flowchart illustrating the processing procedure of the second determination processing illustrated in FIG. 9. FIG. 12 is a flowchart illustrating the processing procedure of the third determination processing illustrated in FIG. 9. FIG. 13 is a flowchart illustrating the processing procedure of the fourth determination processing illustrated in FIG. 9. FIG. 14 is a flowchart illustrating the processing procedure of the fifth determination processing illustrated in FIG. 9. FIG. 15 is a diagram illustrating an example of a computer that executes a processing program.
[0019] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a processing apparatus, a processing method, and a processing program according to the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the following embodiments.
[0020] The processing device according to the embodiment generates a new allowed list starting from legitimate websites, rather than a top-down approach that relies on popularity.
[0021] The processing device selects a small number of potentially trustworthy websites as seeds, crawls these seed sites, and extracts their domain names, web content, and DNS (Domain Name System) records when externally linked from the seed sites.
[0022] The processor then performs machine learning using features of the web content and DNS records of external links discovered during the crawl to filter out untrustworthy domain names, identify trusted sites, and finally generate a currently up-to-date allow list.
[0023] In this way, the processing device uses the link relationships that form the backbone of the Web space, characteristic information in the network structure, and information on a small number of known unreliable domain names as labels to identify and eliminate unreliable domain names that have not been judged so far. This enables the processing device to correctly identify "unpopular but legitimate sites" that are overlooked in top lists, making it possible to identify malicious sites with higher accuracy and reduce false positives, and to control access to malicious sites.
[0024] [Processing Device] Next, the configuration of a processing device according to an embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a processing device according to an embodiment.
[0025] The processing device 10 according to the embodiment is a server device implemented by, for example, loading a predetermined program into a computer or the like including a ROM (Read Only Memory), a RAM (Random Access Memory), a CPU (Central Processing Unit), etc., and having the CPU execute the predetermined program. The processing device 10 also has a communication interface for transmitting and receiving various information to and from other devices connected via a network, etc. As will be described later, the processing device 10 has a crawling function that crawls websites by following links and copies and saves information on web pages and DNS records.
[0026] As shown in FIG. 1, the processing device 10 includes a seed selection unit 11 (selection unit), a first extraction unit 12 , a second extraction unit 13 , a generation unit 14 , and a determination unit 15 .
[0027] The seed selection unit 11 selects reliable websites as seeds and creates a seed list showing the selected websites.
[0028] The first extraction unit 12 performs a web crawl on the websites shown in the seed list, and identifies and extracts domain names externally linked from the websites selected as seeds, along with the web content obtained by the access of this web crawl.
[0029] The second extraction unit 13 performs additional DNS crawling on domain names discovered through external links from seeds, and refers to and extracts DNS records that cannot be obtained by web crawling alone.
[0030] The generation unit 14 identifies trustworthy sites based on the characteristics of information extracted by web crawling and / or the characteristics of information extracted by DNS crawling, and generates an allow list indicating trustworthy sites.
[0031] The generator 14 generates and outputs a list of the identified domain names as an allow list. The allow list is a list of sites that have been determined to be trustworthy in advance based on specific criteria. Sites on the allow list are excluded from malicious judgments and are considered to be non-malicious sites (legitimate sites, benign sites).
[0032] The determination unit 15 determines the timing at which the first extraction unit 12, the second extraction unit 13, and the generation unit 14 re-execute these procedures.
[0033] [Seed Selection Unit] A description will be given of the seed creation process executed by the seed selection unit 11. The seed selection unit 11 selects a limited number of reliable websites as seeds, and creates a seed list indicating the selected seeds.
[0034] The seed selection unit 11 selects reliable websites by, for example, selecting the official websites of listed companies. The seed selection unit 11 lists listed companies in each country's market and selects the official pages of the listed companies as seeds.
[0035] This method is based on the following assumptions: Listed companies have a legal obligation to disclose information. While there are multiple ways to disclose information, most companies generally provide investor relations (IR) information through a website. For this reason, the assumption is that listed company websites are relatively trustworthy in the online space. However, it should be noted that even if a website belongs to a listed company, there is still a chance that it may be tampered with or not be adequately maintained.
[0036] The seed selected by the seed selection unit 11 is merely the starting point for subsequent processing. The decision as to whether or not to select each domain name for the final allow list is made anew by subsequent functional configuration units. Therefore, the seed selection unit 11 does not need to select websites that are 100% trustworthy.
[0037] Here are two specific examples of selecting listed companies. The first is to use a list such as the "Forbes Global 2000" published annually by Forbes (registered trademark). The Forbes Global 2000 is a list of the top 2,000 companies ranked based on various criteria such as sales, net assets, market capitalization, and profits as indicators of company size, and includes many of the world's listed companies.
[0038] The second method is to select companies listed on the securities markets of each country. For example, in Japan, you can select companies listed on the Tokyo Stock Exchange. Similarly, you can select companies listed on stock exchanges around the world.
[0039] In the above selection example, even if you select a listed company, the company name alone will not uniquely determine the URL or domain name of that company's official website.
[0040] Therefore, for example, when using a search engine to search for "company name" plus a string such as "official website," the page that appears first is selected as the URL of the official website.
[0041] In addition, in EC (Electronic Commerce) companies, etc., the top page for general customers (e.g., https: / / www.amazon[.]com / ) and the company homepage (e.g., https: / / www.aboutamazon[.]com / ) may be prepared as separate domain names, and this embodiment mainly assumes the use of the latter.
[0042] The seed list creation method is not limited to the above-mentioned method, and may be a method of manually selecting reliable sites or other methods.
[0043] [First Extraction Unit] The following describes the first extraction process executed by the first extraction unit 12. The first extraction unit 12 performs a web crawl on each URL included in the seed list created by the seed selection unit 11. The first extraction unit 12 extracts links included in web pages obtained by the web crawl, along with server certificate information and web content information, and stores them with a timestamp of when the web crawl was performed.
[0044] Specifically, the first extraction unit 12 extracts links by repeatedly crawling the web recursively up to a specified hierarchy (depth) based on the site selected as a seed. The first extraction unit 12 determines whether the extracted link is an external link based on the domain name of the extracted link, and if the extracted link is an external link, extracts server certificate information and web content information.
[0045] The first extraction unit 12 performs either the first or second process for each extracted link, depending on its PLD (Pay Level Domain). A PLD is the smallest domain unit that users can directly register and consists of a subdomain of a public suffix (e.g., ".com" or "co.uk"). It should be noted that these terms are not necessarily used consistently within the industry. For example, a public suffix is sometimes called an effective top-level domain (eTLD). Other terms with the same meaning as PLD include registrable domain, eTLD+1, apex domain, and effective second (2nd) level domain (eSLD / e2LD).
[0046] The first process is a process in which, if the PLD of the linked site matches the PLD of the seed, the web crawling process is repeated recursively until a pre-specified hierarchy (depth) is reached, and web access is performed again to the linked site to extract the link.
[0047] The second process is a process in which, if the PLD of the linked site does not match the PLD of the seed, a single Web access is made to the linked site, the content (server certificate information and Web content information) is collected, and the extraction is completed.
[0048] Figure 2 shows an overview of what is collected during a web crawl and how it is processed. For example, in Figure 2, if the seed URL is "https: / / example[.]com" and its depth is set to 0, the link with the same PLD at the next depth (Depth 1), "https: / / example[.]com / about / ," is again selected as a link extraction target (first process). On the other hand, the external link with a mismatched PLD, "https: / / security[.]example / ," is accessed up to that URL but not beyond that page (second process).
[0049] [Second Extraction Unit] The second extraction process executed by the second extraction unit 13 will be described. The second extraction unit 13 performs an additional DNS crawl on all domain names discovered by the web crawl. The second extraction unit 13 performs a DNS crawl based on the sites selected as seeds and the domain names of the external links extracted by the first extraction unit 12, and extracts DNS records and the support status of DNS security mechanisms.
[0050] Specifically, the second extraction unit 13 first collects five types of common DNS records (NS, A, AAAA, MX, and TXT) for each domain name. If an A record or an AAAA record exists, the second extraction unit 13 also collects the country and organization corresponding to the IP address.
[0051] The second extraction unit 13 employs, for example, a method of using a known GeoIP database to associate IP addresses with corresponding countries and organizations.
[0052] Furthermore, the second extraction unit 13 verifies the extent to which each domain complies with seven DNS security mechanisms (DNSSEC (DNS Security Extension), CAA (Certification Authority Authorization), SPF (Sender Policy Framework), DKIM (DomainKeys Identified Mail), Domain-based Message Authentication, Reporting, and Conformance (DMARC), Mail Transfer Agent Strict Transport Security (MTA-STS), DNS-based authentication of named entities (DANE)), and collects the information.
[0053] The security mechanisms used in the DNS targeted in this embodiment are those mentioned above, but are not limited to these, as new protocols and methods (e.g., Brand Indicators for Message Identification (BIMI)) are proposed and gradually gain popularity every year.
[0054] [Generation Unit] The following describes the generation process executed by the generation unit 14. The generation unit 14 generates a permission list that lists domain names that are trusted at a certain point in time, using the processing results of the seed selection unit 11, the first extraction unit 12, and the second extraction unit 13. The generation unit 14 has a feature generation unit 141 and a determination unit 142.
[0055] The feature generator 141 generates a feature vector (feature) that indicates the characteristics of the domain name to be determined and the website related to that domain name. The feature generator 141 generates a feature vector that indicates the characteristics of information extracted by web crawling. And / or the feature generator 141 generates a feature vector that indicates the characteristics of information extracted by DNS crawling.
[0056] The determination unit 142 determines whether a website extracted by web crawling is trustworthy using a determination model 1421 (machine learning model). The determination model 1421 is trained so that when a feature vector indicating the characteristics of a domain name to be determined and a website related to the domain name is input, the determination model 1421 outputs a determination result as to whether the site of the domain name is trustworthy.
[0057] The determination unit 142 inputs the feature vector generated by the feature amount generation unit 141 to the determination model 1421. The determination unit 142 obtains the determination result for the domain name corresponding to the feature vector output by the determination model 1421, and thereby excludes domains determined to be unreliable domains.
[0058] [Processing of the Feature Vector Generation Unit] The feature generation unit 141 generates first to fifth feature vectors based on information extracted by web crawling and / or information extracted by DNS crawling. Figures 3 and 4 are diagrams explaining the first to fifth feature vectors.
[0059] [First Feature Vector] First, the first feature vector will be described. The first feature vector indicates the presence or absence of a DNS record as a result of a DNS crawl and the features of the content of the DNS record.
[0060] The feature generation unit 141 determines whether or not five common types of DNS records (NS, A, AAAA, MX, TXT) exist from the DNS records collected by the second extraction unit 13 through the DNS crawl, and if so, indicates the number of such records as an element of the first feature vector ((1) in Figure 3, 1 in (1) in Figure 4).
[0061] The feature generation unit 141 quantifies the country names and organization names corresponding to the IP addresses of the A record and AAAA record in the DNS records, and displays them as elements of the first feature vector (2 to 5 in (1) of Figure 4).
[0062] Furthermore, the feature generator 141 quantifies the degree to which the domain name complies with the seven DNS security mechanisms, and indicates this as an element of the first feature vector (6 in (1) of FIG. 4).
[0063] [Second Feature Vector] Next, the second feature vector will be described. The second feature vector indicates the features of the content of the server certificate information.
[0064] The feature generation unit 141 quantifies the four types of content set in the issuer and subject of the server certificate, i.e., what is set in the issuer's Organization and CountryName and the subject's Organization and CountryName ((2) in Figure 3), and displays them as elements of the second feature vector ((2) in Figure 4).
[0065] [Third Feature Vector] Next, the third feature vector will be described. The third feature vector indicates the characteristics of the title of the website extracted by web crawling and / or the context of the sentences set as the content of the website extracted by web crawling for the domain name to be determined. The third feature vector indicates the characteristics of the title context ((3) in Figure 3, (3) in Figure 4). The third feature vector is a feature amount related to what the title represents.
[0066] The third feature vector is generated to recognize the natural language context of the title of the website of the discovered domain name and to contribute to identifying other domain names that have websites with titles in similar contexts. For this purpose, the feature generator 141 uses the Sentence BERT multilingual model to generate the third feature vector.
[0067] Specifically, the feature generator 141 creates a sentence by concatenating the titles of all websites with domain names discovered when crawling a seed list at a specific time, and inputs the sentence into the Sentence BERT model. As a result, the feature generator 141 obtains a third feature vector with fixed dimensions (768 dimensions) that is independent of the number of words in the title.
[0068] Sentence BERT is a language processing model that uses the Transformer architecture to understand the meaning of sentences. Unlike simple word embedding vectors (e.g., word2vec), Sentence BERT can obtain feature vectors that take into account the relationships between sentences, and demonstrates high accuracy in various natural language processing tasks (e.g., semantic sentence similarity, semantic search, clustering).
[0069] [Fourth Feature Vector] Next, the fourth feature vector will be described. The fourth feature vector indicates the features of the document context of the link text set in the link source of the backlink when it linked to the website extracted by web crawling. The fourth feature vector indicates the features of the link text context ((4) in Figure 3, (4) in Figure 4). The fourth feature vector is a feature quantity related to what the link text represents.
[0070] Here, the feature generation unit 141 obtains a fourth feature vector using Sentence BERT, similar to the third feature vector generation process, in order to identify the context of the string set as link text at the link source when linking to a domain name discovered by web crawling.
[0071] Specifically, the feature generator 141 creates a sentence by concatenating link texts from seed websites linked to the target domain name, and obtains a Sentence BERT feature vector. As a result, the feature generator 141 obtains an additional 768-dimensional feature vector as a fourth feature vector, which is independent of the third feature vector.
[0072] [Fifth Feature Vector] Next, the fifth feature vector will be described. The fifth feature vector indicates the number of websites that linked to the website extracted by web crawling for the domain name to be judged, and the features of the domain names from which the backlinks originate. The fifth feature vector quantifies how many backlinks there are from which domain names, and indicates these as elements of the fifth feature vector.
[0073] The feature generator 141 generates a feature vector that indicates the number of backlinks linking to the website of the target domain name and the characteristics of the domain name.
[0074] For example, in the case of Figure 3, there is only one backlink linking to the website of the target domain name ((5) in Figure 3), so the feature generator 141 constructs a feature vector such that "backlink from example.com = 1" and "backlink from other domains = 0" ((5) in Figure 4).
[0075] The above five categories and the feature vectors that actually constitute them are merely examples, and the feature generating unit 141 may generate feature vectors of other categories that are constructed using other methods.
[0076] [Processing of Determination Unit] Then, using the first to fifth feature amounts generated by the feature amount generation unit 141, unreliable domain names are identified and excluded using a supervised machine learning approach.
[0077] Generally, supervised machine learning involves creating a machine learning model using correct labels and feature vectors, and then applying the machine learning model to new data.
[0078] In this embodiment, when assigning correct answer labels, the conventional design of assigning binary labels of "unreliable (positive)" and "reliable (negative)" in problem setting is not adopted.
[0079] For example, in conventional technology, it has been reported that when machine learning is performed on domain names, using a top list as a method for labeling trustworthy sites can achieve very high accuracy. This is because there is a large difference in the feature distribution between extremely popular sites (popular and trustworthy) and unpopular sites (unpopular and untrustworthy).
[0080] In contrast to this, the present embodiment is different from the conventional method in that it aims to identify unpopular but reliable domain names.
[0081] If we were to rely on top lists to label domain names as trustworthy, as in conventional technology, the model would be biased toward the characteristics of the top lists, and would not be able to output a machine learning model suitable for identifying unpopular but trustworthy domains.
[0082] Therefore, in order to solve this problem, in this embodiment, for example, PU (Positive-Unlabeled) learning is utilized.
[0083] PU learning is a technique that uses both positive and unlabeled data to train supervised machine learning models.
[0084] In this embodiment, the positive data in the training data is data labeled as "unreliable domain names." The unlabeled data in the training data consists of both "reliable domain names and unreliable domain names."
[0085] By using PU learning, it is possible to train a machine learning model (decision model 1421) that is not biased towards the features of the top list, making it possible to identify unpopular but reliable domains.
[0086] Furthermore, by using PU learning, when labeling, results that have already been determined to be unreliable by other mechanisms can be directly used, thereby avoiding the task of manually labeling correct (unreliable) data from scratch.
[0087] The determination unit 142 calculates the probability that a discovered domain name is untrustworthy using the determination model 1421, and determines that a domain name with a probability exceeding a specified threshold is untrustworthy. Note that this threshold can be adjusted to any value.
[0088] [Determination Unit] The processing of the determination unit 15 will now be described. Simply generating the allow list is not sufficient; it must be updated so that domain names that are no longer trustworthy are no longer included in the allow list. To achieve this, the first extraction process, the second extraction process, and the generation process must be re-executed periodically (for example, once a week).
[0089] Therefore, the determination unit 15 determines the timing for re-executing the first extraction process, the second extraction process, and the generation process.
[0090] Ideally, it would be desirable to re-execute all of the processes, including the first extraction process, the second extraction process, and the generation process, every time the seed selection unit 11 updates each website in the seed list. However, continuously monitoring the updates of all websites without exception would require an extremely large amount of calculation and access to each website, which would be a heavy burden and would be unrealistic.
[0091] The determination unit 15 determines the timing of re-execution for the extraction of external links by web crawling, the extraction of server certificate information, and the extraction of DNS records among the first extraction process, the second extraction process, and the generation process, according to a predetermined re-execution algorithm.
[0092] This re-execution algorithm is based on the assumption in the embodiment that website update information required to create the allow list consists of content updates, server certificate updates, and / or DNS record updates, and that it is not necessary to recursively retrieve all web content each time. This is because the purpose of the embodiment is not to thoroughly index the latest web content like a search engine, but to collect information necessary to immediately remove domain names that were once included in the allow list but are no longer valid as of the latest time.
[0093] [Processing Procedure] FIG. 5 is a flowchart showing the processing procedure executed by the processing device 10 shown in FIG.
[0094] The seed selection unit 11 performs a seed selection process to select reliable websites as seeds and create a seed list showing the selected websites (step S1).
[0095] The first extraction unit 12 performs a first extraction process (step S2) in which it performs a web crawl on the websites shown in the seed list and identifies and extracts domain names externally linked from the websites selected as seeds, along with the web content obtained by the access of this web crawl.
[0096] The second extraction unit 13 performs additional DNS crawling on the domain names discovered through external links from the seed, and performs a second extraction process to reference and extract DNS records that cannot be obtained by web crawling alone (step S3).
[0097] The generation unit 14 performs a generation process using the judgment model 1421 to exclude domain names that are not trustworthy at that time, identify trustworthy domain names, and generate and output a list showing the identified domain names as an allowance list (step S4).
[0098] The decision unit 15 performs a decision process to determine the timing for the first extraction unit 12, the second extraction unit 13, and the generation unit 14 to re-execute web crawling, extraction of server certificate information, and extraction of DNS records (step S5).
[0099] [Generation Processing] FIG. 6 is a flowchart showing the processing procedure of the generation processing shown in FIG.
[0100] The generation unit 14 acquires information about the domain name to be judged, specifically, information extracted by web crawling and / or information extracted by DNS crawling (step S11).
[0101] The generating unit 14 generates first to fifth feature vectors based on the information extracted by the web crawl and / or the information extracted by the DNS crawl (step S12).
[0102] The generation unit 14 inputs the first to fifth feature vectors generated in step S12 into the judgment model 1421 to obtain the probability of unreliability, and performs a judgment process to judge domain names that exceed a specified threshold as unreliable (step S13).
[0103] If there is a next domain name to be determined (step S14: Yes), the generation unit 14 returns to step S11, acquires each piece of information related to the next domain name to be determined, and performs the generation and determination process of a feature vector.
[0104] If there is no next domain name to be judged (step S14: No), a list indicating domain names judged to be trustworthy based on the judgment result of step S13 is generated as an allowance list and output (step S15).
[0105] [Re-execution Algorithm] FIG. 7 is a flowchart showing the processing procedure of the re-execution algorithm executed by the decision unit 15.
[0106] The determination unit 15 first performs the following determination on the domain names listed in the seed list created in step S1: The determination unit 15 determines whether L days have passed since the last web crawl for each domain name (step S21).
[0107] If there is a domain name (first domain name) for which L days have passed since the last time a web crawl was performed (step S21: Yes), the determination unit 15 causes the first extraction unit 12 to re-execute the extraction of external links (first external links) (step S22) by re-executing a web crawl for this first domain name.
[0108] If there is no domain name (first domain name) for which L days have passed since the last time the web crawl was performed (step S21: No), or after the processing of step S22, the determination unit 15 performs the following processing for each domain name of the first external link extracted when the seed list was web crawled: First, the determination unit 15 determines whether C days have passed since the last time the server certificate information for each domain name of the first external link was extracted (step S23).
[0109] If there is no domain name (second domain name) for which C days have passed since the last time the server certificate information was extracted (step S23: No), the determination unit 15 determines for each domain name of the first external link whether the expiration date of the server certificate information extracted last time is within one week (step S24).
[0110] If there is a second domain name for which C days have passed since the last time the server certificate information was extracted (step S23: Yes), or if there is a domain name (third domain name) for which the expiration date of the last extracted server certificate information is within one week (step S24: Yes), the determination unit 15 causes the first extraction unit 12 to re-execute the extraction of server certificate information for the domain name (second domain name or third domain name) of this external link (step S25).
[0111] If there is no domain name (third domain name) within one week of the expiration date of the previously extracted server certificate (step S24: No), or after processing in step S25, the determination unit 15 determines whether D days have passed since the last time DNS record extraction for each domain name of the first external link was performed (step S26).
[0112] If there is a domain name (fourth domain name) for which D days have passed since the last DNS record extraction (step S26: Yes), the determination unit 15 causes the second extraction unit 13 to re-extract the DNS record for this fourth domain name (step S27).Then, the generation unit 14 updates the permission list based on the processing results of steps S21 to S27 (step S28).
[0113] In this case, in the embodiment, by setting the parameters so that L>>C≧D, the frequency of recursive retrieval from websites can be reduced, and the information required for the permission list can be collected with a realistic amount of calculation and access.
[0114] In the above description, steps S22, S25, and S27 are described as processing procedures that are executed only for the domain names of the first external links extracted in step S21 that correspond to the predetermined determination results in steps S23, S24, and S26, but this is not limiting. Steps S22, S25, and S27 may be executed for all domain names of the first external links extracted in step S21.
[0115] [Effects of the embodiment] As described above, the processing device 10 according to the embodiment selects a small number of potentially trustworthy websites as seeds, performs a web crawl on these seed sites, and extracts external links. The processing device 10 performs a DNS crawl to extract DNS records based on the sites selected as seeds and the domain names of the external links extracted by the web crawl. The processing device 10 then identifies trustworthy sites and generates an allow list based on the characteristics of the information extracted by the web crawl and / or the characteristics of the information extracted by the DNS crawl.
[0116] At this time, the processing device 10 performs machine learning using features related to the web content and DNS records of external links discovered during the crawl, thereby eliminating unreliable domain names.
[0117] In this way, the processing device 10 utilizes the link relationships that form the backbone of the Web space and characteristic information in the network structure, and also uses information on a small number of known unreliable domain names as labels to identify and eliminate unreliable domain names from among domain names that have not been previously determined.
[0118] This enables the processing device 10 to correctly identify "unpopular but legitimate sites" that are overlooked in the top list, and to identify malicious sites with higher accuracy and fewer false positives, and to control access to malicious sites.
[0119] The processing device 10 also determines the timing for retrying the processes of web crawling, server certificate extraction, and DNS record collection according to a predetermined retry algorithm. In this way, the processing device 10 continuously executes the process for identifying trusted sites, thereby enabling the generation of an updated allow list.
[0120] The processing device 10 may make the determination using only some types of feature vectors among the first to fifth feature vectors.
[0121] For example, even if a DNS record is set, if the website is not up and running, the website content is not set. In such a case, the processing device 10 cannot acquire the third feature vector and the fourth feature vector. Therefore, the processing device 10 performs a determination using, for example, only the first feature vector and the second feature vector. In this case, the processing device 10 provides a number of determination models 1421 corresponding to each combination of input feature vectors, and trains each determination model using training data corresponding to each combination of input feature vectors.
[0122] For example, examples of combinations of feature vectors include a combination of the first and second feature vectors, a combination of the third and fourth feature vectors, and only the fifth feature vector. In this case, three determination models 1421 are provided, one for each of the combination of the first and second feature vectors, the combination of the third and fourth feature vectors, and only the fifth feature vector.
[0123] The processing device 10 may use all elements of each feature vector or only some elements of the feature vector to make the determination. For example, the processing device 10 may use only some elements of all feature vectors, only some elements of the first and second feature vectors, only some elements of the third and fourth feature vectors, or only some elements of the fifth feature vector.
[0124] In this case, the determination model 1421 performs learning using all feature vectors with all elements as training data, and sets the average value for each element of the feature vector as the default value. If some elements of all feature vectors are not input, the determination model 1421 considers the elements to be missing and calculates the probability that the elements are unreliable.
[0125] Furthermore, the processing device 10 may perform a determination by providing the determination models 1421 in the number corresponding to the combination of elements of the input feature vector. The processing device 10 may use different determination models 1421 depending on the type of the input feature vector and the combination of elements.
[0126] [Example of Labeling Method] Labeling of "unreliable domain names" is performed using the following method. Labeling may be performed by the processing device 10 or by another device. In this example, a case where the processing device 10 executes the labeling will be described as an example. For example, the processing device 10 applies a new ring model to each domain name extracted when crawling URLs on the web, determines whether the domain name is unreliable at that time, and performs labeling.
[0127] Here, we propose a new ring model, a hierarchical ring model that expresses the relationships between multiple types of resources related to the Web and domain names.
[0128] 8 is a diagram illustrating a hierarchical ring model. As shown in Fig. 8, the hierarchical ring model 100 arranges, from the innermost to the outermost, the state of the domain name at the time of crawling (A), changes in DNS records since the previous crawling (B), the status of the server certificate at the time of crawling (C), the acquisition status of web content information at the time of crawling (D), and changes in web backlinks since the previous crawling (E).
[0129] The processing device 10 acquires the above information (A) to (E) for the domain name to be judged. The processing device 10 associates the acquired information (A) to (E) with the arrangement of each ring in the ring model 100 and expresses it in a hierarchical ring model structure.
[0130] When an inner ring (for example, (A)) becomes untrusted, the processing device 10 treats the entire ring for this domain name as being untrusted even if an outer ring (for example, (B)) is trusted, and determines that this domain name is untrusted at that point.
[0131] The processing device 10 outputs, as a permission list, only domain names that have passed through all rings of this ring model 100. In other words, the processing device 10 eliminates domain names that are not trustworthy at that time based on the hierarchical information (A) to (E). Note that the ring model 100 shown here is an example, and a ring model in which the indexes used as the ring hierarchy and the number of rings are further expanded may also be used.
[0132] The result of each ring is binary (either "unreliable" or "untrusted"). However, "unreliable" does not equal "trustworthy," meaning that the ring model alone cannot determine reliability.
[0133] Specifically, the reliability assessment of a domain name using this ring model is performed in the following order: a first assessment process regarding the state of the domain name at the time of crawling (A); a second assessment process regarding changes in the DNS record since the previous crawl (B); a third assessment process regarding the state of the server certificate at the time of crawling (C); a fourth assessment process regarding the acquisition status of web content information at the time of crawling (D); and a fifth assessment process regarding changes in web backlinks since the previous crawl (E). The process ends when the domain name becomes untrustworthy.
[0134] In this way, the processing device 10 hierarchically organizes information regarding the status and temporal changes of multiple types of resources related to websites, domain names, and / or DNS records obtained by web crawling for the domain name to be judged, with information that is more trustworthy at the top, and references the information from the top down to gradually judge whether the site is trustworthy or not.
[0135] [First Determination Process] In the first determination process, if domain parking is being used at the time of web crawling or if there is no NS record for the domain name (NXDOMAIN), the processing device 10 determines that the domain name is not trustworthy.
[0136] Domain parking is the practice of registering domain names and directing them to web pages containing advertisements or links to other websites, primarily for the purpose of generating revenue. A parked domain name or an abandoned domain name is considered unreliable at the time, even if it was reliable in the past.
[0137] To determine whether a domain name is using domain parking, for example, you can check whether the domain name's NS record belongs to a known domain parking provider. You can also determine whether a domain name is using domain parking based on the content, content, and appearance of each web page.
[0138] [Second Determination Process] In the second determination process, the processing device 10 determines that a domain name is not trustworthy if the combination of DNS NS records is different between the previous crawl and the current crawl, based on multiple results obtained by performing a re-crawl.
[0139] A change in NS record means that the server pointing to the domain record has changed. Of course, this may be done for legitimate reasons (such as moving a server), but we believe that the reliability of the change needs to be re-evaluated the next time the site is re-crawled.
[0140] [Third Determination Process] In the third determination process, when extracting links by web crawling, if an expired server certificate is used even though the domain name is extracted as an HTTPS (Hyper Text Transfer Protocol Secure) link, the processing device 10 determines that the domain name is untrustworthy. This is because if the domain name is extracted as an HTTPS link but uses an expired server certificate, it is likely that the server certificate is not properly managed or that the domain name will be abandoned in the future, and therefore is untrustworthy.
[0141] In the fourth determination process, if the processing device 10 cannot obtain a web page due to a connection error in HTTPS or HyperText Transfer Protocol (HTTP) during web crawling, the processing device 10 determines that the domain name is unreliable. While it is possible that the server is undergoing maintenance during web crawling or that only specific users can access the site, in this embodiment, safety is emphasized and such sites are not included in the final allow list.
[0142] [Fifth Determination Process] In the fifth determination process, if a re-crawl is performed and multiple results are found, the processing device 10 determines that the site is untrustworthy if the current number of backlinks is less than a predetermined percentage of the previous number. If the current number of backlinks is less than a predetermined percentage of the previous number, this indicates that links have been intentionally removed from many corporate sites that make up the seed list. The fact that a site on the seed list intentionally removed external links suggests the need to reevaluate its content. Note that the predetermined percentage (e.g., 50%) can be any value.
[0143] [Labeling Processing Procedure] FIG. 9 is a flowchart showing the labeling processing procedure.
[0144] The processing device 10 acquires information about the domain name to be judged (step S31). The information about the domain name to be judged is information about the status and temporal changes of multiple types of resources related to the website and / or domain name acquired by the web crawl in the processing of step S4. The information about the domain name to be judged also includes information about the DNS record acquired by the DNS crawl in step S5.
[0145] The processing device 10 expresses each piece of acquired information relating to the domain name to be judged in a hierarchical ring model structure in association with the arrangement of each ring in the ring model (see FIG. 3) (step S32).
[0146] Next, the processing device 10 determines whether the domain name to be determined is trustworthy from the inner ring of the ring model.
[0147] Specifically, the processing device 10 performs a first determination process (step S33) regarding the state of the domain name at the time of crawling (A), a second determination process (step S34) regarding changes in the DNS record since the previous crawling (B), a third determination process (step S35) regarding the state of the server certificate at the time of crawling (C), a fourth determination process (step S36) regarding the acquisition status of web content information at the time of crawling (D), and a fifth determination process (step S37) regarding changes in web backlinks since the previous crawling (E).
[0148] If there is a next domain name to be determined (step S38: Yes), the processing device 10 returns to step S31. Then, the processing device 10 acquires each piece of information related to the next domain name to be determined, expresses it in a ring model structure, and then performs the first to fifth determination processes.
[0149] If there is no next domain name to be judged (step S38: No), the processing device 10 labels the domain name that it judges to be unreliable based on the judgment results of steps S33 to S37 as an "unreliable domain name" and outputs the labeling data (step S39).
[0150] [First Determination Process] FIG. 10 is a flowchart showing the processing procedure of the first determination process shown in FIG.
[0151] The processing device 10 refers to information (A) regarding the status of the domain name at the time of crawling, which corresponds to the innermost ring of the ring model (step S41), and determines whether domain parking is being used at the time of web crawling (step S42).
[0152] If domain parking is not being used at the time of web crawling (step S42: No), the processing device 10 determines whether an NS record for the domain name exists (step S43).
[0153] If an NS record for the domain name exists (step S43: Yes), the processing device 10 determines that the domain name to be judged is trustworthy (step S44).
[0154] On the other hand, if domain parking is being used at the time of web crawling (step S42: Yes), or if there is no NS record for the domain name (step S43: No), the processing device 10 determines that the domain name being judged is not trustworthy (step S45).
[0155] [Second Determination Process] FIG. 11 is a flowchart showing the processing procedure of the second determination process shown in FIG.
[0156] The processing device 10 determines whether the determination result for the domain name to be determined in the first determination process in the previous stage was that the domain name is trustworthy or untrustworthy (step S51).
[0157] If the determination is that the information is not trustworthy (step S51: not trustworthy), the processing device 10 ends the second determination process.
[0158] If it is determined that the server is trustworthy (step S51: trustworthy), the processing device 10 performs the following process. First, the processing device 10 refers to information about the DNS records of the previous crawl and the current crawl to determine changes (B) in the DNS records since the previous crawl, which corresponds to the second ring from the inside of the ring model (step S52).
[0159] The processing device 10 determines whether the combination of DNS NS records is different between the previous crawl and the current crawl (step S53).
[0160] If the combination of DNS NS records is the same between the previous crawl and the current crawl (step S53: No), the processing device 10 determines that the domain name to be judged is trustworthy (step S54).
[0161] On the other hand, if the combination of DNS NS records is different between the previous crawl and the current crawl (step S53: Yes), the processing device 10 determines that the domain name being judged is not trustworthy (step S55).
[0162] [Third Determination Process] FIG. 12 is a flowchart showing the processing procedure of the third determination process shown in FIG.
[0163] The processing device 10 determines whether the determination results for the domain name to be determined in the first and second determination processes in the previous stages were that the domain name is trustworthy or untrustworthy (step S61).
[0164] If the determination is that the information is not trustworthy (step S61: not trustworthy), the processing device 10 ends the third determination process.
[0165] If the determination is that the server is trustworthy (step S61: Trustworthy), the processing device 10 refers to information regarding the state (C) of the server certificate at the time of crawling, which corresponds to the third ring from the inside of the ring model (step S62).
[0166] Then, the processing device 10 determines whether the server certificate extracted during the web crawl is an expired server certificate (step S63).
[0167] If the server certificate has not expired (step S63: No), the processing device 10 determines that the domain name to be judged is trustworthy (step S64).
[0168] On the other hand, if the server certificate has expired (step S63: Yes), the processing device 10 determines that the domain name to be judged is not trustworthy (step S65).
[0169] [Fourth Determination Process] FIG. 13 is a flowchart showing the processing procedure of the fourth determination process shown in FIG.
[0170] The processing device 10 determines whether the determination results for the domain name to be determined in the first to third determination processes in the previous stages were that the domain name is trustworthy or untrustworthy (step S71).
[0171] If it is determined that the information is not trustworthy (step S71: not trustworthy), the processing device 10 ends the fourth determination process.
[0172] If it is determined to be trustworthy (step S71: Trustworthy), the processing device 10 refers to information regarding the acquisition status (D) of web content information during crawling, which corresponds to the second ring from the outside of the ring model (step S72).
[0173] The processing device 10 determines whether or not a web page has been acquired during the web crawl (step S73).
[0174] If the web page can be acquired (step S73: No), the processing device 10 determines that the domain name to be judged is trustworthy (step S74).
[0175] On the other hand, if the web page cannot be acquired (step S73: Yes), the processing device 10 determines that the domain name to be judged is not reliable (step S75).
[0176] [Fifth Determination Process] FIG. 14 is a flowchart showing the processing procedure of the fifth determination process shown in FIG.
[0177] The processing device 10 determines whether the determination results for the domain name to be determined in the first to fourth determination processes in the previous stages were that the domain name is trustworthy or untrustworthy (step S81).
[0178] If it is determined that the information is not trustworthy (step S81: not trustworthy), the processing device 10 ends the fifth determination process.
[0179] If the determination is that the site is trustworthy (step S81: Trustworthy), the processing device 10 refers to information regarding changes in web backlinks (E) since the previous crawl, which corresponds to the outermost ring of the ring model (step S82).
[0180] The processing device 10 determines whether the current number of backlinks is equal to or less than a predetermined ratio of the previous number (step S83).
[0181] If the current number of backlinks is not equal to or less than the predetermined ratio of the previous number (step S83: No), the processing device 10 determines that the domain name to be judged is trustworthy (step S84).
[0182] On the other hand, if the current number of backlinks is equal to or less than the predetermined ratio of the previous number (step S83: Yes), the processing device 10 determines that the domain name being judged is not trustworthy (step S85).
[0183] [System Configuration, etc.] The components of each device shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific distribution and integration of each device is not limited to those shown in the figure, and all or part of the devices can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a central processing unit (CPU) and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic. The program may be executed not only by the CPU but also by other processors such as a graphics processing unit (GPU).
[0184] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.
[0185] [Program] In one embodiment, the processing device 10 can be implemented by installing a processing program that executes the above-described processes as package software or online software on a desired computer. For example, by having an information processing device execute the above-described processing program, the information processing device can function as the processing device 10. The information processing device referred to here includes desktop and notebook personal computers. Furthermore, other information processing devices also include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants).
[0186] The processing device 10 may also be implemented as a server device that provides services related to the above-described processing to a client terminal device used by a user. For example, the server device may be implemented as a server device that provides a service that outputs an authorization list. In this case, the server device may be implemented as a web server or as a cloud that provides services related to the above-described processing through outsourcing.
[0187] 15 is a diagram showing an example of a computer that executes a processing program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.
[0188] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.
[0189] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, application programs 1092, program modules 1093, and program data 1094. That is, the programs that define each process of the processing device 10 are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configuration of the processing device 10 are stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD (Solid State Drive).
[0190] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.
[0191] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.
[0192] REFERENCE SIGNS LIST 10 Processing device 11 Seed selection unit 12 First extraction unit 13 Second extraction unit 14 Generation unit 15 Determination unit 141 Feature generation unit 142 Determination unit 1421 Determination model
Claims
1. A processing device comprising: a selection unit that selects a trusted site as a seed; a first extraction unit that performs a web crawl multiple times to extract external links from the site selected as the seed; a second extraction unit that performs a DNS crawl to extract DNS records based on the site selected as the seed and the domain names of the external links extracted by the first extraction unit; and a generation unit that identifies trusted sites based on characteristics of information extracted by the web crawl and / or characteristics of information extracted by the DNS crawl, and generates an allow list indicating the trusted sites.
2. The processing device described in claim 1, characterized in that the first extraction unit extracts links by repeating web crawling recursively up to a specified hierarchy based on the site selected as the seed, determines whether the extracted link is an external link or not based on the domain name of the extracted link, and if the extracted link is an external link, extracts server certificate information and web content information; and the generation unit generates, based on the information extracted by the web crawl and the information extracted by the DNS crawl, a first feature indicating the presence or absence of a DNS record extracted by the DNS crawl and characteristics of the contents of the DNS record for the domain name to be determined, and a second feature indicating characteristics of the contents of the server certificate information, and determines whether the domain name to be determined is trustworthy based on features including part or all of the first feature and the second feature.
3. The processing device described in claim 1, characterized in that the generation unit generates, based on the information extracted by the Web crawl, a third feature indicating characteristics of the context of the text set as the title of the website obtained by the Web crawl and / or the content of the website extracted by the Web crawl for the domain name to be judged, and a fourth feature indicating characteristics of the context of the document of the link text set at the link source of the backlink when the website extracted by the Web crawl was linked to, and judges whether the domain name to be judged is trustworthy based on features including the third feature and part or all of the fourth feature.
4. The processing device described in claim 1, characterized in that the generation unit generates a fifth feature for the domain name to be judged based on the information extracted by the Web crawl, the fifth feature indicating the number of websites that linked to the websites extracted by the Web crawl and the characteristics of the domain name from which the backlink originates, and judges whether the domain name to be judged is trustworthy based on a feature including part or all of the fifth feature.
5. The processing device described in claim 1, characterized in that the generation unit, when input with features indicating the characteristics of the domain name to be judged and the website related to the domain name, uses a machine learning model that outputs a judgment result as to whether the site of the domain name is trustworthy or not to judge whether the website extracted by the web crawl is trustworthy or not.
6. A processing method executed by a processing device, comprising: a selection step of selecting a trusted site as a seed; a first extraction step of performing a web crawl multiple times to extract external links from the site selected as the seed; a second extraction step of performing a DNS crawl to extract DNS records based on the site selected as the seed and the domain names of the external links extracted in the first extraction step; and a generation step of identifying trusted sites and generating an allowance list indicating the trusted sites based on characteristics of the information extracted by the web crawl and / or characteristics of the information extracted by the DNS crawl.
7. A processing program for causing a computer to execute the following steps: a selection step of selecting a trusted site as a seed; a first extraction step of performing a web crawl multiple times to extract external links from the site selected as the seed; a second extraction step of performing a DNS crawl to extract DNS records based on the site selected as the seed and the domain names of the external links extracted in the first extraction step; and a generation step of identifying trusted sites and generating an allowance list indicating the trusted sites based on characteristics of information extracted by the web crawl and / or characteristics of information extracted by the DNS crawl.
Citation Information
Patent Citations
Information processing device, information processing method, and information processing program
JP2020170478A
Indicating website reputations during user interactions
US20140331119A1
Device for calculating maliciousness of communication destination, method for calculating maliciousness of communication destination, and program for calculating maliciousness of communication destination
WO2016140038A1
