Processing device, processing method, and processing program
The processing device improves malicious site identification and access control by using a ring model to generate a permission list from reliable seed sites, addressing the issue of misjudging 'unpopular but legitimate' websites as malicious and enhancing detection precision.
Patent Information
- Application Number
- PCT/JP2024/019789
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-05-29
- Publication Date
- 2024-12-05
AI Technical Summary
Existing methods for identifying malicious sites often misjudge 'unpopular but legitimate' websites as malicious due to their exclusion from popularity-based top lists, leading to false positives and difficulties in distinguishing between legitimate and phishing sites.
A processing device and method that selects reliable seed sites, performs multiple web crawls to extract external links and DNS records, and uses a ring model to generate a permission list that correctly indicates trusted sites, thereby improving the accuracy of malicious site identification and access control.
This approach effectively reduces false positives by correctly identifying 'unpopular but legitimate' sites and enhancing the precision of malicious site detection, ensuring accurate access control.
Smart Images

Figure JP2024019789_05122024_PF_FP_ABST
Abstract
Description
Processing device, processing method, and processing program
[0001] The present invention relates to a processing device, a processing method, and a processing program.
[0002] An allowlist is a list of sites that have been pre-determined to be trustworthy based on certain criteria. Sites on the allowlist are excluded from malicious judgments and are considered to be non-malicious sites (legitimate sites, benign sites). An allowlist is also sometimes called a safelist or whitelist.
[0003] Allowlists are commonly used in web security to mitigate the problem of "false positives," where non-malicious websites are mistakenly identified as malicious.
[0004] However, the web is extremely vast and dynamically changing, making it difficult to manually select a comprehensive list of trusted sites and continually update and maintain an allow list.
[0005] As a result, "top lists" have become widely used in both industry and academia as de facto allow lists, based on the implicit assumption that "popularity is likely to be safe."
[0006] A top list is a ranking list of websites or domain names in order of popularity based on specific criteria. There are multiple top lists, and each one has significantly different definitions of the data used, definitions of "popularity," and calculation methods.
[0007] A representative and widely used top list is the Tranco Top Million List (Non-Patent Document 1). The Tranco Top Million List is a combination of the Alexa Top 1 Million, Cisco Umbrella 1 Million, and Majestic Million top lists. Alexa Top 1 Million is one of the most commonly used top lists and is determined based on the average number of daily visits and page views to a website. Cisco Umbrella 1 Million is a list of the most popular domain names ranked in order based on DNS observations in the DNS (Domain Name System) service provided by Cisco Umbrella (registered trademark). Majestic Million is a list of website popularity calculated based on web links provided by Majestic SEO (registered trademark).
[0008] The Secrank List (Non-Patent Document 2) is a new top list processing method that uses a voting algorithm based on the DNS request volume of each domain name. There is also the Chrome User Experience Report (CrUX), which ranks the popularity of websites based on data collected from users of the Google Chrome web browser.
[0009] Victor Le Pochat et al., “TRANCO: A Research-Oriented Top Sites Ranking Hardened Against Manipulation”, In: Proc. NDSS (2019).Qinge Xie, et al., “Building an Open, Robust, and Stable Voting-Based Domain Top List”, In: Proc. USENIX Security (2022).
[0010] There are two main examples of using top lists as de facto allow lists:
[0011] For example, let's look at the first example, which is used to remove false positives. To prevent false positives, one example is to use domain names from the Majestic top list as an allow list. Additionally, the open source threat intelligence platform MISP (Malware Information Sharing Platform) uses Alexa Top1k, Umbrella Top 20k, Majestic Top1k, and Tranco Top 10k as options for removing false positives.
[0012] Next, we will explain a second example of its use with benign labeling. Top lists are widely used when using machine learning in the security field, for example, to label domain names as "benign" or to exclude some domain names from analysis as "benign" in the analysis process. Furthermore, when detecting phishing sites, domain names listed at the top of the top list are often selected as "genuine" domains that are likely to be imitated.
[0013] However, because the Top List is a system that lists "popular" sites in order, there is a fundamental problem that "unpopular but good sites" are difficult to list. For example, sites of new companies, sites of new services of existing companies, and sites of services provided to specific regions are unlikely to be listed on the Top List.
[0014] If such sites are not listed, false positives will occur when identifying malicious sites or controlling (filtering) access to them, resulting in legitimate sites being mistakenly identified as malicious, or in the inability to distinguish between less popular legitimate sites and phishing sites.
[0015] The present invention has been made in consideration of the above, and aims to provide a processing device, processing method, and processing program that can generate an allowance list that correctly indicates ``unpopular but legitimate sites,'' thereby identifying malicious sites and improving the accuracy of access control to malicious sites.
[0016] In order to solve the above-mentioned problems and achieve the objectives, the processing device is characterized by having a selection unit that selects trusted sites as seeds, a first extraction unit that performs web crawls multiple times to extract external links from the sites selected as seeds, and a generation unit that identifies trusted sites based on the difference in the crawl results of the web crawls performed by the first extraction unit and / or the status of the crawl results of the web crawls, and generates an allow list indicating the trusted sites.
[0017] According to the present invention, it is possible to generate an allow list that correctly indicates "unpopular but legitimate sites," thereby improving the accuracy of identifying malicious sites and controlling access to malicious sites.
[0018] FIG. 1 is a diagram illustrating an example of the configuration of a processing device according to an embodiment. FIG. 2 is a diagram illustrating an overview of collection targets and processing during web crawling. FIG. 3 is a diagram illustrating a ring model. FIG. 4 is a flowchart illustrating the processing procedure of processing executed by the processing device illustrated in FIG. 1. FIG. 5 is a flowchart illustrating the processing procedure of the generation processing illustrated in FIG. 4. FIG. 6 is a flowchart illustrating the processing procedure of the first determination processing illustrated in FIG. 5. FIG. 7 is a flowchart illustrating the processing procedure of the second determination processing illustrated in FIG. 5. FIG. 8 is a flowchart illustrating the processing procedure of the third determination processing illustrated in FIG. 5. FIG. 9 is a flowchart illustrating the processing procedure of the fourth determination processing illustrated in FIG. 5. FIG. 10 is a flowchart illustrating the processing procedure of the fifth determination processing illustrated in FIG. 5. FIG. 11 is a flowchart illustrating the processing procedure of a re-execution algorithm executed by a determination unit. FIG. 12 is a diagram illustrating an example of a computer that executes a processing program.
[0019] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a processing apparatus, a processing method, and a processing program according to the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the following embodiments.
[0020] The processing device according to the embodiment generates a new allowed list starting from legitimate websites, rather than a top-down approach that relies on popularity.
[0021] The processing device selects a small number of potentially reliable websites as seeds. The processing device performs web crawling on these seed sites, and if external links are provided from the seed sites, extracts their domain names, web content, and DNS (Domain Name System) records. The processing device performs this web crawling multiple times at predetermined intervals.
[0022] The processing device then represents the differences and status of multiple web crawl results for domain names discovered during the web crawl as a ring model. The ring model has a structure in which information that is more likely to be a source of trust among information related to the status and temporal changes of multiple types of resources related to websites and / or domain names is placed in the inner ring. The processing device 10 eliminates unreliable domain names according to the ring model structure, ultimately generating the latest allow list at that time.
[0023] In this way, the processing device applies a ring model that can represent the link relationships that form the backbone of the Web space, temporal changes in network structure, and the reliability dependencies at multiple layers to eliminate unreliable domain names. This enables the processing device to correctly identify "unpopular but legitimate sites" that are overlooked in top lists, thereby identifying malicious sites with higher accuracy and fewer false positives, and controlling access to malicious sites.
[0024] [Processing Device] Next, the configuration of a processing device according to an embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a processing device according to an embodiment.
[0025] The processing device 10 according to the embodiment is a server device implemented by, for example, loading a predetermined program into a computer or the like including a ROM (Read Only Memory), a RAM (Random Access Memory), a CPU (Central Processing Unit), etc., and having the CPU execute the predetermined program. The processing device 10 also has a communication interface for transmitting and receiving various information to and from other devices connected via a network, etc. As will be described later, the processing device 10 has a crawling function that crawls websites via links and copies and saves information on web pages and DNS record information.
[0026] As shown in FIG. 1, the processing device 10 includes a seed selection unit 11 (selection unit), a first extraction unit 12 , a second extraction unit 13 , a generation unit 14 , and a determination unit 15 .
[0027] The seed selection unit 11 performs a seed selection process to select reliable websites as seeds and create a seed list showing the selected websites.
[0028] The first extraction unit 12 performs a first extraction process of crawling the websites shown in the seed list, identifying and extracting domain names externally linked from the websites selected as seeds, along with the web content obtained by the accesses of this web crawl. The first extraction unit 12 executes this web crawl multiple times at predetermined intervals.
[0029] The second extraction unit 13 performs a second extraction process in which an additional DNS crawl is performed on the domain name discovered through an external link from the seed, and the DNS record that cannot be obtained by web crawling alone is referenced and extracted.
[0030] The generation unit 14 identifies trustworthy sites and generates an allow list indicating trustworthy sites based on the difference in the crawl results of the Web crawl executed by the first extraction unit 12 and / or the status of the crawl results of the Web crawl. The allow list is a list of sites that have been determined in advance to be trustworthy based on specific criteria, and sites listed on the allow list are excluded from malicious judgments and are considered to be non-malicious sites (legitimate sites, benign sites).
[0031] The generation unit 14 identifies trustworthy sites by applying a ring model that represents the differences and situations of multiple web crawl results for domain names discovered during web crawling. The generation unit 14 applies the ring model to the domain name to be judged and identifies trustworthy domain names by eliminating untrustworthy domain names according to the ring model structure. The generation unit 14 performs a generation process to generate and output a list indicating the identified domain names as an allow list.
[0032] The determination unit 15 performs a determination process to determine the timing at which the first extraction unit 12, the second extraction unit 13, and the generation unit 14 will re-execute these procedures.
[0033] [Seed Selection Unit] A description will be given of the seed selection process executed by the seed selection unit 11. The seed selection unit 11 selects a limited number of reliable websites as seeds and creates a seed list indicating the selected seeds.
[0034] The seed selection unit 11 selects reliable websites by, for example, selecting the official websites of listed companies. The seed selection unit 11 lists listed companies in each country's market and selects the official pages of the listed companies as seeds.
[0035] This method is based on the following assumptions: Listed companies have a legal obligation to disclose information. While there are multiple ways to disclose information, most companies generally provide investor relations (IR) information through a website. For this reason, the assumption is that listed company websites are relatively trustworthy in the online space. However, it should be noted that even if a website belongs to a listed company, there is still a chance that it may be tampered with or not be adequately maintained.
[0036] The seed selected by the seed selection unit 11 is merely the starting point for subsequent processing. The decision as to whether or not to select each domain name for the final allow list is made anew by subsequent functional configuration units. Therefore, the seed selection unit 11 does not need to select websites that are 100% trustworthy.
[0037] Here are two specific examples of selecting listed companies. The first is to use a list such as the "Forbes Global 2000" published annually by Forbes (registered trademark). The Forbes Global 2000 is a list of the top 2,000 companies ranked based on various criteria such as sales, net assets, market capitalization, and profits as indicators of company size, and includes many of the world's listed companies.
[0038] The second method is to select companies listed on the securities markets of each country. For example, in Japan, you can select companies listed on the Tokyo Stock Exchange. Similarly, you can select companies listed on stock exchanges around the world.
[0039] In the above selection example, even if you select a listed company, the company name alone will not uniquely determine the URL (Uniform Resource Locator) or domain name of that company's official website.
[0040] Therefore, for example, when using a search engine to search for "company name" plus a string such as "official website," the page that appears first is selected as the URL of the official website.
[0041] In addition, in EC (Electronic Commerce) companies, the top page for general customers (e.g., https: / / www.amazon[.]com / ) and the company homepage (e.g., https: / / www.aboutamazon[.]com / ) may be prepared as separate domain names, and this embodiment mainly assumes the use of the latter.
[0042] The seed list creation method is not limited to the above-mentioned method, and may be a method of manually selecting reliable sites or other methods.
[0043] [First Extraction Unit] The following describes the first extraction process executed by the first extraction unit 12. The first extraction unit 12 performs a web crawl on each URL included in the seed list created by the seed selection unit 11. The first extraction unit 12 extracts links included in web pages obtained by the web crawl, along with server certificate information and web content information, and stores them with a timestamp of when the web crawl was performed.
[0044] Specifically, the first extraction unit 12 extracts links by repeatedly crawling the web recursively up to a specified hierarchy (depth) based on the site selected as a seed. The first extraction unit 12 determines whether the extracted link is an external link based on the domain name of the extracted link, and if the extracted link is an external link, extracts server certificate information and web content information.
[0045] The first extraction unit 12 performs either the first or second process for each extracted link, depending on its PLD (Pay Level Domain). A PLD is the smallest domain unit that users can directly register and consists of a subdomain of a public suffix (e.g., ".com" or "co.uk"). It should be noted that these terms are not necessarily used consistently within the industry. For example, a public suffix is sometimes called an effective top-level domain (eTLD). Other terms with the same meaning as PLD include registrable domain, eTLD+1, apex domain, and effective second (2nd) level domain (eSLD / e2LD).
[0046] The first process is a process in which, if the PLD of the linked site matches the PLD of the seed, the web crawling process is repeated recursively until a pre-specified hierarchy (depth) is reached, and web access is performed again to the linked site to extract the link.
[0047] The second process is a process in which, if the PLD of the linked destination does not match the PLD of the seed, a single Web access is made to the linked destination, the content (server certificate information and Web content information) is collected, and the extraction is completed.
[0048] Figure 2 shows an overview of what is collected during a web crawl and how it is processed. For example, in Figure 2, if the seed URL is "https: / / example[.]com" and its depth is set to 0, the link with the same PLD at the next depth (Depth 1), "https: / / example[.]com / about / ," is again selected as a link extraction target (first process). On the other hand, the external link with a mismatched PLD, "https: / / security[.]example / ," is accessed up to that URL but not beyond that page (second process).
[0049] [Second Extraction Unit] The second extraction process executed by the second extraction unit 13 will be described. The second extraction unit 13 additionally executes DNS crawling for all domain names discovered by the web crawling. The second extraction unit 13 executes DNS crawling based on the sites selected as seeds and the domain names of the external links extracted by the first extraction unit 12, and extracts DNS records and the support status of DNS security mechanisms.
[0050] Specifically, the second extraction unit 13 first collects five types of common DNS records (NS, A, AAAA, MX, and TXT) for each domain name. If an A record or an AAAA record exists, the second extraction unit 13 also collects the country and organization corresponding to the IP address.
[0051] The second extraction unit 13 employs, for example, a method of using a known GeoIP database to associate IP addresses with corresponding countries and organizations.
[0052] Furthermore, the second extraction unit 13 verifies the extent to which each domain complies with seven DNS security mechanisms (DNSSEC (DNS Security Extension), CAA (Certification Authority Authorization), SPF (Sender Policy Framework), DKIM (DomainKeys Identified Mail), Domain-based Message Authentication, Reporting, and Conformance (DMARC), Mail Transfer Agent Strict Transport Security (MTA-STS), DNS-based authentication of named entities (DANE)), and collects the information.
[0053] The security mechanisms used in the DNS targeted in this embodiment are those mentioned above, but are not limited to these, as new protocols and methods (e.g., Brand Indicators for Message Identification (BIMI)) are proposed and gradually gain popularity every year.
[0054] [Generation Unit] The following describes the generation process executed by the generation unit 14. The generation unit 14 uses the processing results of the seed selection unit 11, the first extraction unit 12, and the second extraction unit 13 to generate an authorization list that lists domain names that are trusted at a certain point in time.
[0055] Here, in the embodiment, a new ring model is proposed to identify and eliminate domain names that are not trustworthy at that time for each domain name extracted when the first extraction unit 12 web crawls the URLs of the seed list created by the seed selection unit 11.
[0056] A common ring model is the security architecture on a single system represented by a protection ring, which has been discussed since the 1970s.
[0057] In contrast to this, in the embodiment, a new hierarchical ring model is adopted to represent the relationships between multiple types of resources related to the Web and domain names, rather than the ring of a single system.
[0058] 3 is a diagram illustrating a ring model. As shown in Fig. 3, the ring model 100 according to the embodiment arranges, from the innermost to the outermost, the state of the domain name at the time of crawling (A), changes in DNS records since the previous crawling (B), the status of the server certificate at the time of crawling (C), the acquisition status of web content information at the time of crawling (D), and changes in web backlinks since the previous crawling (E).
[0059] The generation unit 14 acquires the above information (A) to (E) for the domain name to be judged. The generation unit 14 associates the acquired information (A) to (E) with the arrangement of each ring in the ring model 100 and expresses it in a hierarchical ring model structure.
[0060] When an inner ring (for example, (A)) becomes untrusted, the generation unit 14 treats the entire ring for this domain name as being untrusted even if an outer ring (for example, (B)) is trusted, and determines that this domain name is untrustworthy at that point.
[0061] The generation unit 14 outputs as a permission list only domain names that have passed through all rings of this ring model 100. In other words, the generation unit 14 eliminates domain names that are not trustworthy at that time based on the hierarchical information (A) to (E). Note that the ring model 100 shown here is an example, and a ring model in which the indexes used as the ring hierarchy and the number of rings are further expanded may also be used.
[0062] The result of each ring is binary (either "unreliable" or "untrusted"). However, "unreliable" does not equal "trustworthy," meaning that the ring model alone cannot determine reliability.
[0063] Specifically, the reliability assessment of a domain name using this ring model is performed in the following order: a first assessment process regarding the state of the domain name at the time of crawling (A); a second assessment process regarding changes in the DNS record since the previous crawl (B); a third assessment process regarding the state of the server certificate at the time of crawling (C); a fourth assessment process regarding the acquisition status of web content information at the time of crawling (D); and a fifth assessment process regarding changes in web backlinks since the previous crawl (E). The process ends when the domain name becomes untrustworthy.
[0064] In this way, the generation unit 14 hierarchically organizes information regarding the status and temporal changes of multiple types of resources related to websites, domain names, and / or DNS records obtained by web crawling for the domain name to be judged, with information that is more trustworthy at the top, and references the information from the top down to gradually judge whether the site is trustworthy or not.
[0065] [First Determination Process] In the first determination process, if domain parking is being used at the time of web crawling or if there is no NS record for the domain name (NXDOMAIN), the generation unit 14 determines that the domain name is not trustworthy.
[0066] Domain parking is the practice of registering domain names and directing them to web pages containing advertisements or links to other websites, primarily for the purpose of generating revenue. A parked domain name or an abandoned domain name is considered unreliable at the time, even if it was reliable in the past.
[0067] To determine whether a domain name is using domain parking, for example, you can check whether the domain name's NS record belongs to a known domain parking provider. You can also determine whether a domain name is using domain parking based on the content, content, and appearance of each web page.
[0068] [Second Determination Process] In the second determination process, the generation unit 14 determines that a domain name is not trustworthy if the combination of DNS NS records is different between the previous crawl and the current crawl, based on multiple results obtained by performing a re-crawl.
[0069] A change in NS record means that the server pointing to the domain record has changed. Of course, this may be done for legitimate reasons (such as moving a server), but we believe that the reliability of the change needs to be re-evaluated the next time the site is re-crawled.
[0070] [Third Determination Process] In the third determination process, when extracting links by web crawling, if an expired server certificate is used even though the domain name is extracted as an HTTPS (Hyper Text Transfer Protocol Secure) link, the generation unit 14 determines that the domain name is untrustworthy. This is because if the domain name is extracted as an HTTPS link but uses an expired server certificate, it is likely that the server certificate is not properly managed or that the domain name will be abandoned in the future, and therefore is untrustworthy.
[0071] [Fourth Determination Process] In the fourth determination process, if a web page cannot be acquired due to a connection error in HTTPS or HTTP (Hyper Text Transfer Protocol) during web crawling, the generation unit 14 determines that the domain name is not trustworthy. Although it is possible that the server is undergoing maintenance during web crawling or that only specific users can access the domain, in the embodiment, emphasis is placed on safety, and such sites are not included in the final allow list.
[0072] [Fifth Determination Process] In the fifth determination process, if a re-crawl is performed and multiple results are found, and the current number of backlinks is less than a predetermined percentage of the previous number, the generation unit 14 determines that the site is untrustworthy. If the current number of backlinks is less than a predetermined percentage of the previous number, this indicates that links have been intentionally removed from many corporate sites that make up the seed list. The fact that a site on the seed list intentionally removed external links suggests the need to reevaluate its content. Note that the predetermined percentage (e.g., 50%) may be any value.
[0073] [Determination Unit] The processing of the determination unit 15 will be described. The above-mentioned ring model 100 is designed to refer to multiple crawl results. For this reason, it is not sufficient to execute the first extraction process and the second extraction process just once. In order to eliminate unreliable domain names as expected in the ring model 100, it is necessary to continuously re-execute the first extraction process, the second extraction process, and the generation process at regular intervals (for example, one week).
[0074] Therefore, the determination unit 15 determines the timing for re-executing the first extraction process, the second extraction process, and the generation process.
[0075] Ideally, it would be desirable to re-execute all of the processes, including the first extraction process, the second extraction process, and the generation process, every time the seed selection unit 11 updates each website in the seed list. However, continuously monitoring the updates of all websites without exception would require an extremely large amount of calculation and access to each website, which would be a heavy burden and would be unrealistic.
[0076] The determination unit 15 determines the timing of re-execution for the extraction of external links by web crawling, the extraction of server certificate information, and the extraction of DNS records among the first extraction process, the second extraction process, and the generation process, according to a predetermined re-execution algorithm.
[0077] This re-execution algorithm is based on the assumption in the embodiment that website update information required to create the allow list consists of content updates, server certificate updates, and / or DNS record updates, and that it is not necessary to recursively retrieve all web content each time. This is because the purpose of the embodiment is not to thoroughly index the latest web content like a search engine, but to collect information necessary to immediately remove domain names that were once included in the allow list but are no longer valid as of the latest time.
[0078] [Processing Procedure] FIG. 4 is a flowchart showing the processing procedure executed by the processing device 10 shown in FIG.
[0079] The seed selection unit 11 performs a seed selection process to select reliable websites as seeds and create a seed list showing the selected websites (step S1).
[0080] The first extraction unit 12 performs a first extraction process of crawling the websites shown in the seed list, identifying and extracting domain names externally linked from the websites selected as seeds, along with the web content obtained by the accesses of this web crawl (step S2). The first extraction unit 12 performs this web crawl multiple times at predetermined intervals.
[0081] The second extraction unit 13 performs a second extraction process in which it performs additional DNS crawling on the domain names discovered through external links from the seeds, and refers to and extracts DNS records that cannot be obtained by web crawling alone (step S3). The second extraction unit 13 performs this DNS crawling multiple times at predetermined intervals.
[0082] The generation unit 14 performs a generation process to identify reliable domain names, generate a list of the identified domain names as an allow list, and output the list (step S4). The generation unit 14 expresses the differences and situations of multiple crawl results for domain names discovered during the web crawl in a ring model structure. The generation unit 14 eliminates unreliable domain names by making step-by-step decisions according to the ring model structure.
[0083] The decision unit 15 performs a decision process to determine the timing for the first extraction unit 12, the second extraction unit 13, and the generation unit 14 to re-execute web crawling, extraction of server certificate information, and extraction of DNS records (step S5).
[0084] [Generation Processing] FIG. 5 is a flowchart showing the processing procedure of the generation processing shown in FIG.
[0085] The generation unit 14 acquires information about the domain name to be judged (step S11). The information about the domain name to be judged is information about the status and temporal changes of multiple types of resources related to the website and / or domain name extracted by the web crawl in the processing of step S2. The information about the domain name to be judged also includes information about the DNS record extracted by the DNS crawl in step S3.
[0086] The generating unit 14 expresses each piece of information relating to the acquired domain name to be judged in a hierarchical ring model structure in association with the arrangement of each ring of the ring model (see FIG. 3) (step S12).
[0087] Next, the generation unit 14 determines whether the domain name to be determined is trustworthy from the inner ring of the ring model.
[0088] Specifically, the generation unit 14 performs a first determination process (step S13) regarding the state of the domain name at the time of crawling (A), a second determination process (step S14) regarding changes in the DNS record since the previous crawling (B), a third determination process (step S15) regarding the state of the server certificate at the time of crawling (C), a fourth determination process (step S16) regarding the acquisition status of web content information at the time of crawling (D), and a fifth determination process (step S17) regarding changes in web backlinks since the previous crawling (E) in that order.
[0089] If there is a next domain name to be determined (step S18: Yes), the generation unit 14 returns to step S11. Then, the generation unit 14 acquires each piece of information related to the next domain name to be determined, expresses it in a ring model structure, and then performs the first to fifth determination processes.
[0090] If there is no next domain name to be judged (step S18: No), the generation unit 14 generates and outputs a list of domain names that have been judged to be trustworthy based on the judgment results of steps S13 to S17 as an allowance list (step S19).
[0091] [First Determination Process] FIG. 6 is a flowchart showing the processing procedure of the first determination process shown in FIG.
[0092] The generation unit 14 refers to information (A) about the status of the domain name at the time of crawling, which corresponds to the innermost ring of the ring model (step S21), and determines whether domain parking is being used at the time of web crawling (step S22).
[0093] If domain parking is not being used at the time of web crawling (step S22: No), the generation unit 14 determines whether an NS record for the domain name exists (step S23).
[0094] If an NS record for the domain name exists (step S23: Yes), the generation unit 14 determines that the domain name to be judged is trustworthy (step S24).
[0095] On the other hand, if domain parking is being used at the time of web crawling (step S22: Yes), or if there is no NS record for the domain name (step S23: No), the generation unit 14 determines that the domain name being judged is not trustworthy (step S25).
[0096] [Second Determination Process] FIG. 7 is a flowchart showing the processing procedure of the second determination process shown in FIG.
[0097] The generation unit 14 determines whether the determination result for the domain name to be determined in the first determination process in the previous stage was that the domain name is trustworthy or that the domain name is not trustworthy (step S31).
[0098] If it is determined that the data is not trustworthy (step S31: not trustworthy), the generation unit 14 ends the second determination process.
[0099] If it is determined that the site is trustworthy (Step S31: Trustworthy), the generation unit 14 performs the following process. First, the generation unit 14 refers to information about the DNS records from the previous crawl and the current crawl to determine changes (B) in the DNS records since the previous crawl, which corresponds to the second ring from the inside of the ring model (Step S32).
[0100] The generation unit 14 determines whether the combination of DNS NS records is different between the previous crawl and the current crawl (step S33).
[0101] If the combination of DNS NS records is the same between the previous crawl and the current crawl (step S33: No), the generation unit 14 determines that the domain name to be judged is trustworthy (step S34).
[0102] On the other hand, if the combination of DNS NS records is different between the previous crawl and the current crawl (step S33: Yes), the generation unit 14 determines that the domain name being judged is not trustworthy (step S35).
[0103] [Third Determination Process] FIG. 8 is a flowchart showing the processing procedure of the third determination process shown in FIG.
[0104] The generation unit 14 determines whether the determination results for the domain name to be determined in the first and second determination processes in the previous stages were that the domain name is trustworthy or untrustworthy (step S41).
[0105] If it is determined that the data is not trustworthy (step S41: not trustworthy), the generation unit 14 ends the third determination process.
[0106] If the determination is that the certificate is trustworthy (step S41: Trustworthy), the generation unit 14 refers to information regarding the state (C) of the server certificate at the time of crawling, which corresponds to the third ring from the inside of the ring model (step S42).
[0107] Then, the generation unit 14 determines whether the server certificate extracted during the web crawl is an expired server certificate (step S43).
[0108] If the server certificate has not expired (step S43: No), the generation unit 14 determines that the domain name to be judged is trustworthy (step S44).
[0109] On the other hand, if the server certificate has expired (step S43: Yes), the generation unit 14 determines that the domain name to be judged is not trustworthy (step S45).
[0110] [Fourth Determination Process] FIG. 9 is a flowchart showing the procedure of the fourth determination process shown in FIG.
[0111] The generation unit 14 determines whether the determination results for the domain name to be determined in the first to third determination processes in the previous stages were that the domain name is trustworthy or untrustworthy (step S51).
[0112] If it is determined that the data is not trustworthy (step S51: not trustworthy), the generation unit 14 ends the fourth determination process.
[0113] If it is determined to be trustworthy (step S51: Trustworthy), the generation unit 14 refers to information regarding the acquisition status (D) of web content information during crawling, which corresponds to the second ring from the outside of the ring model (step S52).
[0114] The generation unit 14 determines whether or not a web page has been acquired during the web crawl (step S53).
[0115] If the web page can be acquired (step S53: No), the generation unit 14 determines that the domain name to be judged is trustworthy (step S54).
[0116] On the other hand, if the web page cannot be acquired (step S53: Yes), the generation unit 14 determines that the domain name to be judged is not reliable (step S55).
[0117] [Fifth Determination Process] FIG. 10 is a flowchart showing the processing procedure of the fifth determination process shown in FIG.
[0118] The generation unit 14 determines whether the determination results for the domain name to be determined in the first to fourth determination processes in the previous stages were that the domain name is trustworthy or that it is not trustworthy (step S61).
[0119] If it is determined that the data is not trustworthy (step S61: not trustworthy), the generation unit 14 ends the fifth determination process.
[0120] If the determination is that the site is trustworthy (step S61: Trustworthy), the generation unit 14 refers to information regarding changes in web backlinks (E) since the previous crawl, which corresponds to the outermost ring of the ring model (step S62).
[0121] The generation unit 14 determines whether the current number of backlinks is equal to or less than a predetermined ratio of the previous number (step S63).
[0122] If the current number of backlinks is not equal to or less than the predetermined ratio of the previous number (step S63: No), the generation unit 14 determines that the domain name to be judged is trustworthy (step S64).
[0123] On the other hand, if the current number of backlinks is equal to or less than the predetermined ratio of the previous number (step S63: Yes), the generation unit 14 determines that the domain name to be judged is not trustworthy (step S65).
[0124] [Re-execution Algorithm] FIG. 11 is a flowchart showing the processing procedure of the re-execution algorithm executed by the determination unit 15.
[0125] The determination unit 15 first performs the following determination on the domain names listed in the seed list created in step S1: The determination unit 15 determines whether L days have passed since the last web crawl for each domain name (step S71).
[0126] If there is a domain name (first domain name) for which L days have passed since the last time a web crawl was performed (step S71: Yes), the determination unit 15 causes the first extraction unit 12 to re-execute the extraction of external links (first external links) (step S72) by re-executing a web crawl for this first domain name.
[0127] If there is no domain name (first domain name) for which L days have passed since the last time the web crawl was performed (step S71: No), or after the processing of step S72, the determination unit 15 performs the following processing for each domain name of the first external link extracted when the seed list was web crawled: First, the determination unit 15 determines whether C days have passed since the last time the server certificate information for each domain name of the first external link was extracted (step S73).
[0128] If there is no domain name (second domain name) for which C days have passed since the last time the server certificate information was extracted (step S73: No), the determination unit 15 determines for each domain name of the first external link whether the expiration date of the server certificate information extracted last time is within one week (step S74).
[0129] If there is a second domain name for which C days have passed since the last time the server certificate information was extracted (step S73: Yes), or if there is a domain name (third domain name) for which the expiration date of the last extracted server certificate information is within one week (step S74: Yes), the determination unit 15 causes the first extraction unit 12 to re-execute the extraction of server certificate information for the domain name (second domain name or third domain name) of this external link (step S75).
[0130] If there is no domain name (third domain name) within one week of the expiration date of the previously extracted server certificate (step S74: No), or after processing in step S75, the determination unit 15 determines whether D days have passed since the last time DNS record extraction for each domain name of the first external link was performed (step S76).
[0131] If there is a domain name (fourth domain name) for which D days have passed since the last DNS record extraction (step S76: Yes), the determination unit 15 causes the second extraction unit 13 to re-extract the DNS record for this fourth domain name (step S77).Then, the generation unit 14 updates the permission list based on the processing results of steps S71 to S77 (step S78).
[0132] In this case, in the embodiment, by setting the parameters so that L>>C≧D, the frequency of recursive retrieval from websites can be reduced, and the information required for the permission list can be collected with a realistic amount of calculation and access.
[0133] In the above description, steps S72, S75, and S77 are described as processing procedures that are executed only for the domain names of the first external links extracted in step S71 that correspond to the predetermined determination results in steps S73, S74, and S76, but this is not limiting. Steps S72, S75, and S77 may be executed for all domain names of the first external links extracted in step S71.
[0134] [Effects of the embodiment] In this way, the processing device 10 according to the embodiment selects a small number of potentially trustworthy websites as seeds, and performs multiple web crawls on these seed sites. The processing device 10 identifies trustworthy sites based on differences in the web crawl results and / or the status of the web crawl results, and generates an allow list indicating the trustworthy sites.
[0135] Therefore, the processing device 10 generates a new allow list starting from legitimate websites. By utilizing the link relationships and network structure that form the backbone of the Web space, the processing device 10 can correctly identify "unpopular but legitimate sites" that are overlooked in the top list, thereby identifying malicious sites with higher accuracy and reducing false positives, and controlling access to malicious sites.
[0136] Furthermore, the processing device 10 hierarchically associates information indicating the differences and status of multiple crawl results for a domain name discovered during a web crawl with the arrangement of each ring of the ring model 100, in which information that is a more reliable starting point is arranged in the inner ring.The processing device 10 then references the information from the top, i.e., from the inner ring, and gradually determines whether a site is trustworthy.In this case, the processing device 10 performs a reliability determination in order from the most reliable starting point, and prioritizes the determination result for information that is closer to the starting point of reliability, thereby appropriately generating an allow list indicating trustworthy sites.
[0137] The processing device 10 also determines the timing for retrying the processes of web crawling, server certificate extraction, and DNS record collection according to a predetermined retry algorithm. In this way, the processing device 10 continuously executes the process for identifying trusted sites, thereby enabling the generation of an updated allow list.
[0138] The processing device 10 does not necessarily need to represent each piece of information regarding the status and temporal changes of multiple types of resources related to websites and / or domain names in a ring model structure. Even if step S12 in Fig. 5 is omitted, the processing device 10 will perform the determination process according to the ring model 100 by performing the first to fifth determination processes (steps S13 to S17) in the order shown in Fig. 5.
[0139] [System Configuration, etc.] The components of each device shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific distribution and integration of each device is not limited to those shown in the figure, and all or part of the devices can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a central processing unit (CPU) and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic. The program may be executed not only by the CPU but also by other processors such as a graphics processing unit (GPU).
[0140] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.
[0141] [Program] In one embodiment, the processing device 10 can be implemented by installing a processing program that executes the above-described processes as package software or online software on a desired computer. For example, by having an information processing device execute the above-described processing program, the information processing device can function as the processing device 10. The information processing device referred to here includes desktop and notebook personal computers. Furthermore, other information processing devices also include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants).
[0142] The processing device 10 may also be implemented as a server device that provides services related to the above-described processing to a client terminal device used by a user. For example, the server device may be implemented as a server device that provides a service that outputs an authorization list. In this case, the server device may be implemented as a web server or as a cloud that provides services related to the above-described processing through outsourcing.
[0143] 12 is a diagram showing an example of a computer that executes a processing program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.
[0144] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.
[0145] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, application programs 1092, program modules 1093, and program data 1094. That is, the programs that define each process of the processing device 10 are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configuration of the processing device 10 are stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD (Solid State Drive).
[0146] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.
[0147] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.
[0148] REFERENCE SIGNS LIST 10 Processing device 11 Seed selection unit 12 First extraction unit 13 Second extraction unit 14 Generation unit 15 Determination unit
Claims
1. A processing device comprising: a selection unit that selects trusted sites as seeds; a first extraction unit that performs multiple web crawls to extract external links from the sites selected as seeds; and a generation unit that identifies trusted sites based on the difference in the crawl results of the web crawls performed by the first extraction unit and / or the status of the crawl results of the web crawls, and generates an allow list indicating the trusted sites.
2. The processing device described in claim 1, characterized in that the generation unit identifies the reliable site by applying a ring model in which information that is more likely to be a source of trust among information regarding the status and changes over time of multiple types of resources related to websites and / or domain names is placed in an inner ring.
3. The processing device described in claim 2, further comprising a second extraction unit that performs a DNS crawl based on the site selected as the seed and the domain name of the external link extracted by the first extraction unit, and collects DNS records and the compatibility status of DNS security mechanisms, wherein the generation unit hierarchically associates information regarding the status and temporal changes of multiple types of resources related to the website, domain name, and / or DNS record obtained by the Web crawl and the DNS crawl for the domain name to be determined with the arrangement of each ring in the ring model, and eliminates unreliable domain names based on the hierarchical information.
4. The processing device described in claim 1, characterized in that the generation unit hierarchically organizes information regarding the status and temporal changes of multiple types of resources related to the website and / or domain name obtained by the web crawl for the domain name to be judged, with information that is more of a starting point for trust being at the top, and refers to the information from the top down to gradually judge whether the site is trustworthy or not.
5. A processing method executed by a processing device, comprising: a selection step of selecting a trusted site as a seed; a first extraction step of performing a Web crawl multiple times to extract external links from the site selected as the seed; and a generation step of identifying trusted sites based on the difference in the crawl results of the Web crawl performed in the first extraction step and / or the status of the crawl results of the Web crawl, and generating an allow list indicating the trusted sites.
6. A processing program for causing a computer to execute the following steps: a selection step of selecting a trusted site as a seed; a first extraction step of performing a Web crawl multiple times to extract external links from the site selected as the seed; and a generation step of identifying trusted sites and generating an allowance list indicating the trusted sites based on the difference in the crawl results of the Web crawl performed in the first extraction step and / or the status of the crawl results of the Web crawl.
Citation Information
Patent Citations
Indicating website reputations during user interactions
US20140331119A1
Device for calculating maliciousness of communication destination, method for calculating maliciousness of communication destination, and program for calculating maliciousness of communication destination
WO2016140038A1
