Processing device, processing method, and processing program
The processing device addresses the challenge of misidentifying 'unpopular but legitimate' sites by generating a permission list through web crawls and DNS record extraction, improving the accuracy of malicious site identification and access control.
Patent Information
- Application Number
- PCT/JP2024/019686
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-05-29
- Publication Date
- 2024-12-05
AI Technical Summary
Existing methods for identifying malicious sites often misjudge 'unpopular but legitimate' sites as malicious due to their exclusion from popularity-based top lists, leading to false positives and difficulties in distinguishing between legitimate and phishing sites.
A processing device and method that selects reliable seed sites, performs web crawls to extract external links and DNS records, and generates a permission list to identify trusted sites, thereby including 'unpopular but legitimate' sites and reducing false positives.
Improves the accuracy of identifying malicious sites and controlling access by correctly indicating legitimate but unpopular sites, reducing false positives and enhancing precision in malicious site detection.
Smart Images

Figure JP2024019686_05122024_PF_FP_ABST
Abstract
Description
Processing device, processing method, and processing program
[0001] The present invention relates to a processing device, a processing method, and a processing program.
[0002] An allowlist is a list of sites that have been pre-determined to be trustworthy based on certain criteria. Sites on the allowlist are excluded from malicious judgments and are considered to be non-malicious sites (legitimate sites, benign sites). An allowlist is also sometimes called a safelist or whitelist.
[0003] Allowlists are commonly used in web security to mitigate the problem of "false positives," where non-malicious websites are mistakenly identified as malicious.
[0004] However, the web is extremely vast and dynamically changing, making it difficult to manually select a comprehensive list of trusted sites and continually update and maintain an allow list.
[0005] As a result, "top lists" have become widely used in both industry and academia as de facto allow lists, based on the implicit assumption that "popularity is likely to be safe."
[0006] A top list is a ranking list of websites or domain names in order of popularity based on specific criteria. There are multiple top lists, and each one has significantly different definitions of the data used, definitions of "popularity," and calculation methods.
[0007] A representative and widely used top list is the Tranco Top Million List (Non-Patent Document 1). The Tranco Top Million List is a combination of the Alexa Top 1 Million, Cisco Umbrella 1 Million, and Majestic Million top lists. Alexa Top 1 Million is one of the most commonly used top lists and is determined based on the average number of daily visits and page views to a website. Cisco Umbrella 1 Million is a list of the most popular domain names ranked in order based on DNS observations in the DNS (Domain Name System) service provided by Cisco Umbrella (registered trademark). Majestic Million is a list of website popularity calculated based on web links provided by Majestic SEO (registered trademark).
[0008] The Secrank List (Non-Patent Document 2) is a new top list processing method that uses a voting algorithm based on the DNS request volume of each domain name. There is also the Chrome User Experience Report (CrUX), which ranks the popularity of websites based on data collected from users of the Google Chrome web browser.
[0009] Victor Le Pochat et al., “TRANCO: A Research-Oriented Top Sites Ranking Hardened Against Manipulation”, In: Proc. NDSS (2019).Qinge Xie, et al., “Building an Open, Robust, and Stable Voting-Based Domain Top List”, In: Proc. USENIX Security (2022).
[0010] There are two main examples of using top lists as de facto allow lists:
[0011] For example, let's look at the first example, which is used to remove false positives. To prevent false positives, one example is to use domain names from the Majestic top list as an allow list. Additionally, the open source threat intelligence platform MISP (Malware Information Sharing Platform) uses Alexa Top1k, Umbrella Top 20k, Majestic Top1k, and Tranco Top 10k as options for removing false positives.
[0012] Next, we will explain a second example of its use with benign labeling. Top lists are widely used when using machine learning in the security field, for example, to label domain names as "benign" or to exclude some domain names from analysis as "benign" in the analysis process. Furthermore, when detecting phishing sites, domain names listed at the top of the top list are often selected as "genuine" domains that are likely to be imitated.
[0013] However, because the Top List is a system that lists "popular" sites in order, there is a fundamental problem that "unpopular but good sites" are difficult to list. For example, sites of new companies, sites of new services of existing companies, and sites of services provided to specific regions are unlikely to be listed on the Top List.
[0014] If such sites are not listed, false positives will occur when identifying malicious sites or controlling (filtering) access to them, resulting in legitimate sites being mistakenly identified as malicious, or in the inability to distinguish between less popular legitimate sites and phishing sites.
[0015] The present invention has been made in consideration of the above, and aims to provide a processing device, processing method, and processing program that can generate an allowance list that correctly indicates ``unpopular but legitimate sites,'' thereby identifying malicious sites and improving the accuracy of access control to malicious sites.
[0016] In order to solve the above-mentioned problems and achieve the objectives, the processing device is characterized by having a selection unit that selects trusted sites as seeds, a first extraction unit that performs a web crawl on the sites selected as seeds by the selection unit and extracts external links from the sites selected as seeds, and a generation unit that identifies trusted sites based on the external links extracted by the first extraction unit and generates an allow list indicating the trusted sites.
[0017] According to the present invention, it is possible to generate an allow list that correctly indicates "unpopular but legitimate sites," thereby improving the accuracy of identifying malicious sites and controlling access to malicious sites.
[0018] Fig. 1 is a diagram showing an example of the configuration of a processing device according to an embodiment. Fig. 2 is a flowchart showing the processing procedure executed by the processing device shown in Fig. 1. Fig. 3 is a diagram showing an overview of collection targets and processing during web crawling. Fig. 4 is a flowchart showing the processing procedure of a re-execution algorithm executed by a determination unit. Fig. 5 is a diagram showing an example of a computer that executes a processing program.
[0019] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a processing apparatus, a processing method, and a processing program according to the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the following embodiments.
[0020] The processing device according to the embodiment generates a new allowed list starting from legitimate websites, rather than a top-down approach that relies on popularity.
[0021] The processing device selects a small number of potentially trustworthy websites as seeds. The processing device performs web crawling on these seed sites, and when external links are received from the seed sites, the processing device extracts the domain names, web content, and DNS (Domain Name System) records of those sites, and then extracts the trusted domain names as an allow list. The processing device continuously performs these processes to generate the latest allow list at any given time.
[0022] In this way, the processor utilizes the link relationships and network structure that form the backbone of the Web space. This enables the processor to correctly identify "unpopular but legitimate sites" that are overlooked in top lists, thereby identifying malicious sites with higher accuracy and fewer false positives, and controlling access to malicious sites.
[0023] [Processing Device] Next, the configuration of a processing device according to an embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a processing device according to an embodiment. Fig. 2 is a flowchart showing the processing procedure of processing executed by the processing device 10 shown in Fig. 1.
[0024] The processing device 10 according to the embodiment is a server device implemented by, for example, loading a predetermined program into a computer or the like including a ROM (Read Only Memory), a RAM (Random Access Memory), a CPU (Central Processing Unit), etc., and having the CPU execute the predetermined program. The processing device 10 also has a communication interface for transmitting and receiving various information to and from other devices connected via a network, etc. As will be described later, the processing device 10 has a crawling function that crawls websites via links and copies and saves information on web pages and DNS record information.
[0025] As shown in FIG. 1, the processing device 10 includes a seed selection unit 11 (selection unit), a first extraction unit 12 , a second extraction unit 13 , a generation unit 14 , and a determination unit 15 .
[0026] The seed selection unit 11 performs a seed selection process to select reliable websites as seeds and create a seed list showing the selected websites (step S1 in FIG. 2).
[0027] The first extraction unit 12 performs a first extraction process in which it performs a web crawl on the websites shown in the seed list and identifies and extracts domain names externally linked from the websites selected as seeds, along with the web content obtained by the access of this web crawl (step S2 in Figure 2).
[0028] The second extraction unit 13 performs a second extraction process in which it performs additional DNS crawling on the domain names discovered through external links from the seed, and references and extracts DNS records that cannot be obtained by web crawling alone (step S3 in Figure 2).
[0029] At this point, the generation unit 14 performs a generation process to identify reliable domain names, generate a list of the identified domain names as an allow list, and output the list (step S4 in FIG. 2). The allow list is a list of sites that have been determined to be reliable in advance based on specific criteria, and sites on the allow list are excluded from malicious judgments and are considered to be non-malicious sites (legitimate sites, benign sites).
[0030] The determination unit 15 performs a determination process to determine the timing for the first extraction unit 12, the second extraction unit 13, and the generation unit 14 to re-execute web crawling, extraction of server certificate information, and extraction of DNS records (step S5 in Figure 2).
[0031] [Seed Generation Unit] A description will be given of the seed selection process executed by the seed selection unit 11. The seed selection unit 11 selects a limited number of reliable websites as seeds and creates a seed list indicating the selected seeds.
[0032] The seed selection unit 11 selects reliable websites by, for example, selecting the official websites of listed companies. The seed selection unit 11 lists listed companies in each country's market and selects the official pages of the listed companies as seeds.
[0033] This method is based on the following assumptions: Listed companies have a legal obligation to disclose information. While there are multiple ways to disclose information, most companies generally provide investor relations (IR) information through a website. For this reason, the assumption is that listed company websites are relatively trustworthy in the online space. However, it should be noted that even if a website belongs to a listed company, there is still a chance that it may be tampered with or not be adequately maintained.
[0034] The seed selected by the seed selection unit 11 is merely the starting point for subsequent processing. The decision as to whether or not to select each domain name for the final allow list is made anew by subsequent functional configuration units. Therefore, the seed selection unit 11 does not need to select websites that are 100% trustworthy.
[0035] Here are two specific examples of selecting listed companies. The first is to use a list such as the "Forbes Global 2000" published annually by Forbes (registered trademark). The Forbes Global 2000 is a list of the top 2,000 companies ranked based on various criteria such as sales, net assets, market capitalization, and profits as indicators of company size, and includes many of the world's listed companies.
[0036] The second method is to select companies listed on the securities markets of each country. For example, in Japan, you can select companies listed on the Tokyo Stock Exchange. Similarly, you can select companies listed on stock exchanges around the world.
[0037] In the above selection example, even if you select a listed company, the company name alone will not uniquely determine the URL (Uniform Resource Locator) or domain name of that company's official website.
[0038] Therefore, for example, when using a search engine to search for "company name" plus a string such as "official website," the page that appears first is selected as the URL of the official website.
[0039] In addition, in EC (Electronic Commerce) companies, the top page for general customers (e.g., https: / / www.amazon[.]com / ) and the company homepage (e.g., https: / / www.aboutamazon[.]com / ) may be prepared as separate domain names, and this embodiment mainly assumes the use of the latter.
[0040] The seed list creation method is not limited to the above-mentioned method, and may be a method of manually selecting reliable sites or other methods.
[0041] [First Extraction Unit] The following describes the first extraction process executed by the first extraction unit 12. The first extraction unit 12 performs a web crawl on each URL included in the seed list created by the seed selection unit 11. The first extraction unit 12 extracts links included in web pages obtained by the web crawl, along with server certificate information and web content information, and stores them with a timestamp of when the web crawl was performed.
[0042] Specifically, the first extraction unit 12 extracts links by repeatedly crawling the web recursively up to a specified hierarchy (depth) based on the site selected as a seed. The first extraction unit 12 determines whether the extracted link is an external link based on the domain name of the extracted link, and if the extracted link is an external link, extracts server certificate information and web content information.
[0043] The first extraction unit 12 performs either the first or second process for each extracted link, depending on its PLD (Pay Level Domain). A PLD is the smallest domain unit that users can directly register and consists of a subdomain of a public suffix (e.g., ".com" or "co.uk"). It should be noted that these terms are not necessarily used consistently within the industry. For example, a public suffix is sometimes called an effective top-level domain (eTLD). Other terms with the same meaning as PLD include registrable domain, eTLD+1, apex domain, and effective second (2nd) level domain (eSLD / e2LD).
[0044] The first process is a process in which, if the PLD of the linked site matches the PLD of the seed, the web crawling process is repeated recursively until a pre-specified hierarchy (depth) is reached, and web access is performed again to the linked site to extract the link.
[0045] The second process is a process in which, if the PLD of the linked site does not match the PLD of the seed, a single Web access is made to the linked site, the content (server certificate information and Web content information) is collected, and the extraction is completed.
[0046] Figure 3 shows an overview of the targets and processing involved in web crawling. For example, in Figure 3, if the seed URL is "https: / / example[.]com" and its depth is set to 0, the link with the same PLD at the next depth (Depth 1), "https: / / example[.]com / about / ," is again included as a target for link extraction (Process 1). On the other hand, the external link with a mismatched PLD, "https: / / security[.]example / ," is accessed up to that URL but not beyond that page (Process 2).
[0047] [Second Extraction Unit] The second extraction process executed by the second extraction unit 13 will be described. The second extraction unit 13 additionally executes DNS crawling for all domain names discovered by the web crawling. The second extraction unit 13 executes DNS crawling based on the sites selected as seeds and the domain names of the external links extracted by the first extraction unit 12, and extracts DNS records and the support status of DNS security mechanisms.
[0048] Specifically, the second extraction unit 13 first collects five types of common DNS records (NS, A, AAAA, MX, and TXT) for each domain name. If an A record or an AAAA record exists, the second extraction unit 13 also collects the country and organization corresponding to the IP address.
[0049] The second extraction unit 13 employs, for example, a method of using a known GeoIP database to associate IP addresses with corresponding countries and organizations.
[0050] Furthermore, the second extraction unit 13 verifies the extent to which each domain complies with seven DNS security mechanisms (DNSSEC (DNS Security Extension), CAA (Certification Authority Authorization), SPF (Sender Policy Framework), DKIM (DomainKeys Identified Mail), Domain-based Message Authentication, Reporting, and Conformance (DMARC), Mail Transfer Agent Strict Transport Security (MTA-STS), DNS-based authentication of named entities (DANE)), and collects the information.
[0051] The security mechanisms used in the DNS targeted in this embodiment are those mentioned above, but are not limited to these, as new protocols and methods (e.g., Brand Indicators for Message Identification (BIMI)) are proposed and gradually gain popularity every year.
[0052] [Generation Unit] The following describes the generation process executed by the generation unit 14. The generation unit 14 determines whether the domain name of the site selected as the seed and the domain name extracted as the external link of the seed are trustworthy according to a predetermined condition or a combination of a plurality of predetermined conditions, and generates an allow list based on the determination result.
[0053] Specifically, the generation unit 14 extracts trusted domain names that meet certain conditions from the domain names of the external links of each seed extracted when the first extraction unit 12 crawls the web from the URLs of the seed list created by the seed selection unit 11, and generates an allowance list.
[0054] The certain conditions are that each of the following conditions is satisfied individually, or that a condition utilizing a combination of the conditions is satisfied (for example, a combination of the first condition and the third condition).
[0055] The first condition is that the content can be correctly acquired without errors via web access (HTTP (Hyper Text Transfer Protocol) or HTTPS (Hyper Text Transfer Protocol Secure)). The second condition is that there is no certificate error (for example, expired server certificate information) during HTTPS access.
[0056] The third condition is that the DNS records are correctly configured and can be retrieved without error. The fourth condition is that N or more DNS security mechanisms are configured and can be retrieved or verified without error.
[0057] However, the first to fourth conditions are not limited to the above conditions. The generation unit 14 may determine whether the domain name is trustworthy using a different method and determine whether to add it to the allow list.
[0058] [Determination Unit] The processing of the determination unit 15 will now be described. Simply generating the allow list is not sufficient; it must be updated so that domain names that are no longer trustworthy are no longer included in the allow list. To achieve this, the first extraction process, the second extraction process, and the generation process must be re-executed periodically (for example, once a week).
[0059] Therefore, the determination unit 15 determines the timing for re-executing the first extraction process, the second extraction process, and the generation process.
[0060] Ideally, it would be desirable to re-execute all of the processes, including the first extraction process, the second extraction process, and the generation process, every time the seed selection unit 11 updates each website in the seed list. However, continuously monitoring the updates of all websites without exception would require an extremely large amount of calculation and access to each website, which would be a heavy burden and would be unrealistic.
[0061] The determination unit 15 determines the timing of re-execution for the extraction of external links by web crawling, the extraction of server certificate information, and the extraction of DNS records among the first extraction process, the second extraction process, and the generation process, according to a predetermined re-execution algorithm.
[0062] This re-execution algorithm is based on the assumption in the embodiment that website update information required to create the allow list consists of content updates, server certificate updates, and / or DNS record updates, and that it is not necessary to recursively retrieve all web content each time. This is because the purpose of the embodiment is not to thoroughly index the latest web content like a search engine, but to collect information necessary to immediately remove domain names that were once included in the allow list but are no longer valid as of the latest time.
[0063] [Re-execution Algorithm] FIG. 4 is a flowchart showing the processing procedure of the re-execution algorithm executed by the decision unit 15.
[0064] The determination unit 15 first performs the following determination on the domain names listed in the seed list created in step S1: The determination unit 15 determines whether L days have passed since the last web crawl for each domain name (step S11).
[0065] If there is a domain name (first domain name) for which L days have passed since the last time a web crawl was performed (step S11: Yes), the determination unit 15 causes the first extraction unit 12 to re-execute the extraction of external links (first external links) (step S12) by re-executing a web crawl for this first domain name.
[0066] If there is no domain name (first domain name) for which L days have passed since the last time the web crawl was performed (step S11: No), or after the processing of step S12, the determination unit 15 performs the following processing for each domain name of the first external link extracted when the seed list was web crawled: First, the determination unit 15 determines whether C days have passed since the last time the server certificate information for each domain name of the first external link was extracted (step S13).
[0067] If there is no domain name (second domain name) for which C days have passed since the last time the server certificate information was extracted (step S13: No), the determination unit 15 determines for each domain name of the first external link whether the expiration date of the server certificate information extracted last time is within one week (step S14).
[0068] If there is a second domain name for which C days have passed since the last time the server certificate information was extracted (step S13: Yes), or if there is a domain name (third domain name) for which the expiration date of the last extracted server certificate information is within one week (step S14: Yes), the determination unit 15 causes the first extraction unit 12 to re-execute the extraction of server certificate information for the domain name (second domain name or third domain name) of this external link (step S15).
[0069] If there is no domain name (third domain name) within one week of the expiration date of the previously extracted server certificate (step S14: No), or after processing in step S15, the determination unit 15 determines whether D days have passed since the last time DNS record extraction for each domain name of the first external link was performed (step S16).
[0070] If there is a domain name (fourth domain name) for which D days have passed since the last DNS record extraction (step S16: Yes), the determination unit 15 causes the second extraction unit 13 to re-extract the DNS record for this fourth domain name (step S17).Then, the generation unit 14 updates the permission list based on the processing results of steps S11 to S17 (step S18).
[0071] In this case, in the embodiment, by setting the parameters so that L>>C≧D, the frequency of recursive retrieval from websites can be reduced, and the information required for the permission list can be collected with a realistic amount of calculation and access.
[0072] In the above description, steps S12, S15, and S17 are described as processing procedures that are executed only for the domain names of the first external links extracted in step S11 according to the predetermined determination results in steps S13, S14, and S16, but this is not limiting. Steps S12, S15, and S17 may be executed for all domain names of the first external links extracted in step S11.
[0073] Effects of the embodiment In this way, the processing device 10 according to the embodiment selects a small number of potentially trustworthy websites as seeds. The processing device 10 performs web crawling on the seed sites to extract external links, and when external links are found from the seed sites, it extracts the domain names, web content, and DNS records of those external links. The processing device 10 then identifies trustworthy sites based on the extracted external links and generates an allow list.
[0074] Therefore, the processing device 10 generates a new allow list starting from legitimate websites. By utilizing the link relationships and network structure that form the backbone of the web space, the processing device 10 can correctly identify "unpopular but legitimate sites" that are overlooked in the top list. This enables the processing device 10 to identify malicious sites with high accuracy and reduce false positives, and to control access to malicious sites.
[0075] The processing device 10 also determines the timing for retrying the processes of web crawling, server certificate extraction, and DNS record collection according to a predetermined retry algorithm. In this way, the processing device 10 continuously executes the process for identifying trusted sites, thereby enabling the generation of an updated allow list.
[0076] [System Configuration, etc.] The components of each device shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific distribution and integration of each device is not limited to those shown in the figure, and all or part of the devices can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a central processing unit (CPU) and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic. The program may be executed not only by the CPU but also by other processors such as a graphics processing unit (GPU).
[0077] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.
[0078] [Program] In one embodiment, the processing device 10 can be implemented by installing a processing program that executes the above-described processes as package software or online software on a desired computer. For example, by having an information processing device execute the above-described processing program, the information processing device can function as the processing device 10. The information processing device referred to here includes desktop and notebook personal computers. Furthermore, other information processing devices also include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants).
[0079] The processing device 10 may also be implemented as a server device that provides services related to the above-described processing to a client terminal device used by a user. For example, the server device may be implemented as a server device that provides a service that outputs an authorization list. In this case, the server device may be implemented as a web server or as a cloud that provides services related to the above-described processing through outsourcing.
[0080] 5 is a diagram showing an example of a computer that executes a processing program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.
[0081] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.
[0082] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, application programs 1092, program modules 1093, and program data 1094. That is, the programs that define each process of the processing device 10 are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configuration of the processing device 10 are stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD (Solid State Drive).
[0083] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.
[0084] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.
[0085] REFERENCE SIGNS LIST 10 Processing device 11 Seed selection unit 12 First extraction unit 13 Second extraction unit 14 Generation unit 15 Determination unit
Claims
1. A processing device comprising: a selection unit that selects a trusted site as a seed; a first extraction unit that performs a web crawl on the site selected as the seed and extracts external links from the site selected as the seed; and a generation unit that identifies trusted sites based on the external links extracted by the first extraction unit and generates an allow list indicating the trusted sites.
2. The processing device according to claim 1, wherein the selection unit lists companies listed on the stock exchanges of each country and selects the official pages of the listed companies as seeds.
3. The processing device described in claim 1, characterized in that the first extraction unit extracts links by repeating web crawling recursively up to a specified hierarchy based on the site selected as the seed, determines whether the extracted link is an external link based on the domain name of the extracted link, and if the extracted link is an external link, extracts server certificate information and web content information.
4. The processing device according to claim 3, further comprising a second extraction unit that performs a DNS crawl based on the site selected as the seed and the domain name of the external link extracted by the first extraction unit, and collects DNS records and the compatibility status of DNS security mechanisms.
5. The processing device described in claim 1, characterized in that the generation unit determines whether the domain name of the site selected as the seed and the domain name extracted as an external link of the seed are trustworthy according to a predetermined condition or a combination of multiple predetermined conditions, and generates the allowable list based on the determination result.
6. The processing device according to claim 4, characterized in that the first extraction unit further has a decision unit that determines the timing of re-execution of the Web crawl, the extraction of the server certificate information, and the extraction of the DNS record in accordance with a predetermined re-execution algorithm.
7. A processing method executed by a processing device, comprising: a selection step of selecting a trusted site as a seed; a first extraction step of performing a web crawl on the site selected as the seed and extracting external links from the site selected as the seed; and a generation step of identifying trusted sites based on the external links extracted in the first extraction step and generating an allow list indicating the trusted sites.
8. A processing program for causing a computer to execute the following steps: a selection step of selecting a trusted site as a seed; a first extraction step of performing a web crawl on the site selected as the seed and extracting external links from the site selected as the seed; and a generation step of identifying trusted sites based on the external links extracted in the first extraction step and generating an allow list indicating the trusted sites.
Citation Information
Patent Citations
Spam tracking analysis reporting system
US20120166458A1
Indicating website reputations during user interactions
US20140331119A1
