Device for generating a list of dangerous websites, method for generating a list, and program for generating a list of dangerous websites
The device efficiently generates a comprehensive list of dangerous websites by analyzing search results for frequently occurring domains and periodically verifying site relevance, addressing inefficiencies and inaccuracies in existing methods.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- KDDI CORP
- Filing Date
- 2023-05-09
- Publication Date
- 2026-05-12
AI Technical Summary
Existing methods for generating lists of dangerous websites are inefficient, incomplete, and prone to incorrectly flagging legitimate sites as unsafe, and methods using Whois information are time-consuming and hindered by privacy protections.
A device and method that collects learning search results, extracts frequently occurring domains, uses multiple conditions for dangerous site detection, and manages a list of URLs by periodically verifying their relevance and language consistency.
Efficiently generates a comprehensive list of dangerous sites, reducing time and labor while improving detection accuracy and adaptability to language and regional variations.
Smart Images
Figure 0007857251000001 
Figure 0007857251000002
Abstract
Description
Technical Field
[0001] The present invention relates to a technique for collecting URLs of dangerous sites.
Background Art
[0002] Conventionally, there are dangerous sites that cause disadvantages to users who have viewed websites such as phishing sites or malware distribution sites. Since many of these websites cannot be determined to be dangerous sites at first glance, a system that suppresses damage by automatically determining them is desired.
[0003] Therefore, for example, there are services that respond with safety to inquiries by comparing with a list of non-safe sites (for example, Non-Patent Document 1), services that notify the safety of URLs displayed in search results (for example, Non-Patent Document 2), etc. are provided.
[0004] Furthermore, in Non-Patent Document 3, a method of performing machine learning based on Whois information (registrant name, registration date, contact information, etc.) and identifying domains related to botnets has been proposed. Also, in Patent Document 1, a method of detecting forgery of a site based on the transition status of a host when accessing a specific site has been proposed.
Prior Art Documents
Patent Documents
[0005]
Patent Document 1
Non-Patent Documents
[0006]
Non-Patent Document 1
[0007] Maintaining a list of unsafe sites presented several challenges: it could take time for sites to be added to the list, and the list's lack of comprehensiveness resulted in low detection rates. Furthermore, since the determination of whether a site is safe or unsafe relied on the assumption that the webpage had not been tampered with, there was a risk that a legitimate site that had been compromised might be incorrectly flagged as safe.
[0008] Furthermore, in methods for detecting malicious domains using Whois information, it is necessary to collect Whois information for all domains in the list when creating the list. In addition, obtaining Whois information takes time, and in recent years, some domains have begun to protect the privacy of their Whois information, making it increasingly difficult to determine whether a site is dangerous or not. Even with the method described in Patent Document 1, verification is required for each URL, so creating a list of dangerous sites is time-consuming and labor-intensive.
[0009] The present invention aims to provide a list generation device, a list generation method, and a list generation program that can efficiently generate a list of dangerous sites with improved comprehensiveness. [Means for solving the problem]
[0010] The list generation device according to the present invention comprises: a learning data collection unit that collects multiple learning search results using a predetermined number of the top frequently occurring search queries in a web search service; a domain extraction unit that extracts domains that frequently appear in the learning search results according to predetermined rules; a discovery data collection unit that collects search results for dangerous site detection using search queries consisting of multiple conditions; a dangerous site extraction unit that extracts URLs containing domains different from those extracted by the domain extraction unit from the dangerous site detection search results; and a list management unit that manages the URLs extracted by the dangerous site extraction unit as a list of dangerous sites.
[0011] The domain extraction unit may extract top-level domains as the frequently occurring domains.
[0012] The domain extraction unit may extract attribute-type domains that have been registered in advance as frequently occurring domains.
[0013] The aforementioned discovery data collection unit may include the site category in the multiple conditions.
[0014] The aforementioned data collection unit for discovery may include multiple search keywords as part of the multiple conditions.
[0015] The aforementioned learning data collection unit may set an upper limit on the number of sites to be collected as search results for learning.
[0016] The dangerous site extraction unit may apply an index for evaluating the importance of words in a document, treating the multiple learning search results and the dangerous site discovery search results as documents and the domains as words, and extract URLs that include domains whose scores on the index exceed a predetermined threshold.
[0017] For each URL included in the list, the list management unit checks, at a predetermined timing, whether it is included in the search results using the same search query as when it was searched by the discovery data collection unit, and may delete it from the list if it is not included in the search results.
[0018] For each URL included in the list, the list management unit checks, at a predetermined timing, whether the language used on the web page based on the URL has changed, and may delete it from the list if it has changed.
[0019] The list generation method according to the present invention includes, in a web search service, a learning data collection step of collecting a plurality of learning search results using a predetermined number of top frequent search queries, a domain extraction step of extracting domains that frequently appear in the learning search results according to a predetermined rule, a discovery data collection step of collecting search results for discovering dangerous sites using a search query consisting of a plurality of conditions, a dangerous site extraction step of extracting URLs including domains different from the domains extracted in the domain extraction step from the search results for discovering dangerous sites, and a list management step of managing the URLs extracted in the dangerous site extraction step as a list of dangerous sites, which is executed by a computer.
[0020] The list generation program according to the present invention is for causing a computer to function as the list generation device.
Advantages of the Invention
[0021] According to the present invention, a list of dangerous sites with improved comprehensiveness is efficiently generated.
Brief Description of the Drawings
[0022] [Figure 1] It is a diagram showing the functional configuration of the list generation device in the embodiment. [Figure 2] It is a flowchart exemplifying the procedure of the list generation method in the embodiment. [Modes for carrying out the invention]
[0023] [First Embodiment] The following describes a first embodiment of the present invention. The list generation device of this embodiment collects URLs of dangerous websites on the internet and generates and manages a blacklist to restrict user access.
[0024] Search results from web search services include frequently occurring top-level domains (TLDs) and less frequently occurring TLDs, with many dangerous sites, including hacked websites, belonging to the latter group. The reasons for this are thought to be as follows: In other words, due to geographical and linguistic differences in keyword searches by search engines, for example, search results within Japan often contain URLs ending in ".jp" or ".com," while search results within Canada often contain URLs ending in ".ca" or ".com." However, dangerous sites are often tampered with through attacks targeting vulnerable legitimate sites overseas, and therefore, they often have ccTLDs other than ".jp" but are web pages written in Japanese.
[0025] This embodiment utilizes this feature to list URLs containing rare TLDs as dangerous sites when a TLD different from frequently occurring TLDs appears in the search results.
[0026] Figure 1 shows the functional configuration of the list generation device 1 in this embodiment. The list generation device 1 is an information processing device (computer) such as a server device or a personal computer, and includes a control unit 10 and a storage unit 20, as well as various data input / output devices and communication devices.
[0027] The control unit 10 controls the entire list generation device 1 and operates as the respective functional units described later by appropriately reading and executing various programs stored in the storage unit 20, thereby realizing each function in this embodiment. The control unit 10 may be a CPU.
[0028] The storage unit 20 is a storage area for various programs and data that enable the hardware group to function as a list generation device 1, and may be ROM, RAM, flash memory, or a hard disk drive (HDD). Specifically, the storage unit 20 stores a program (list generation program) for causing the control unit 10 to execute each of the functions of this embodiment, as well as a list of URLs to be managed, and various databases, etc.
[0029] The control unit 10 includes a learning data collection unit 11, a domain extraction unit 12, a discovery data collection unit 13, a dangerous site extraction unit 14, and a list management unit 15.
[0030] The learning data collection unit 11 collects multiple learning search results using a predetermined number of the most frequently occurring search queries in a web search service. Specifically, the learning data collection unit 11 collects the responses, which are the search results for each of the N search queries that are ranked highly in overall search results (for example, the search ranking for all categories over the past 30 days) in statistical information from web search services (for example, Google Trends, etc.) in order to collect TLDs that frequently appear in a particular language and region.
[0031] The training data collection unit 11 uses the search results for these N search queries as training data. If the number of responses for a single query is large, the training data collection unit 11 may set an upper limit X on the number of sites to be acquired as training search results.
[0032] The domain extraction unit 12 extracts domains that frequently appear in the training search results collected by the training data collection unit 11 according to predetermined rules. Specifically, the domain extraction unit 12 may extract top-level domains (TLDs) as frequently occurring domains. Alternatively, the domain extraction unit 12 may extract attribute-type domains that have been pre-registered as candidates as frequently occurring domains. An attribute-type domain is a domain that, in addition to a TLD (e.g., ".jp"), includes attributes that distinguish it according to the type of organization, such as ".co.jp" or ".or.jp".
[0033] When only TLDs are targeted, for example, in Japanese search results within Japan, ".com", ".jp", etc. appear frequently, while the appearance rate of domains from other countries, which consist of web pages written in other languages, is low. There are no specific rules for extracting frequently occurring TLDs, but for example, a predetermined number of TLDs may be extracted from the top when sorted in descending order of frequency, or TLDs whose frequency exceeds a threshold. Alternatively, a predetermined number of TLDs, or those exceeding a threshold, may be extracted based on an evaluation value obtained by weighting the frequency according to the output order of the search results.
[0034] The discovery data collection unit 13 collects search results for detecting dangerous websites using search queries consisting of multiple conditions, from the same web search service as the learning data collection unit 11. Here, under conditions where the number of search results is not excessive, it is thought that the proportion of dangerous sites in the search results will increase. Therefore, the discovery data collection unit 13 collects search results for the following search queries.
[0035] For example, the discovery data collection unit 13 uses multiple conditions, including the site category, as the search query. Alternatively, the discovery data collection unit 13 uses multiple conditions, including multiple search keywords, as the search query. Furthermore, the discovery data collection unit 13 may create a search query using multiple search keywords for each category, incorporating both of these conditions.
[0036] The dangerous site extraction unit 14 extracts URLs containing domains different from the frequently occurring domains extracted by the domain extraction unit 12 from the search results for dangerous site detection collected by the discovery data collection unit 13.
[0037] The list management unit 15 manages the URLs extracted by the dangerous site extraction unit 14 as a list of dangerous sites and provides them as a blacklist to restrict user access.
[0038] In the case of dangerous websites where the page data of a legitimate site has been tampered with, the data is corrected over time, and the site changes from a dangerous site to a legitimate site. Therefore, maintaining a blacklist of legitimate sites is an undesirable situation. Therefore, the list management unit 15 stores the discovery date of the dangerous site and the original search query that led to its discovery in the storage unit 20, and periodically performs searches to check its status.
[0039] In other words, the list management unit 15 periodically checks, at predetermined intervals, whether each URL included in the generated list is included in the search results using the same search query as when it was searched by the discovery data collection unit 13, and removes it from the list if it is not included in the search results. Furthermore, the list management unit 15 may, at predetermined intervals, check whether the language used on the webpage based on each URL included in the list has changed, and if it has changed, it may determine that the site has been changed to a canonical site (corrected) and remove it from the list.
[0040] Figure 2 is a flowchart illustrating the procedure for generating a list in this embodiment. Here, the domains to be extracted from search results for training purposes and search results for detecting dangerous sites are referred to as TLDs.
[0041] In step S1, the learning data collection unit 11 extracts N frequently occurring search queries from the web search service. In step S2, the training data collection unit 11 collects training search results for each of the N search queries extracted in step S1.
[0042] In step S3, the domain extraction unit 12 extracts frequently occurring TLDs from the N search results collected in step S2.
[0043] In step S4, the discovery data collection unit 13 extracts m search queries from each of the n categories (not all categories, but art, entertainment, games, finance, etc.) that consist of multiple keywords that rank highly in search results. In step S5, the discovery data collection unit 13 collects search results for discovering dangerous sites using the extracted total of n × m search queries.
[0044] In step S6, the dangerous site extraction unit 14 extracts URLs from the search results for dangerous site detection collected in step S5 that have a different TLD from the frequently occurring TLD extracted in step S3. In step S7, the list management unit 15 stores the URLs extracted in step S6 in the storage unit 20 as a list of dangerous sites.
[0045] [Second Embodiment] The following describes a second embodiment of the present invention. The functional configuration of the list generation device 1 in this embodiment is the same as that of the first embodiment (Figure 1), but the processing content of the dangerous site extraction unit 14 is different.
[0046] The dangerous site extraction unit 14 applies an index such as TF-IDF or BM25 to evaluate the importance of words in a document, treating multiple training search results and discovery search results as documents and domains as words. As a result, the dangerous site extraction unit 14 extracts URLs that contain domains whose scores on this index exceed a predetermined threshold.
[0047] Specifically, the dangerous site extraction unit 14 treats the search results for dangerous site detection and the N training data as a total of N+1 document data, and calculates a score (word importance) for each domain that appears in the search results for dangerous site detection. The dangerous site extraction unit 14 adds URLs with high scores to the list of dangerous sites.
[0048] According to the embodiment described above, the list generation device 1 learns frequently occurring domains (TLDs or attribute-type domains) by obtaining search results using top-ranking search queries in search trends, and collects URLs that do not contain frequently occurring domains in newly searched search results for dangerous site detection, considering them to be dangerous sites including tampered sites.
[0049] As a result, the list generation device 1 can increase the number of dangerous sites in the list according to the number of search queries, thus improving the efficiency, or comprehensiveness, of generating lists of dangerous sites compared to conventional methods. Furthermore, since the process can be completed solely by analyzing the domains of the URLs obtained as search results, time and labor costs are reduced. The URLs of dangerous websites collected in this way can be used as a blacklist to restrict user access. For example, by comparing URLs in emails with the list, dangerous websites can be detected without accessing them.
[0050] Here, with the list generation method of list generation device 1, for example, when using search results within Japan, it is possible to miss dangerous sites containing frequently occurring TLDs such as ".com" and ".jp". However, it can collect more dangerous sites than conventional methods. Furthermore, by applying the above method to different languages or countries, it becomes possible to collect dangerous sites containing frequently occurring TLDs within Japan.
[0051] The list generation device 1 can easily extract dangerous sites from the URLs of search results by using TLDs as the target of analysis. Furthermore, by targeting attribute-type domains, the list generation device 1 can perform more detailed analysis than by targeting TLDs, thereby improving the comprehensiveness of the list.
[0052] The list generation device 1 can efficiently discover dangerous sites by appropriately narrowing down the search conditions and increasing the appearance rate of dangerous sites, either by including the site category in the search query for discovering dangerous sites or by using multiple search keywords.
[0053] The list generator 1 can appropriately learn frequently occurring domains without excessively increasing the processing load by setting an upper limit on the number of sites collected as search results for training.
[0054] The list generator 1 may apply existing metrics for evaluating the importance of words in a document to evaluate the importance of domains in search results for dangerous site detection, and extract URLs containing domains with high scores as dangerous sites. This allows the list generator 1 to appropriately extract URLs containing domains that are less likely to appear in the learning search results but are concentrated in the search results for detecting dangerous sites.
[0055] The list generator 1 periodically checks whether each URL included in the list can be searched using the same search query as when it was discovered. If it is no longer searched, it can determine that the tampered data on the site has been corrected. This allows the list generator 1 to periodically update the list of dangerous sites and maintain an appropriate state.
[0056] The list generator 1 periodically checks whether the language used on the webpage has changed for each URL included in the list. If it has changed, it can determine that the tampered data on the site in question has been corrected. This allows the list generator 1 to periodically update the list of dangerous sites and maintain an appropriate state.
[0057] Furthermore, the aforementioned embodiment makes it possible to prevent access to dangerous websites, for example, thereby contributing to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote sustainable industrialization and foster innovation."
[0058] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. Furthermore, the effects described in the embodiments described above are merely a list of the most preferred effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.
[0059] The list generation method by the list generation device 1 is implemented by software. When implemented by software, the programs constituting this software are installed on an information processing device (computer). These programs may be distributed to users by being recorded on removable media such as a CD-ROM, or by being downloaded to the user's computer via a network. Furthermore, these programs may be provided to the user's computer as a web service via a network without being downloaded. [Explanation of Symbols]
[0060] 1. List Generator 10 Control Unit 11. Training Data Collection Unit 12 Domain Extraction Section 13. Data Collection Unit for Discovery 14. Dangerous Site Extraction Section 15. List Management Department 20 Memory section
Claims
1. A web search service includes a learning data collection unit that collects multiple training search results using a predetermined number of the top frequently occurring search queries, and A domain extraction unit that extracts domains that frequently appear in the aforementioned learning search results according to predetermined rules, A discovery data collection unit collects search results for identifying dangerous websites using search queries consisting of multiple conditions, A dangerous site extraction unit extracts URLs containing domains different from those extracted by the domain extraction unit from the search results for the aforementioned dangerous site detection, A list generation device comprising: a list management unit that manages the URLs extracted by the dangerous site extraction unit as a list of dangerous sites.
2. The list generation device according to claim 1, wherein the domain extraction unit extracts top-level domains as the frequently occurring domains.
3. The list generation device according to claim 1, wherein the domain extraction unit extracts attribute-type domains that have been registered in advance as frequently occurring domains.
4. The list generation device according to any one of claims 1 to 3, wherein the discovery data collection unit includes site categories in the plurality of conditions.
5. The list generation device according to any one of claims 1 to 3, wherein the discovery data collection unit includes a plurality of search keywords as the plurality of conditions.
6. The list generation device according to any one of claims 1 to 3, wherein the learning data collection unit sets an upper limit on the number of sites to be collected as search results for learning.
7. The dangerous site extraction unit applies an index for evaluating the importance of words in a document, treating the plurality of learning search results and the dangerous site discovery search results as documents and domains as words, and extracts URLs that include domains whose score of the index exceeds a predetermined threshold, according to any one of claims 1 to 3.
8. The list generation device according to any one of claims 1 to 3, wherein the list management unit checks at a predetermined time whether each URL included in the list is included in the search results using the same search query as when it was searched by the discovery data collection unit, and deletes it from the list if it is not included in the search results.
9. The list generation device according to any one of claims 1 to 3, wherein the list management unit checks at a predetermined time whether the language used on the web page based on each URL included in the list has changed, and deletes it from the list if it has changed.
10. In a web search service, a training data collection step involves collecting multiple training search results using a predetermined number of the top frequently occurring search queries, and A domain extraction step in which domains that frequently appear in the aforementioned training search results are extracted according to predetermined rules, A discovery data collection step that collects search results for detecting dangerous sites using search queries consisting of multiple conditions, A dangerous site extraction step which extracts URLs containing domains different from those extracted in the domain extraction step from the search results for the aforementioned dangerous site detection, A list generation method in which a computer performs a list management step for managing the URLs extracted in the dangerous site extraction step as a list of dangerous sites.
11. A list generation program for causing a computer to function as a list generation device according to any one of claims 1 to 3.