Dangerous site list generation device, list generation method, and list generation program

The list generation device enhances the efficiency and comprehensiveness of identifying dangerous sites by analyzing search results for rare TLDs and updating lists based on search queries and language changes, addressing the limitations of existing methods.

JP2024162084AActive Publication Date: 2024-11-21KDDI CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2023077271
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-09
Publication Date
2024-11-21
Estimated Expiration
2043-05-09

AI Technical Summary

Technical Problem

Existing methods for generating lists of dangerous sites are time-consuming, incomplete, and prone to errors due to site tampering, especially when Whois information is protected or when legitimate sites are compromised.

Method used

A list generation device that collects learning search results, extracts domains using predetermined rules, and evaluates URL importance through indices like TF-IDF, managing a list of dangerous sites by regularly updating based on search queries and language changes.

Benefits of technology

Efficiently generates a comprehensive list of dangerous sites, reducing time and labor while improving detection accuracy by focusing on rare TLDs and updating the list to exclude corrected sites.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024162084000001_ABST
    Figure 2024162084000001_ABST
Patent Text Reader

Abstract

To provide a list generation device, list generation method, and list generation program capable of efficiently generating a list of dangerous sites with improved comprehensiveness.SOLUTION: A list generation device 1 includes: a learning data collection unit 11 configured to collect multiple search results for learning using a predetermined number of frequently appearing upper-order search queries in a web search service; a domain extraction unit 12 configured to extract domains that frequently appear in the search results for learning according to a predetermined rule; a discovery data collection unit 13 configured to collect search results for discovering dangerous sites using search queries consisting of multiple conditions: a dangerous site extraction unit 14 configured to extract URLs including domains different from the domains extracted by the domain extraction unit 12 from the search results for discovering dangerous sites; and a list management unit 15 configured to manage the URLs extracted by the dangerous site extraction unit 14 as a list of dangerous sites.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] The present invention relates to a technique for collecting URLs of dangerous sites. [Background technology]

[0002] In the past, there have been dangerous websites, such as phishing sites and malware distribution sites, that cause harm to users who visit them. Since it is often difficult to tell at a glance that these websites are dangerous, there is a need for a system that can automatically identify these websites and prevent damage.

[0003] To address this issue, there are services available that respond to inquiries about safety by comparing the site with a list of unsafe sites (e.g., Non-Patent Document 1), and services that notify users of the safety of URLs displayed in search results (e.g., Non-Patent Document 2).

[0004] Furthermore, Non-Patent Document 3 proposes a method for identifying domains related to botnets by performing machine learning based on Whois information (registrant name, registration date, contact information, etc.). Furthermore, Patent Document 1 proposes a method for detecting site tampering based on the transition status of a host when a specific site is accessed. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6055726 [Non-patent literature]

[0006] [Non-Patent Document 1] Google LLC, Google Safe Browsing, Internet,<https: / / developers.google.com / safe-browsing / v4 / > , February 27, 2023 [Non-patent document 2] Trend Micro, Inc., Virus Buster Cloud "Trend Toolbar" function, Internet,<https: / / helpcenter.trendmicro.com / ja-jp / article / tmka-18502> , January 10, 2023 [Non-patent document 3] Masahiro Hisayama and Ryoichi Sasaki, "A Method for Identifying Malicious Domains Using Domain WHOIS Structure," Multimedia, Distributed, Cooperative and Mobile (DICOMO2016) Symposium, July 2016 Summary of the Invention [Problem to be solved by the invention]

[0007] When maintaining a list of unsafe sites, it can take time for the list to be posted, and the detection rate is low due to the list's low comprehensiveness. In addition, since the judgment of whether a site is safe or unsafe is based on the assumption that the web page has not been tampered with, there is a risk that a legitimate site may be notified as safe if it has been tampered with.

[0008] Furthermore, in the detection method of malicious domains using Whois information, it is necessary to collect Whois information for all domains in the list when creating the list. Also, it takes time to obtain Whois information, and in recent years, there are domains that protect the privacy of their Whois information, making it difficult to determine whether a site is dangerous or not. Even with the technique of Patent Document 1, verification is required for each URL, and creating a list of dangerous sites requires time and labor costs.

[0009] An object of the present invention is to provide a list generation device, a list generation method, and a list generation program that can efficiently generate a list of dangerous sites with improved comprehensiveness. [Means for solving the problem]

[0010] The list generation device of the present invention comprises a learning data collection unit that collects multiple learning search results using a predetermined number of frequently occurring search queries in a web search service; a domain extraction unit that extracts domains that frequently appear in the learning search results according to predetermined rules; a discovery data collection unit that collects search results for discovering dangerous sites using search queries consisting of multiple conditions; a dangerous site extraction unit that extracts URLs that include domains different from the domains extracted by the domain extraction unit from the search results for discovering dangerous sites; and a list management unit that manages the URLs extracted by the dangerous site extraction unit as a list of dangerous sites.

[0011] The domain extraction unit may extract top-level domains as the frequently occurring domains.

[0012] The domain extraction unit may extract a pre-registered attribute domain as the frequently occurring domain.

[0013] The discovery data collector may include a site category in the plurality of conditions.

[0014] The discovery data collection unit may include a plurality of search keywords as the plurality of conditions.

[0015] The learning data collection unit may set an upper limit on the number of sites to be collected as the learning search results.

[0016] The dangerous site extraction unit may apply an index for evaluating the importance of words in documents by treating the multiple search results for learning and the search results for detecting dangerous sites as documents and domains as words, and extract URLs containing domains whose score of the index exceeds a predetermined threshold.

[0017] The list management unit may check, at a predetermined timing, whether each URL included in the list is included in a search result using the same search query as when the discovery data collection unit searched, and may delete the URL from the list if it is not included in the search result.

[0018] The list management unit may check, at a predetermined timing, for each URL included in the list whether the language used in the web page based on that URL has changed, and if there has been a change, remove the URL from the list.

[0019] The list generation method of the present invention is carried out by a computer, and includes a learning data collection step for collecting multiple learning search results using a predetermined number of frequently occurring search queries in a web search service; a domain extraction step for extracting domains that frequently appear in the learning search results in accordance with predetermined rules; a discovery data collection step for collecting search results for discovering dangerous sites using search queries consisting of multiple conditions; a dangerous site extraction step for extracting URLs that include domains different from the domains extracted in the domain extraction step from the search results for discovering dangerous sites; and a list management step for managing the URLs extracted in the dangerous site extraction step as a list of dangerous sites.

[0020] A list generation program according to the present invention causes a computer to function as the list generation device. [Effects of the Invention]

[0021] According to the present invention, a list of dangerous sites with improved comprehensiveness is efficiently generated. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 2 is a diagram illustrating a functional configuration of a list generation device according to an embodiment. [Figure 2] 1 is a flowchart illustrating the procedure of a list generation method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0023] [First embodiment] A first embodiment of the present invention will be described below. The list generation device of this embodiment collects URLs of dangerous sites on the Internet and generates and manages a blacklist for restricting access by users.

[0024] Among the search results of web search services, there are frequently appearing top-level domains (TLDs) and infrequently appearing TLDs, and among the latter there are many dangerous sites, including tampered sites. The reasons for this are thought to be as follows: That is, due to geographical and linguistic differences in keyword searches by search engines, for example, search results in Japan tend to have many URLs ending in ".jp" or ".com," while search results in Canada tend to have many URLs ending in ".ca" or ".com." However, many dangerous sites have been altered through attacks targeting vulnerable legitimate overseas sites, and so they often have web pages written in Japanese with a ccTLD other than ".jp."

[0025] The list generating device of this embodiment utilizes this feature to list URLs containing rare TLDs as risky sites when a TLD different from a frequently occurring TLD appears in the search results.

[0026] FIG. 1 is a diagram showing the functional configuration of a list creation device 1 according to this embodiment. The list creation device 1 is an information processing device (computer) such as a server device or a personal computer, and includes a control unit 10, a storage unit 20, input / output devices for various data, communication devices, and the like.

[0027] The control unit 10 is a part that controls the entire list creation device 1, and operates as each functional unit described below by appropriately reading and executing various programs stored in the storage unit 20, thereby realizing each function in this embodiment. The control unit 10 may be a CPU.

[0028] Storage unit 20 is a storage area for various programs for causing a group of hardware to function as list creation device 1, various data, and the like, and may be a ROM, RAM, flash memory, hard disk drive (HDD), or the like. Specifically, the storage unit 20 stores a program (list generation program) for causing the control unit 10 to execute each function of this embodiment, as well as a list of URLs to be managed, various databases, and the like.

[0029] The control unit 10 includes a learning data collection unit 11 , a domain extraction unit 12 , a discovery data collection unit 13 , a dangerous site extraction unit 14 , and a list management unit 15 .

[0030] The learning data collection unit 11 collects a plurality of learning search results using a predetermined number of frequently occurring search queries in a web search service. Specifically, in order to collect TLDs that appear frequently in a specific language and region, the learning data collection unit 11 collects responses, which are search results for each of N search queries that are ranked high in overall searches (e.g., search rankings for the past 30 days in all categories) in statistical information from web search services (e.g., Google Trends, etc.).

[0031] The learning data collection unit 11 sets the search results for these N search queries as learning data. Here, if the number of responses to one query is large, the learning data collection unit 11 may set an upper limit X on the number of sites to be acquired as search results for learning.

[0032] The domain extraction unit 12 extracts domains that frequently appear in the search results for learning collected by the learning data collection unit 11 according to a predetermined rule. Specifically, the domain extraction unit 12 may extract top-level domains (TLDs) as frequently occurring domains. Alternatively, the domain extraction unit 12 may extract attribute-based domains that have been registered in advance as candidates as frequently occurring domains. An attribute-type domain is a domain that includes attributes distinguished by the type of organization in addition to the TLD (for example, ".jp"), such as ".co.jp" or ".or.jp."

[0033] When only TLDs are considered, for example, search results in Japanese within Japan frequently include ".com" and ".jp," while the appearance rate of foreign domains with web pages written in other languages ​​is low. The predetermined rule for extracting frequently occurring TLDs is not limited, but for example, a predetermined number of TLDs may be extracted from the top when sorted in descending order of occurrence rate, or TLDs with occurrence rates equal to or greater than a threshold. Alternatively, a predetermined number of TLDs or those equal to or greater than a threshold may be extracted based on an evaluation value obtained by weighting occurrence rates according to the order of output of search results.

[0034] The discovery data collection unit 13 collects search results for discovering risky sites using a search query consisting of multiple conditions in the same web search service as the learning data collection unit 11. Here, under conditions where the number of search results is not too large, it is considered that the ratio of dangerous sites among the search results will increase. Therefore, the discovery data collection unit 13 collects search results for the following search queries.

[0035] For example, the discovery data collection unit 13 may set a plurality of conditions including a site category as a search query, or may set a plurality of conditions including a plurality of search keywords as a search query. Furthermore, the discovery data collection unit 13 may create a search query using multiple search keywords for each category, including both of these conditions.

[0036] The dangerous site extraction unit 14 extracts URLs including domains different from the frequently appearing domains extracted by the domain extraction unit 12 from the search results for dangerous site discovery collected by the discovery data collection unit 13.

[0037] The list management unit 15 manages the URLs extracted by the dangerous site extraction unit 14 as a list of dangerous sites, and provides it as a blacklist for restricting access by users.

[0038] In the case of risky sites, if the page data of a legitimate site has been tampered with, the site will be corrected over time and will change from a risky site to a legitimate site. Therefore, it is not desirable to keep a legitimate site registered on the blacklist. For this reason, the list management unit 15 stores the discovery date of the dangerous site and the original search query used to discover it in the storage unit 20, and periodically performs searches to check the survival status.

[0039] That is, the list management unit 15 periodically checks at a predetermined timing whether each URL included in the generated list is included in the search results using the same search query as when the discovery data collection unit 13 searched, and if it is not included in the search results, it deletes it from the list. In addition, the list management unit 15 may check at a predetermined timing for each URL included in the list whether the language used in the web page based on this URL has changed, and if there has been a change, determine that the URL has changed to a legitimate site (modified) and remove it from the list.

[0040] FIG. 2 is a flowchart illustrating the procedure of the list generation method according to this embodiment. Here, the domain to be extracted from the search results for learning and the search results for detecting dangerous sites is referred to as a TLD.

[0041] In step S1, the learning data collection unit 11 extracts N frequently occurring search queries in a web search service. In step S2, the learning data collection unit 11 collects learning search results for each of the N search queries extracted in step S1.

[0042] In step S3, the domain extraction unit 12 extracts frequently appearing TLDs from the N search results collected in step S2.

[0043] In step S4, the discovery data collection unit 13 extracts m search queries each consisting of multiple keywords that appear at the top of search results in each of n categories (not all categories, but art, entertainment, games, finance, etc.). In step S5, the discovery data collection unit 13 collects search results for discovering dangerous sites using the extracted total of n×m search queries.

[0044] In step S6, the dangerous site extraction unit 14 extracts URLs having TLDs different from the frequently appearing TLDs extracted in step S3 from among the URLs included in the search results for detecting dangerous sites collected in step S5. In step S7, the list management unit 15 stores the URLs extracted in step S6 in the storage unit 20 as a list of dangerous sites.

[0045] [Second embodiment] A second embodiment of the present invention will be described below. The functional configuration of the list creation device 1 of this embodiment is the same as that of the first embodiment (FIG. 1), but the processing content of the dangerous site extraction unit 14 is different.

[0046] The dangerous site extraction unit 14 applies an index such as TF-IDF or BM25 for evaluating the importance of words in documents, regarding multiple learning search results and discovery search results as documents and domains as words, thereby extracting URLs containing domains whose index scores exceed a predetermined threshold.

[0047] Specifically, the dangerous site extraction unit 14 regards the search results for detecting dangerous sites and the N pieces of learning data as a total of N+1 pieces of document data, and calculates the score (word importance) of each domain that appears in the search results for detecting dangerous sites. The dangerous site extraction unit 14 adds URLs with domains that have high scores to a list of dangerous sites.

[0048] According to the above-described embodiment, the list generation device 1 learns frequently occurring domains (TLDs or attribute-type domains) by obtaining search results using search queries that are top trending searches, and collects URLs that do not contain frequently occurring domains from the newly searched search results for discovering dangerous sites, regarding them as dangerous sites including tampered sites.

[0049] This allows the list creation device 1 to increase the number of dangerous sites in the list according to the number of search queries, thereby improving the efficiency of creating a list of dangerous sites, i.e., the comprehensiveness, compared to conventional methods. In addition, since the process can be completed by simply analyzing the domains of the URLs obtained as search results, time and labor costs are reduced. The URLs of dangerous sites collected in this way can be used as a blacklist to restrict user access. For example, by comparing URLs contained in emails with the list, dangerous sites can be detected without being accessed.

[0050] Here, according to the list generation method by the list generation device 1, when using search results in Japan, for example, it is possible that dangerous sites containing frequently used TLDs such as ".com" and ".jp" will be overlooked, but it is possible to collect more dangerous sites than conventional methods. Also, by applying the above method in different languages ​​or countries, it is possible to collect dangerous sites containing frequently used TLDs in Japan.

[0051] By analyzing TLDs, the list creation device 1 can easily extract risky sites from the URLs in the search results. Furthermore, by targeting attribute-type domains, the list creation device 1 is able to perform a more detailed analysis than by targeting TLDs, thereby improving the comprehensiveness of the list.

[0052] By including the site category in the search query for discovering dangerous sites or by using multiple search keywords, the list generation device 1 can appropriately narrow down the search conditions, increase the occurrence rate of dangerous sites, and discover them efficiently.

[0053] By setting an upper limit on the number of sites to be collected as search results for learning, the list creation device 1 can appropriately learn frequently occurring domains without increasing the processing load too much.

[0054] The list generation device 1 may apply existing indicators for evaluating the importance of words in documents to evaluate the importance of domains in search results for detecting dangerous sites, and extract URLs containing domains with high scores as dangerous sites. This allows the list creation device 1 to appropriately extract URLs that include domains that are unlikely to appear in the search results for learning but that appear in large numbers in the search results for detecting dangerous sites.

[0055] The list creation device 1 periodically checks whether each URL included in the list is searched for using the same search query as when it was discovered, and if the URL is no longer searched for, it can determine that the altered data for the site in question has been corrected. This allows the list creation device 1 to periodically update the list of dangerous sites and keep it in an appropriate state.

[0056] The list creation device 1 periodically checks whether the language used on the web page for each URL included in the list has changed, and if there has been a change, it can determine that the tampered data for the site in question has been corrected. This allows the list creation device 1 to periodically update the list of dangerous sites and keep it in an appropriate state.

[0057] Furthermore, the above-described embodiment can prevent access to dangerous sites, for example, and therefore contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "Develop resilient infrastructure, promote sustainable industrialization and foster innovation."

[0058] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the above-described embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

[0059] The list generation method by the list generation device 1 is realized by software. When realized by software, the programs that make up this software are installed on an information processing device (computer). These programs may be recorded on removable media such as CD-ROMs and distributed to users, or may be distributed by being downloaded to the user's computer via a network. Furthermore, these programs may be provided to the user's computer as a web service via a network without being downloaded. [Explanation of symbols]

[0060] 1 List Generator 10 Control Unit 11 Learning Data Collection Unit 12 Domain Extraction 13 Discovery Data Collection Unit 14 Dangerous site extraction section 15 List Management Department 20 Memory section

Claims

1. a learning data collection unit that collects a plurality of learning search results using a top predetermined number of frequently occurring search queries in a web search service; a domain extraction unit that extracts domains that frequently appear in the learning search results in accordance with a predetermined rule; a discovery data collection unit that collects search results for discovering dangerous sites using a search query consisting of multiple conditions; a dangerous site extraction unit that extracts URLs including domains different from the domains extracted by the domain extraction unit from the search results for discovering dangerous sites; a list management unit that manages the URLs extracted by the dangerous site extraction unit as a list of dangerous sites.

2. The list generating device according to claim 1 , wherein the domain extraction unit extracts top-level domains as the frequently occurring domains.

3. 2. The list generating device according to claim 1, wherein the domain extraction unit extracts pre-registered attribute type domains as the frequently occurring domains.

4. 4. The list generating device according to claim 1, wherein the discovery data gathering section includes a site category in the plurality of conditions.

5. 4. The list generating device according to claim 1, wherein the discovery data collecting section includes a plurality of search keywords as the plurality of conditions.

6. 4. The list generating device according to claim 1, wherein the learning data collecting unit sets an upper limit on the number of sites to be collected as the learning search results.

7. A list generation device as described in any one of claims 1 to 3, wherein the dangerous site extraction unit applies an index for evaluating the importance of words in documents by treating the multiple learning search results and the search results for discovering dangerous sites as documents and domains as words, and extracts URLs including domains whose score of the index exceeds a predetermined threshold.

8. A list generation device as described in any one of claims 1 to 3, wherein the list management unit checks at a predetermined timing for each URL included in the list whether it is included in a search result using the same search query as when searched by the discovery data collection unit, and removes it from the list if it is not included in the search results.

9. A list generation device as described in any one of claims 1 to 3, wherein the list management unit checks at a predetermined timing for each URL included in the list whether the language used in the web page based on that URL has changed, and if there has been a change, removes the URL from the list.

10. A learning data collection step of collecting a plurality of learning search results using a top predetermined number of frequently occurring search queries in a web search service; a domain extraction step of extracting domains that frequently appear in the learning search results according to a predetermined rule; A discovery data collection step of collecting search results for discovering dangerous sites using a search query including a plurality of conditions; a dangerous site extraction step of extracting URLs including domains different from the domains extracted in the domain extraction step from the search results for discovering dangerous sites; a list management step of managing the URLs extracted in the dangerous site extraction step as a list of dangerous sites, the list generation method being executed by a computer.

11. A list generating program for causing a computer to function as the list generating device according to any one of claims 1 to 3.

Citation Information

Patent Citations

  • Forged site detection method and computer program

    JP2007233904A

  • Methods and systems for analyzing data related to potential online fraud

    JP2008521149A

  • Analyzer, detector, system and program

    JP2021033421A

  • Multi-output logic circuit

    JP1985055726A