Domain risk estimation system and method

Abstract

To estimate with high accuracy the risk of reaching a malignant URL in each domain of a web site in order to effectively prevent access to malignant URLs by users.SOLUTION: The present invention collects web access records of each user and repeatedly executes tracking accesses down to the immediately preceding one until an entry point is reached from a malignant URL on the basis of the collected access records and thereby reconstructs, for all malignant URLs, an access path leading from the entry point to the malignant URLs and estimates the risk of reaching a malignant URL, respectively, with regard to all domains on the constructed access path.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a domain risk estimation system and method for estimating the risk of reaching a malicious URL for each domain of a website. [Background technology]

[0002] The Internet is an essential social communication infrastructure, and many people access websites in their daily lives and for various business purposes. However, accessing these websites carries the risk of accessing malicious URLs, which can lead to malware infections and various types of attacks, such as social engineering attacks. To prevent access to such malicious URLs, blacklists that preregister malicious URLs have been used, but these blacklists cannot block all access to malicious URLs. For example, blacklists must be updated from time to time to register malicious URLs, but there are cases where users access malicious URLs before the updates.

[0003] The blacklists in which malicious URLs are registered are the accumulated know-how of each security vendor, and the criteria for determining whether a URL is malicious vary from one security vendor to another. For this reason, the blacklists of malicious URLs provided by each security vendor are fundamentally different, and there is no unified list. Furthermore, the current situation is that the blacklists of malicious URLs are either manually generated by analysts at each security vendor, or generated by analysts using tools and verifying them. At the research level, a technology has been proposed that uses machine learning technology to cluster URLs based on past labeling results to determine whether they are malicious or not, but it is still difficult to completely prevent users from reaching malicious URLs.

[0004] As described above, there is a limit to the method of blocking users' access to malicious URLs by only referring to a blacklist in which malicious sites are registered. Therefore, although there is a growing social demand for technology that effectively and accurately blocks users' access to malicious URLs without relying entirely on the blacklist, no technology that can meet this demand has yet been devised. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Terry Nelms etc ”WebWitness: Investigating, Categorizing, and Mitigating Malware Download Paths” 24th USENIX Security Symposium, August 12-14, 2015 · Washington, DC Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been devised in consideration of the above-mentioned problems, and its purpose is to provide a domain risk estimation system and method that can accurately estimate the risk of reaching a malicious URL for each domain of a website, in order to effectively prevent users from accessing malicious URLs. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the inventors have invented a domain risk estimation system that collects access records to websites of each user, detects the first access in a series of web access activities as an entry point, and repeatedly traces the previous access from each URL, including a malicious URL, of the website to the entry point based on the collected access records, refers to all the traced access histories, constructs an access path indicating the history of access to a series of URLs by the user from each URL to the entry point, and estimates the risk of reaching a malicious URL for each of the domains in the constructed access path.

[0008] A domain risk estimation system according to a first aspect of the present invention estimates, for each domain of a website, a risk that a user will reach a malicious URL after accessing the domain, the system comprising: an access record collection means for collecting records of access to the website by each user; an entry point detection means for detecting an initial access in a series of web access activities as an entry point; and a detection means for detecting an entry point from a malicious URL of the website based on the access records collected by the access record collection means. detection The method is characterized by comprising an access path construction means for constructing an access path indicating the user's access history of a series of URLs from each of the URLs to the entry point by repeatedly tracing the previous access until the entry point detected by the access path construction means is reached, and a risk estimation means for estimating the risk of reaching a malicious URL for each domain appearing in the access path constructed by the access path construction means.

[0009] A domain risk estimation system according to a second aspect of the present invention is characterized in that, in the first aspect of the present invention, the risk estimation means estimates the risk based on the following risk level equation (1). Risk level = Number of times that a user who accessed the estimated domain reaches a malicious URL on subsequent accesses / Total number of accesses to the domain... (1)

[0010] The domain risk estimation system of the third invention is characterized in that, in the second invention, the risk estimation means determines that a domain whose risk level is equal to or above a predetermined threshold is a risky domain, and further includes a warning notification means for notifying a warning to a user who attempts to access a risky domain determined by the risk estimation means.

[0011] The domain risk estimation system of the fourth invention is characterized in that, in the second invention, the risk estimation means determines that a domain whose risk level is equal to or above a predetermined threshold is a risky domain, and further includes control means for controlling so as to block user access to the risky domain determined by the risk estimation means.

[0012] A domain risk estimation system according to a fifth aspect of the present invention is the above-mentioned Access path construction method is characterized in that it tracks the previous access by attempting reload tracing, and if the previous access cannot be traced by the reload tracing and if a source tab ID is recorded in the access record, it tracks the access within the tab in which the source tab ID is set, and if a source tab ID is not recorded in the access record, it tracks the access within the same tab that has a common tab ID, which is an identifier of the tab, and further if the previous access cannot be traced within the same tab, it tracks the access within all other tabs.

[0013] A domain risk estimation method according to a sixth aspect of the present invention is a method for estimating, for each domain of a website, a risk that a user will reach a malicious URL after accessing the domain, the method comprising the steps of: collecting access records of each user to the website; detecting an initial access in a series of web access activities as an entry point; and detecting an entry point from a malicious URL of the website based on the access records collected in the access record collection step. detection By repeatedly tracing the previous access until the entry point detected by the step is reached, an access path is constructed that shows the user's access history to a series of URLs from each of the above URLs to the entry point. path The method includes a construction step, and a risk estimation step of estimating the risk of reaching a malicious URL for each domain that appears in the access path constructed by the access path construction step. Effect of the Invention

[0014] According to the present invention having the above-mentioned configuration, when a user reaches a domain that is likely to lead to a malicious URL in the next or subsequent accesses, a warning can be issued or communication can be blocked at that domain. This makes it possible to prevent a user from reaching a malicious URL. That is, according to the present invention, it is possible to detect a domain that will ultimately lead to a malicious URL even if it does not host malicious content. This makes it possible to avoid access to a malicious URL that has not yet been registered on a blacklist by blocking access at a dangerous domain that exists before the malicious URL. As a result, it becomes possible to block user access to malicious URLs with high efficiency and accuracy. [Brief description of the drawings]

[0015] [Figure 1] 1 is a diagram showing the overall configuration of a domain risk estimation system to which the present invention is applied. [Diagram 2] FIG. 1 is a diagram showing a process of a domain risk estimation system to which the present invention is applied. [Diagram 3] FIG. 13 shows a detailed flow of the access tracking phase. [Figure 4] FIG. 2 is a diagram for explaining risky domains identified in the present invention. [Diagram 5] FIG. 11 is a diagram for explaining an example of calculating a risk level. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, a domain risk estimation system to which the present invention is applied will be described in detail with reference to the drawings.

[0017] 1 shows the overall configuration of a domain risk estimation system 1 to which the present invention is applied. The domain risk estimation system 1 includes a user terminal 2 and a server 4, each of which is connected to a public communication network 10 formed of an Internet line.

[0018] The user terminal 2 is composed of a terminal device capable of accessing the Web, such as a desktop personal computer, a notebook PC, a mobile terminal, a smartphone, a tablet terminal, or a wearable terminal.

[0019] The server 4 collects records of access to websites by each user terminal 2. The server 4 performs access tracking, which will be described later, based on the collected access records. From the tracked accesses, the server 4 estimates the risk of reaching a malicious URL for each domain.

[0020] The processing operations performed by the server 4 up to estimating the risk of reaching a malicious URL for each domain will be described below.

[0021] 2 is a diagram showing the process of the domain risk estimation system 1 to which the present invention is applied. The processing operations of the domain risk estimation system are roughly classified into an access record collection phase A, an access tracking phase B, and a risk estimation phase C.

[0022] In the access record collection phase A, the server 4 collects the access records from each user terminal 2 to the website via the public communication network 10. The server 4 temporarily stores the collected access records. Examples of data of the access records collected by the server 4 include, for example, the URL of the HTTP request, a timestamp indicating the time of request issuance, a referrer, a tab ID identifying a browser tab (its uniqueness is guaranteed only within the same session), a URL (tab URL) indicated in the browser tab, a resource type identifying a main frame, an image, a font, a script, etc., a page transition type identifying bookmark access, reloading, URL typing, etc., a user ID, an ID of the tab from which the current tab was generated (a source tab ID) (for example, the source tab ID is recorded when a new window is generated by target="_blank" or window.open(), etc.), etc. For the resource type and the page transition type, values ​​defined in the Google Chrome APIs webRequest and history may be directly acquired. In addition, the accessed URL is evaluated using the data collected from each user terminal 2, and the evaluation result is stored. For example, the URL's Google Safe Browsing (GSB) evaluation result, Alexa Traffic Rank ranking, etc. After the access records are collected by the server 4 in this manner, the process proceeds to the access tracking phase B.

[0023] In the access tracking phase B, the server 4 repeatedly tracks the previous access from each malicious URL until it reaches the entry point based on the collected access records. This entry point is an access that is not consecutive to the previous access, in other words, the URL that the user first accessed. In other words, this entry point is the first access that is not an access that follows a link, for example, the URL that is first accessed through a bookmark access or a search by a search engine, etc.

[0024] The outline of this access tracking phase B is as shown in Figure 2. First, it is confirmed whether or not there is an untracked malicious URL. If there is no untracked malicious URL, the process proceeds to risk estimation phase C. On the other hand, if there is an untracked malicious URL, the previous access record is tracked. When the previous access record can be tracked, it is determined whether or not it corresponds to an entry point. If it corresponds to an entry point, the access history from the entry point to the malicious URL is saved as an access path, and the process of checking whether or not there is an untracked malicious URL is repeated. On the other hand, if it does not correspond to an entry point, the tracking of the previous access record is repeated. Details of this access record tracking phase B will be explained below.

[0025] 3 shows a detailed flow of this access tracking phase B. When transitioning to access tracking phase B, first in step S11, reload tracking is attempted to track the previous access. In such a case, it is determined whether the page transition type is "reload". As a result, if the page transition type is "reload", it is determined that the current page is a reloaded page, and reload tracking is performed in step S12. On the other hand, if the page transition type is other than "reload", transition is made to step S13.

[0026] In the reload trace in step S12, a so-called reload trace within the same tab is performed to search for the same tab URL as the current access record and the most recent access record having the same URL. In such a case, reload trace is performed to investigate the access records within the same tab, and only if there is no match, investigate the access records of other tabs. After the reload trace is completed, the processing operation is terminated.

[0027] When the process proceeds to step S13, it is determined whether or not a source tab ID is set in the access record. If a source tab ID is set, the process proceeds to step S14, and if a source tab ID is not set, the process proceeds to step S15.

[0028] When the process proceeds to step S14, so-called source tab tracking is performed, which tracks accesses within the tab specified by the source tab ID. In this case, if a source tab ID is specified, it is determined that the current access was generated from the specified tab, and source tab tracking is performed to track the access records within that tab. If referer information is available, referer tracking is performed, which selects, as the previous access record, the access record whose URL matches the referer from among the access records between the current access and the immediately previous mainframe access. On the other hand, if referer information is not provided or an appropriate record cannot be found, so-called mainframe tracking is performed, which selects the immediately previous access record to the mainframe as the previous access record. After mainframe tracking is completed, the processing operation ends.

[0029] When the process proceeds to step S15, same-tab tracking is performed to track the previous access record within the same tab. Here, the same tab refers to a tab with the same tab ID, which is an identifier for the tab, and tracking within the same tab means analyzing and tracking only logs with matching tab IDs. As with the source tab tracking method, the records are analyzed using referrer tracking and mainframe tracking technology, and the previous access record is identified from the past access records of the user on the currently displayed tab. As a result, if the previous access can be found in step S16, the processing operation is terminated, but if the previous access cannot be found, the process proceeds to step S17.

[0030] When the process proceeds to step S17, all tab tracking is performed to track all access records of the user. In all tab tracking, referrer tracking is performed first if referer information is present, as with source tab tracking and same tab tracking. In cases where the referer holds only origin information, all tab tracking determines the access record of the immediately preceding main frame having a URL with a matching origin as the previous access, that is, performs origin tracking. Note that in source tab tracking in step S14 and same tab tracking in step S15, origin tracking is not performed because the previous access can be identified with high accuracy by mainframe tracking. If an appropriate entry cannot be found through the above processing operations, mainframe tracking may be further performed in this all tab tracking.

[0031] In this way, in this access tracking phase B, reload tracking, source tab tracking, and same tab tracking, which are more efficient at tracking access than all tab tracking, are given priority, and all tab tracking, which is less efficient and does not necessarily provide highly reliable tracking, is given a lower priority. Therefore, according to the present invention, this access tracking can be performed with high efficiency.

[0032] When the previous access record can be traced through the process shown in steps S11 to S17 described above, it is determined each time whether or not it corresponds to an entry point. If the traced access corresponds to an entry point, the series of access history from the malicious URL to the previous access, which has been traced back repeatedly, is saved as an access path. On the other hand, if the traced access does not correspond to an entry point, an attempt is made to trace the previous access record, and in such a case, the process shown in steps S11 to S17 is also executed. This process is executed repeatedly until the entry point is reached.

[0033] The access paths created as described above are generated and saved for all access records to malicious URLs.

[0034] The determination of whether a tracked access corresponds to an entry point may be based on whether it corresponds to a bookmark access, a session reconstruction, a web search, an omnibar access, a direct address input, or a start page access, as described below.

[0035] Bookmark access is determined based on whether the page transition type is "auto_bookmark". If the page transition type is "auto_bookmark", it is determined that the user selected the bookmark for the current page and visited it, and the page is recognized as the entry point.

[0036] A session is determined to have been reconstructed if the page transition type is "reload" and there has been no access for a certain period of time in the same tab. Browsers can reconstruct sessions by restoring recently closed browser tabs, for example, and if it is determined that the session has been reconstructed, this is recognized as the entry point.

[0037] When the URL matches the top page of a major search engine, or the page transition type is "form_submit" and the URL is the search result page of a major search engine, it is determined that a web search has been performed and this is recognized as the entry point. Note that this web search does not include searches within the site.

[0038] Omnibar access in the Chrome browser combines the functions of a search box and an address bar, but when the page transition type is "generated", it is determined that the access is via the omnibar and is recognized as an entry point.

[0039] A typed address occurs when a user edits the current URL in the address bar to move to another page, such as the top page, as part of a continuous browsing activity. Only when the page transition type is "typed" and the domain of the previous page and the current page are different, is this recognized as a typed address and is determined as an entry point.

[0040] When the page transition type is "start_page", the start page access determines that the page was opened by launching the browser, using a program argument or default settings, and recognizes this as the entry point. Note that even if you click a link on an external app, Chrome is launched on the OS with the URL of the link as an argument, so the page transition type will be "start_page".

[0041] Table 1 shows an example of access tracking performed as described above in the access tracking phase B. In Table 1, for each access time, in addition to the tab ID and the actually accessed URL, the source tab ID is shown if one was assigned, as well as the page transition type, resource type, and tracking method.

[0042] [Table 1]

[0043] Next, the process moves to risk estimation phase C. Figure 4 is a diagram for explaining risky domains identified in the risk estimation phase, and shows, as an example, the page transitions made by a user who accessed domain P. In this diagram, after accessing domain P, the user accesses domain Q or domain R, and then from domain Q to domain S or URL22, and from domain S to URL21. It also shows that from domain R, the user accesses URL23, domain T, and URL26, and from domain T to URLs 24, 25, and 27.

[0044] From the access paths generated in access tracking phase B, we can see that a certain user accessed domains S, Q, and P before accessing URL 21. Similarly, from another access path generated in access tracking phase B, we can see that another user accessed domains T, R, and P before accessing URL 24. By analyzing the access paths generated in access tracking phase B, we can understand the access tree structure as shown in Figure 4.

[0045] In this case, there may be cases where the URLs at the end are malicious. In the example of Fig. 4, URL21, URL22, URL25, and URL27 are all malicious. Such malicious URLs may be identified by referring to a blacklist in which malicious URLs are registered.

[0046] This access path is constructed by the server 4, but it is not essential to draw a path diagram such as that shown in Figure 4 above so that it can be displayed to the system administrator; it is sufficient if the access path generated in the access tracking phase B is prepared.

[0047] As explained above, in the risk estimation phase C, the risk of reaching a malicious URL is estimated for each of the domains in the access path constructed by the access path construction means. In the example of Fig. 4, the risk that the URL of the next access to domains Q, R, S, and T is a malicious URL is estimated. If the URL of domains Q, R, S, and T is a malicious URL, there is a possibility that the malicious URL will be reached when the domain is accessed. In the risk estimation phase C, among these, the domains that are highly likely to reach a malicious URL in the next or subsequent accesses are identified as risky domains.

[0048] As a method for estimating the risk of reaching a malicious URL for each domain, the risk may be estimated based on the following risk level formula (1).

[0049] Risk level = Number of times that a user who accessed the estimated domain reaches a malicious URL on subsequent accesses / Total number of accesses to the domain... (1)

[0050] In other words, this risk level indicates the probability that the next access to a domain will reach a malicious URL out of the total number of accesses to the domain. The next access includes not only the second access after the first access, but also the third access and all subsequent accesses, and the number of accesses to a malicious URL is counted.

[0051] For example, suppose that there are six total accesses to domain X, URL31, URL32, URL33, URL34, URL35, and URL36, as shown in Figure 5, and four of the six accesses reach a malicious URL. The remaining two accesses reach a benign URL. The risk level in this case is 4 / 6 = 66.7% based on formula (1) above.

[0052] After actually calculating such a risk level, the risk level may be compared with a preset threshold, and if the risk level is equal to or higher than the threshold, the domain may be judged as dangerous, and if the risk level is lower than the threshold, the domain may not be judged as dangerous. The threshold can be set freely by the system operator, and it becomes possible to decide whether to classify as malicious domains those domains that have even the slightest possibility of reaching a malicious URL, or only those that have a very high possibility of reaching a malicious URL.

[0053] In the example of Figure 4, an access path is depicted that is constructed from multiple access paths that reach seven different URLs, four of which are malicious URLs, and accesses through domain Q and domain S all reach the malicious URLs. Therefore, these domains are considered to be risky domains. Accesses through domain T reach both malicious and safe URLs, but if the risk level of domain T reaches a malicious URL with a probability equal to or higher than a threshold, it is considered to be a risky domain.

[0054] In the present invention, the determination of such dangerous domains is not limited to referring to the risk level described above, but may be made by any method that refers to the access path and is based on whether or not a malicious URL is reached in a subsequent access made by a user after visiting each domain in a series of accesses.

[0055] After calculating the risk level, the degree of danger may be determined in three or more stages, in addition to determining whether the domain is dangerous or not using a threshold value. In such a case, the threshold value may be set in two or more stages, and the degree of danger of the dangerous domain may be determined depending on whether the domain is above a certain threshold value and below a certain threshold value.

[0056] A warning may be issued to a user attempting to access a dangerous domain determined through the risk estimation phase. In such a case, the server 4 may notify each user terminal 2 of the dangerous domain, and when the user attempts to access or requests an access to the dangerous domain at each user terminal 2, the user terminal 2 may be alerted to the high risk. Alternatively, when a user attempts to access a dangerous domain at each user terminal 2, communication may be controlled to block the user's access to the dangerous domain.

[0057] In this way, the domain risk estimation system 1 to which the present invention is applied can warn users when they reach or attempt to access a domain that may lead to a malicious URL, or can block communication to that domain. This makes it possible to prevent users from reaching a malicious URL in advance.

[0058] In particular, individual domains themselves often do not host malicious content, and such domains are often not included in blacklists. Among the subsequent URLs visited after accessing each domain, there are many cases where URLs that should be identified as malicious are present, even if they are not currently identified as malicious. Therefore, by blocking communication at this domain, the possibility of users reaching malicious URLs can be reduced more than ever before.

[0059] Furthermore, even if the domain itself is not malicious but links to a malicious URL several hops away, it has not been registered on the blacklist. However, with the present invention, since it is possible to identify dangerous domains as described above, it is also possible to register and manage the dangerous domain itself on the blacklist. [Explanation of symbols]

[0060] 1. Domain Risk Estimation System 2. User terminal 4 Server 10 Public communication network

Claims

1. A domain risk estimation system that estimates a risk of a user arriving at a malicious URL after accessing each domain of a website, comprising: An access record collecting means for collecting a record of access to a website by each user; an entry point detection means for detecting an initial access in a series of web access activities as an entry point; an access path construction means for constructing an access path indicating a user's access history to a series of URLs from each URL to the entry point by repeatedly tracing the previous access from the malicious URL of the Web site to the entry point detected by the entry point detection means based on the access records collected by the access record collection means; and a risk estimation means for estimating the risk of reaching a malicious URL for each domain appearing in the access path constructed by the access path construction means. A domain risk estimation system characterized by:

2. The risk estimation means estimates the risk based on the following risk level formula (1):

2. The domain risk estimation system according to claim 1, Risk level = the number of times a user who accessed the estimated domain reaches a malicious URL in subsequent accesses / the total number of accesses to the domain (1)

3. The risk estimation means determines a domain whose risk level is equal to or higher than a predetermined threshold as a dangerous domain, Further, a warning notification means is provided for notifying a user who is about to access a dangerous domain determined by the risk estimation means.

3. The domain risk estimation system according to claim 2,

4. The risk estimation means determines a domain whose risk level is equal to or higher than a predetermined threshold as a dangerous domain, Further, a control means is provided for controlling the blocking of access by a user to a dangerous domain determined by the risk estimation means.

3. The domain risk estimation system according to claim 2,

5. The access path construction means includes: By attempting reload tracking, the previous access is tracked, If the previous access cannot be traced by the reload trace and if a source tab ID is recorded in the access record, trace the access within the tab to which the source tab ID is set, If the source tab ID is not recorded in the access record, the accesses within the same tab with a common tab ID, which is an identifier of the tab, are tracked. Furthermore, if the previous access cannot be traced within the same tab, accesses within all other tabs will be traced.

2. The domain risk estimation system according to claim 1,

6. A domain risk estimation method for estimating a risk that a user will arrive at a malicious URL after accessing each domain of a website, comprising: an access record collecting step of collecting a record of access to a website by each user; an entry point detection step of detecting an initial access in a series of Web access activities as an entry point; an access path construction step for constructing an access path showing a user's access history to a series of URLs from each URL to the entry point by repeatedly tracing the previous access from the malicious URL of the Web site to the entry point detected by the entry point detection step based on the access records collected by the access record collection step; and a risk estimation step of estimating a risk of reaching a malicious URL for each domain appearing in the access path constructed by the access path construction step. A domain risk estimation method comprising:

Images