A method for detecting dark web addresses

Abstract

The application discloses a dark web address detection method, and relates to the technical field of network security, and comprises the following steps: S100, a normal website top-level domain name library and a dark web website top-level domain name library are established; S200, a set of uncamouflaged dark web addresses is collected and is recorded as C0; a set of normal website addresses is collected and is recorded as S0; S300, all top-level domain names in C0 that are in the dark web website top-level domain name library are replaced by domain names in the normal website top-level domain name library respectively, so that a set of camouflaged dark web addresses C1, C2, C3, … CN is obtained, N is an integer, and is equal to the number of domain names in the normal website top-level domain name library; S400, C0, C1, C2, C3, … CN and S0 are combined to form training data sets {C0, S0}, {C1, S0}, {C2, S0}, {C3, S0}, … {CN, S0}, etc., the method disclosed by the application is suitable for the characteristics of the dark web, features suitable for dark web URL classification are proposed, and the dark web URL is detected by using dark web feature strings and dark web website URL features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, and in particular to a dark web address detection method. BACKGROUND

[0002] The "dark web" refers to a hidden network, which cannot be searched and accessed by ordinary netizens through conventional means, and needs to use some special software, configuration or authorization to log in.

[0003] The following problems exist in the detection process of the public security department on the website: 1. Many features cannot be applied to the classification of dark web URLs: some existing URL classification algorithms extract features from the web page corresponding to the URL. However, dark web websites need special software and network configuration to access, and cannot be directly accessed, so the features used for dark web URL classification cannot include the features of the web page; 2. Cannot quickly identify dark web websites: the existing URL classification algorithm takes the URL as the input of the classifier, and uses the classifier to calculate the category of the URL. This method does not fully utilize the dark web features, and the URL of the dark web website has its own feature string, which realizes the rapid detection of the dark web URL. For example, the URL in the Tor network usually takes ".onion" as the top-level domain name; 3. Cannot detect dark web websites disguised as normal websites: in order to evade supervision, the disseminators of dark web websites will replace some dark web feature strings to disguise as normal websites, so as to evade supervision. For example, the top-level domain name ".onion" of the dark web is replaced with ".edu" to disguise as a website of an educational institution. The URL feature of the replaced dark web website address also changes, and the dark web address cannot be accurately detected.

[0004] Therefore, the present application discloses a dark web address detection method, which is compared with the prior art. The method disclosed in the present application is aimed at the characteristics of the dark web, and can quickly identify the URL of the dark web website according to the dark web feature string. For the dark web URL disguised as a normal website, different disguised address training data sets are constructed, and the features extracted from the URL of the dark web website are used for the detection of the dark web URL, so that the disguised dark web URL can be effectively detected. SUMMARY

[0005] The present application aims to overcome the shortcomings of the prior art, and provides a dark web address detection method. Compared with the prior art, the method disclosed in the present application is aimed at the characteristics of the dark web, proposes features suitable for dark web URL classification, uses the feature of the URL of the dark web website for the detection of the dark web URL, quickly identifies the URL of the dark web website according to the dark web feature string, and can effectively detect the disguised dark web URL.

[0006] The present application is realized by the following technical scheme: a dark web address detection method, the detection method comprising the steps of:

[0007] S100: Establishing a normal website top-level domain name library and a dark web website top-level domain name library;

[0008] S200: Collecting a set of unmasked dark web addresses, denoted as C0; collecting a set of normal website addresses, denoted as S0;

[0009] S300: Replacing all top-level domain names in C0 that are in the dark web website top-level domain name library with domain names in the normal website top-level domain name library to obtain a set of masked dark web addresses C1, C2, C3, … CN, N being an integer equal to the number of domain names in the normal website top-level domain name library;

[0010] S400: Forming training data sets {C0, S0}, {C1, S0}, {C2, S0}, {C3, S0}, …, {CN, S0} from C0, C1, C2, C3, … CN and S0 respectively;

[0011] S500: Learning the training data sets using classification algorithms to obtain a plurality of classifiers T = {t0, t1, t2, t3, t4, …, tN}, T being a set of classifiers;

[0012] S600: Obtaining a website address to be detected, and first detecting whether the top-level domain name of the website address is in the dark web website top-level domain name library. If it is, marking the address as a dark web address, and if it is not, proceeding to the next step;

[0013] S700: Detecting the address using each classifier in T to obtain the probability of the address being detected as a normal website and the probability of the address being detected as a dark web website by each classifier, and the detection result set being and Pi represents the probability of the address being detected as a normal website by the i-th classifier, Pi represents the probability of the address being detected as a dark web website by the i-th classifier, and

[0014] S800: Dynamically setting the weight of each classifier according to the top-level domain name of the detected website and if the top-level domain name of the detected address corresponds to the i-th set of masked addresses, the value of αi will be greater than the weight of other classifiers, and the classifier is set as the associated classifier of the detected address; i

[0015] S900: Calculating the classification result after weighting all classifiers ​A classifier detection interval β is set, 0 < β ≤ 1; if W1-W2 ≥ (W1+W2)*β, it is detected as a normal website address, if W2-W1 ≥ (W1+W2)*β, it is detected as a dark web website address, and otherwise, it is detected as a suspected dark web website address.

[0016] Preferably, the normal website top-level domain name library includes cn, com, net, org, edu and top, and the dark web website top-level domain name library includes onion and i2p.

[0017] Preferably, in step S600, the website address to be detected is obtained by webpage acquisition, email acquisition or two-dimensional code acquisition.

[0018] Preferably, in step S500, the classification algorithm includes a decision tree algorithm, a neural network algorithm, a support vector machine algorithm or a logistic regression algorithm.

[0019] The present application discloses a dark web address detection method, compared with the prior art:

[0020] First, generate a dark web top-level domain name library and a normal website top-level domain name library, directly detect dark web addresses according to the dark web top-level domain name, for uncamouflaged dark web addresses, directly detect whether the top-level domain name is in the dark web top-level domain name library, if so, the uncamouflaged dark web address can be quickly detected;

[0021] Second, train multiple classifiers using the revised data set, and detect camouflaged dark web addresses by combining the results of multiple classifiers, revise the training data set according to the camouflage mode of the dark web address, so that the training data set can identify camouflaged dark web addresses; use classifiers generated by different training data sets to simultaneously detect URLs and determine whether they are dark web addresses;

[0022] Third, directly detect whether a URL is a dark web website without logging in through related software environment, which is fast and not affected by network environment. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The flowchart of the detection method in the present application. DETAILED DESCRIPTION

[0024] The embodiments of the present application will be described in detail below, which are implemented on the premise of the technical scheme of the present application, and detailed implementation modes and specific operation processes are given, but the protection scope of the present application is not limited to the following embodiments.

[0025] As shown in Figure 1 The present application discloses a dark web address detection method, which comprises the following steps:

[0026] S100: Establishing a normal website top-level domain name library and a dark web website top-level domain name library;

[0027] S200: Collecting a set of uncamouflaged dark web addresses, denoted as C0; collecting a set of normal website addresses, denoted as S0;

[0028] S300: Replacing all top-level domain names in C0 that are in the dark web website top-level domain name library with domain names in the normal website top-level domain name library to obtain a set of camouflaged dark web addresses C1, C2, C3, … CN, N being an integer equal to the number of domain names in the normal website top-level domain name library;

[0029] S400: Forming training data sets {C0, S0}, {C1, S0}, {C2, S0}, {C3, S0}, …, {CN, S0} from C0, C1, C2, C3, … CN and S0 respectively;

[0030] S500: Learning the training data sets using a classification algorithm to obtain a plurality of classifiers T = {t0, t1, t2, t3, t4, …, tN}, T being a set of classifiers;

[0031] S600: Obtaining a website address to be detected, first detecting whether the top-level domain name of the website address is in the dark web website top-level domain name library, if yes, marking the address as a dark web address, if not, going to the next step;

[0032] S700: Detecting the address using each classifier in T to obtain the probability of the address being detected as a normal website and the probability of the address being detected as a dark web website by each classifier, the detection result set being and Pi represents the probability of the address being detected as a normal website by the i-th classifier, Pi represents the probability of the address being detected as a dark web website by the i-th classifier, and

[0033] S800: Dynamically setting the weight of each classifier according to the top-level domain name of the detected website and if the top-level domain name of the detected address corresponds to the i-th camouflaged address set, the value of α i will be greater than the weight of other classifiers, and the classifier is set as the associated classifier of the detected address;

[0034] S900: Calculating the classification result after weighting all classifiers Setting a classifier detection interval β, 0 < β ≤ 1; if W1-W2 ≥ (W1+W2)*β, the address is detected as a normal website address, if W2-W1 ≥ (W1+W2)*β, the address is detected as a dark web website address, otherwise, the address is detected as a suspected dark web website address.

[0035] For the convenience of understanding, a specific embodiment is disclosed.

[0036] The embodiment discloses a detection method of dark web addresses, and the detection method comprises the following steps:

[0037] Step one: establishing a normal website top-level domain name library and a dark web website top-level domain name library. The normal website top-level domain name library comprises ".cn", ".com", ".net", ".org", ".edu", ".top" and the like, and the dark web website top-level domain name library comprises ".onion", ".i2p" and the like;

[0038] Step two: collecting a set of uncamouflaged dark web addresses, denoted as C0; and collecting a set of normal website addresses, denoted as S0;

[0039] Step three: replacing all top-level domain names in the dark web website top-level domain name library in C0 with top-level domain names in the normal website top-level domain name library to obtain a set of camouflaged dark web addresses C1, C2, C3, … For example, C1 corresponds to a set of addresses after replacing ".onion" and ".i2p" with ".cn", C2 corresponds to a set of addresses after replacing ".onion" and ".i2p" with ".com", C3 corresponds to a set of addresses after replacing ".onion" and ".i2p" with ".net", and the like;

[0040] Step four: combining C0, C1, C2, C3, … and S0 to form training data sets {C0, S0}, {C1, S0}, {C2, S0}, {C3, S0}, …;

[0041] Step five: learning the above training data sets by using a classification algorithm such as a decision tree algorithm to obtain a plurality of classifiers t0, t1, t2, t3, t4, … For example, t0 is a classifier obtained from {C0, S0}, t1 is a classifier obtained from {C1, S1}, and the like, and let T be a set of the above classifiers, T = {t0, t1, t2, t3, t4, …};

[0042] Step six: for a website address to be detected, first, detecting whether the top-level domain name of the website address is in the dark web website top-level domain name library. If yes, marking the address as a dark web address, and if no, going to the next step;

[0043] Step seven: detecting the address by using each classifier in T. Obtaining the probability of detecting the address as a normal website and the probability of detecting the address as a dark web website by each classifier. The detection result set is and Pi represents the probability of detecting the address as a normal website by the i-th classifier, Pi represents the probability that the i-th classifier detects the address as a dark web website, and

[0044] Step eight: dynamically set the weight of each classifier according to the top-level domain name of the detected website And if the top-level domain name of the detected address corresponds to the i-th set of pseudo addresses, the value of a i will be greater than the weight of other classifiers, and the classifier is called the associated classifier of the detected address.

[0045] Step nine: calculate the classification results after weighting all classifiers

[0046] Step ten: set the classifier detection interval β, 0 < β ≤ 1.

[0047] Step eleven: if W1-W2≥(W1+W2)*β, it is detected as a normal website address, if W2-W1≥(W1+W2)*β, it is detected as a dark web website address. Otherwise, it is detected as a suspected dark web website address.

[0048] In summary, compared with the prior art, the dark web top-level domain name library and the normal website top-level domain name library are generated, and the dark web address is directly detected according to the dark web top-level domain name. For the unmasked dark web address, directly detect whether the top-level domain name is in the dark web top-level domain name library, if it is, the unmasked dark web address can be quickly detected. Using the revised data set, a plurality of classifiers are trained, and the results of the plurality of classifiers are combined to detect the masked dark web address. According to the way of masking the dark web address, the training data set is revised so that the training data set can identify the masked dark web address. The classifiers generated by using different training data sets are used to detect the URL to determine whether it is a dark web address. According to the URL, it is directly detected whether it is a dark web website, without logging in through the related software environment, which is fast and not affected by the network environment.

[0049] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any skilled person in the art can make equivalent replacements or changes to the technical solutions and inventive concepts of the present application within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

[0050] It is to be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component" can include a combination of two or more components. Additionally, the terms "comprise," "comprises," and "comprising," or any variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Furthermore, unless otherwise indicated herein, the terms "first," "second," "third," etc., are used herein merely as labels, and are not intended to impose ordinal import.

Claims

1. A dark web address detection method, characterized by, The detection method comprises the steps of: S100: establishing a normal website top-level domain name library and a dark web website top-level domain name library; S200: collecting a set of uncamouflaged dark web addresses, denoted as C0; S300: replacing all top-level domain names in C0 that are in the dark web website top-level domain name library with domain names in the normal website top-level domain name library to obtain a set of camouflaged dark web addresses C1, C2, C3, … CN, N being an integer equal to the number of domain names in the normal website top-level domain name library; S400: combining C0, C1, C2, C3, … CN and S0 to form training data sets {C0, S0}, {C1, S0}, {C2, S0}, {C3, S0}, … {CN, S0}; S500: learning the training data sets by using a classification algorithm to obtain a plurality of classifiers T = {t0, t1, t2, t3, t4, … tN}, T being a set of classifiers; S600: obtaining a website address to be detected, first detecting whether the top-level domain name of the website address is in the dark web website top-level domain name library, if yes, marking the address as a dark web address, and if not, going to the next step; The normal website top-level domain name library comprises cn, com, net, org, edu and top, and the dark web website top-level domain name library comprises onion and i2p. S700: detecting the address with each classifier in T, obtaining the probability that each classifier detects the address as a normal website and the probability that each classifier detects the address as a dark web website, and the detection result set is and denotes the probability that the i-th classifier detects the address as a normal website, denotes the probability that the i-th classifier detects the address as a dark web website, and S800: dynamically set the weight of each classifier {a0, a1, a2, …, a N} according to the top-level domain name of the detected website i , And if the top-level domain name of the detected address corresponds to the i-th set of pseudo addresses, the value of a i is greater than the weight of other classifiers, set the classifier as the associated classifier of the detected address; S900: Calculate the classification result after weighting all classifiers Set the classifier detection interval β, 0 < β ≤ 1; if W1-W2 ≥ (W1+W2)*β, it is detected as a normal website address, if W2-W1 ≥ (W1+W2)*β, it is detected as a dark web website address, otherwise it is detected as a suspected dark web website address.

2. The dark web address detection method of claim 1, wherein, In step S600, the obtaining of the website address to be detected comprises obtaining from a webpage, obtaining from an email or obtaining from a two-dimensional code.

3. The darknet address detection method of claim 2, wherein, In step S500, the classification algorithm comprises a decision tree algorithm, a neural network algorithm, a support vector machine algorithm or a logistic regression algorithm.

4. The darknet address detection method of claim 1, wherein, ​

Images