A specific type of magnet search website discovery detection method

Abstract

The application discloses a specific type of magnet search website discovery detection method, based on the obtained website static characteristics, the website is classified to obtain the magnet search website and the magnet navigation website, and then the dynamic characteristics such as the search return result of the website is analyzed by using the automatic test tool, the problem that the harmful and harmless magnet search website cannot be distinguished based on the static characteristics is effectively solved by using the dynamic analysis mode, the specific type of magnet search website can be accurately detected and discovered, and meanwhile, the website classification sample and the website illegal and irregular sample are also expanded, so that the specific type of magnet search website can be more accurately discovered and detected.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer network, and particularly relates to a specific type of magnet search website discovery and detection method. BACKGROUND

[0002] BitTorrent, BT for short, is the most popular peer-to-peer network file sharing protocol on the Internet. Magnet URI scheme is a program used for information retrieval and document download in a peer-to-peer network, that is, a uniform resource locator based on file metadata in a peer-to-peer network, which is usually composed of a protocol name, a resource locator, a Hash method name and a string of 40 hexadecimal digits.

[0003] Magnet search website, referred to as magnet search website, stores a large number of magnet links and related file information, and users can obtain the magnet link corresponding to the file by searching for the target file name. Magnet navigation website, referred to as navigation website, aggregates multiple magnet search websites and provides jump links to magnet search websites for users.

[0004] In recent years, with the rapid development of Internet information, a large number of users search for music, movies, software, e-books and scientific papers through magnet search websites, and download files through peer-to-peer network tools. At the same time, some magnet search websites provide illegal and irregular content such as pornography and violence in search results for users. However, due to the characteristics of magnet search websites such as domestic and foreign distribution and fast domain name change, the existing methods of discovering magnet search websites by relying on manual work and manually searching and detecting whether magnet search websites provide illegal and irregular content for users have the problems of timeliness and comprehensiveness. In order to maintain network information security, it is urgent to solve the problems of extensive, accurate and efficient discovery and detection of specific type of magnet search websites. SUMMARY

[0005] Therefore, the present application provides a specific type of magnet search website discovery and detection method, which realizes the discovery and detection of specific type of magnet search websites.

[0006] The specific type of magnet search website discovery and detection method provided by the present application comprises the following steps:

[0007] Step 1, extracting the website features of magnet search websites, magnet navigation websites and ordinary websites to establish a training sample data set; training the website classifier with the training sample data set to obtain a trained website classifier;

[0008] Step 2, obtaining all links on the page of the magnetic navigation website, and using the main domain name to remove the intermediate website to obtain the website features of the intermediate website; inputting the website features of the intermediate website into the website classifier to determine the magnetic search website or the magnetic navigation website; and taking all links on the page of the determined magnetic search website or the magnetic navigation website as the to-be-detected magnetic search website and the to-be-detected magnetic navigation website;

[0009] Step 3, according to the harmful magnetic search website and the harmful magnetic content search word, inputting the homepage search box of the harmful magnetic search website to obtain a search return result page and marking whether it is harmful, extracting the harmful link text and the file hash value of the link address as a suspected magnetic file hash value; inputting the search box of the to-be-detected magnetic search website and the to-be-detected magnetic navigation website to obtain a search return result page, extracting the file hash value of the link text and the link address, and comparing the file hash value with the suspected magnetic file hash value; if they are the same, it is determined that the magnetic search website provides harmful content, otherwise, it does not provide harmful content.

[0010] Further, the website classifier in the step 1 is a website classifier based on a support vector machine.

[0011] Further, the website features in the step 1 are described by using a website multi-feature data structure composed of a first-level domain name, a second-level domain name, an IP address attribution, an Alexa global comprehensive ranking, an Alexa China ranking, a domain name creation time, a website title, website keywords and website description.

[0012] Further, the website keywords have a weight TF-TDF, and the calculation method of the weight TF-TDF is as follows:

[0013] TF-TDF = TF w *IDF w

[0014]

[0015]

[0016] wherein, TF w is a TF value of the website keyword calculated by using a term frequency-inverse document frequency algorithm, and IDF w is an IDF value of the website keyword calculated by using the term frequency-inverse document frequency algorithm.

[0017] Further, the method for obtaining all links on the page of the known magnetic navigation website in the step 2 is to collect all links on the page of the known magnetic navigation website by using a breadth-first crawler.

[0018] Further, the website features of the intermediate website in step 2 are obtained in the following manner: the website IP address attribution is obtained by using the local domain name resolution and the network interface query method, and the domain name that cannot be resolved is marked as 0; the website Alexa ranking and domain name creation time are obtained by using the network interface query method, and the domain name that cannot be obtained is marked as 0; the website title is obtained by using the " / html / head / title" path expression; the website keywords are obtained by using the " / html / head / meta[@name='description']" path expression; and the website description is obtained by using the " / html / head / meta[@name='keywords']" path expression.

[0019] Further, the search box input in the homepage of the harmful magnet search website in step 3 is in the following manner: the input search box with the label name input and the type text on the page is obtained, the search term and the carriage return symbol are inputted, and then the link text is extracted by using the " / / a / text()" path expression and the link address is extracted by using the " / / a / @href" path expression in the returned result page.

[0020] Further, the file hash value of the link text and the link address in the returned result page of the search in step 3 is in the following manner: the file hash value with the length of 40 bits and the case-insensitive in the link address string is extracted by using the "[a-zA-Z0-9]{40}" expression.

[0021] Beneficial effects:

[0022] 1. The website is classified into a magnet search website and a magnet navigation website based on the obtained website static features, and then the dynamic features such as the website search returned result are analyzed by using the automatic test tool, so that the problem that the harmful and harmless magnet search websites cannot be distinguished based on the static features is effectively solved by using the dynamic analysis method, the specific type magnet search website can be accurately detected and found, the website classification sample and the website illegal and irregular sample are also expanded, and the specific type magnet search website can be more accurately detected and found;

[0023] 2. The website is classified according to the website static features by using the support vector machine classification model, the probability output can be provided, the training speed is fast, the overfitting risk is small, and the like, and the discovery and detection efficiency of the specific type magnet search website is further improved;

[0024] 3、The application calculates the weight of the keyword according to the frequency of the keyword, and the higher the weight is, the more representative the keyword is, and the stronger the identification of the feature is, so that some common words in the website are filtered as much as possible when the feature is extracted, and the website keywords with high weight are reserved, so that the accuracy of the website feature expression and the extraction efficiency are further improved. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 A specific type of magnetic search website discovery and detection method provided by the application is shown in the website feature extraction and classification process schematic diagram.

[0026] Figure 2 A specific type of magnetic search website discovery and detection method provided by the application is shown in the website feature extraction and classification process schematic diagram.

[0027] Figure 3 A specific type of magnetic search website discovery and detection method provided by the application is shown in the website feature extraction and classification process schematic diagram.

[0028] Figure 4 The detail page and address of the magnetic link of the existing magnetic search website are shown in the schematic diagram. DETAILED DESCRIPTION

[0029] The application will be described in detail below with reference to the drawings and examples.

[0030] The core idea of the specific type of magnetic search website discovery and detection method provided by the application is: first, the website classifier is trained according to the known magnetic search, navigation website and ordinary website samples; then, the search results of the known harmful magnetic search website and search keywords are obtained to build a specific type of magnetic search website verification tool; then, the breadth-first crawler starting from the magnetic navigation website is used to collect and classify the website to obtain the suspected magnetic file hash value in the main domain name deduplication mode; finally, the classified magnetic search and navigation website are detected by the illegal harmful magnetic search website by inputting the search keywords by using the automatic test tool.

[0031] The specific type of magnetic search website discovery and detection method provided by the application includes the following steps:

[0032] Step 1, according to the known magnetic search, navigation website and ordinary website sample set, the website features of the sample are extracted, and the features are represented and normalized to establish a training sample data set; the website classifier is trained by using the training sample data set, and the trained website classifier is obtained, as shown in the drawing. Figure 1 The ordinary website sample can be collected from the Alexa website ranking directory.

[0033] Step 1.1, establish a sample set, determine the website multi-feature data structure composed of primary domain name, secondary domain name, IP address location, Alexa global comprehensive ranking, Alexa China ranking, domain name creation time, website title, website keyword and website description.

[0034] According to the magnetic force search, navigation website and general website samples collected from the Alexa website ranking directory, a sample set is established. Through analysis of the samples in the sample set, it is found that the magnetic force search related websites have the following characteristics:

[0035] (1) The relatively small primary domain names such as “.cc”, “.top”, “.xyz”, “.co” and “.gq” are used a lot;

[0036] (2) Multiple websites share the same secondary domain name, such as “cilimao” and “zhongzilou”;

[0037] (3) Usually accessed in Hong Kong, China or using CLOUDFLARE CDN for domain name hosting;

[0038] (4) Alexa global comprehensive ranking is greater than the global threshold, and China ranking is greater than the domestic threshold;

[0039] (5) The domain name creation time is later than a certain time threshold; the website title, keyword and description usually contain words such as “magnetic force”, “search”, “engine”, “seed”, “download” and “link”.

[0040] Therefore, the website feature data structure determined by the application includes: website primary domain name, secondary domain name, IP address location, Alexa global comprehensive ranking, Alexa China ranking, domain name creation time, website title, website keyword and website description, and the specific content is shown in Table 1.

[0041] Table 1 website feature table corresponding field and explanation

[0042]

[0043] Step 1.2, based on the website multi-feature data structure, analyze the sample set to extract website features and establish a training sample data set.

[0044] For each sample in the data set constructed in step 1.1, extract URL features, website attribute features and website text features, construct a website feature vector, and then perform representation and normalization processing on the features; define positive samples and negative samples, and establish a training sample data set.

[0045] Wherein, the URL feature FURL includes a website first-level domain name fTLD, a website second-level domain name fSLD, and an IP address belonging place fip_loc.

[0046]

[0047] The website attribute feature F site includes an Alexa global ranking f A_R , an Alexa China ranking f A_CN , and a domain name creation time f Date . The specific definitions are shown in the following table:

[0048]

[0049] The website text feature F HTML includes a website title f title , a website keyword f keyword , and a website description f description . The specific definitions are shown in the following table:

[0050]

[0051] In order to further improve the accuracy of feature expression and the feature extraction efficiency, the application calculates the weight of the keyword according to the frequency of the keyword appearance, and the higher the weight is, the more representative the keyword is, and the stronger the identification of the feature is. In the feature extraction, some common words in the website are filtered as much as possible, and the website keyword with a higher weight is reserved. Specifically, the application adopts an improved term frequency-inverse document frequency algorithm (TF-IDF) to calculate the TF value of each keyword and the corresponding IDF value. Generally, the more the keyword appears in the text of the positive sample of the magnet search website, and the less the keyword appears in the text of the negative sample, the more representative the website keyword is. In the application, the calculation formula of the TF value and the IDF value of the website keyword is as follows:

[0052]

[0053]

[0054] Wherein, 1 is added to the denominator of IDFw to avoid the denominator being 0.

[0055] Then, the weight TF-TDF of the website keyword is calculated according to the TF value and the IDF value of the website keyword, and the calculation formula is: TF-TDF = TF*IDF. Through actual analysis, the website keywords with a higher weight TF-TDF include BT, BitTorrent, magnet, seed, Magnet, torrent, resource, etc.

[0056] Step 1.3, training the website classifier by using the training sample dataset, obtaining the trained website classifier.

[0057] After the conversion of the website attributes and characters into the feature vectors, a general machine learning model is constructed for classification training, and then the feature vectors of the websites generated in step 1.2 are used to train the SVM classifier.

[0058] The training sample dataset is represented as D i = (x i , y i ), i = 1, 2, …, n, wherein x i is the training sample, y i is the classification label of the training sample, and n is the total number of training samples. The classification label takes the values of "1" and "-1", "1" representing a torrent search website, and "-1" representing a website not belonging to a torrent search website. The training sample x i is taken as a point, and a certain type of website is represented by a classification hyperplane, so the distance from the training sample x i to the classification hyperplane is represented as δ i = y i (wx i +b), and the classification hyperplane can be defined as w T x+b=0, so the judgment of whether the training sample belongs to a certain type of website can be converted into the judgment of whether the value of g(x) = wx+b is greater than zero. The interval from the point to the classification hyperplane can be represented as At the same time, maximizing the classification interval is equivalent to minimizing w 2 , so the quadratic programming problem can be represented as

[0059] The website feature vectors of the training samples in the training sample dataset are input into the SVM classifier to obtain the output results 1 or -1, completing the training of the SVM classifier. Then, the trained SVM classifier is detected by using the test set, completing the training of the website classifier.

[0060] Step 2, starting from the known torrent navigation website, obtaining all the links on the page, using the main domain name to remove duplicates to obtain the websites to be analyzed, and then using the network interface and local domain name resolution method to obtain the website features of the websites to be analyzed; inputting the website features into the website classifier obtained in step 1 to determine which ones belong to torrent search or torrent navigation websites; for the determined torrent search or torrent navigation websites, continue to collect and access all the links and website features on the pages, and take them as the detected torrent search websites and the detected torrent navigation websites.

[0061] Specifically, the application prioritizes from a previously known magnetic navigation website, adopts breadth-first crawler to collect and access all links on the page, in the crawling process, for each website after the main domain name deduplication, adopts network interface and local domain name resolution method to obtain its website characteristics; then adopts the website classifier obtained in step 1 to identify the website belonging to magnetic search or magnetic navigation, continues to collect and access all links on the page, such as shown in Figure 2 Specifically, the application prioritizes from a previously known magnetic navigation website, adopts breadth-first crawler to collect and access all links on the page, in the crawling process, for each website after the main domain name deduplication, adopts network interface and local domain name resolution method to obtain its website characteristics; then adopts the website classifier obtained in step 1 to identify the website belonging to magnetic search or magnetic navigation, continues to collect and access all links on the page, such as shown in

[0062] Step 2.1, adopt breadth-first crawler to collect and access all links on the page of the magnetic navigation website and the website characteristics, adopt the following method to obtain the website characteristics: for the website URL, adopt the first and second domain name deduplication, that is, the URL with the same first and second domain name is identified as the same website; for the website IP address attribution, adopt the method of local domain name resolution and network interface query, the domestic website is accurate to the city, the foreign website is accurate to the country, and the domain name that cannot be resolved is marked as 0; for the website Alexa ranking and domain name creation time, adopt network interface query to obtain, and mark 0 for the one that cannot be obtained; the website title is obtained by adopting the path expression " / html / head / title"; the website keyword is obtained by adopting the path expression " / html / head / meta[@name='description']"; the website description is obtained by adopting the path expression " / html / head / meta[@name='keywords']".

[0063] Step 2.2, input the website characteristics obtained in step 2.1 into the website classifier obtained in step 1, so as to identify the website belonging to magnetic search or magnetic navigation.

[0064] Step 2.3, for the website belonging to magnetic search or magnetic navigation obtained in step 2.2, continue to adopt breadth-first crawler to collect and access all links on the page of the magnetic navigation website and the website characteristics, and record it as the magnetic search website to be detected and the magnetic navigation website to be detected.

[0065] Step 3, through the known harmful magnet search site and a plurality of harmful magnet content search words, using an automated testing tool, search box input on the homepage of the harmful magnet search site, get the link text and link address in the search return result page, mark whether the link text and link address are harmful; for the link text and link address marked as harmful above, extract the file hash value thereof as a suspected magnet file hash value; then search the search box of the magnet search site and magnet navigation site obtained in step 2, and input the search words, get the link text and link address in the search return result page, extract the file hash value of the link text and link address, and compare the file hash value with the suspected magnet file hash value. If they are consistent, it is determined that the magnet search site provides illegal harmful content, otherwise it is determined that the magnet search site does not provide illegal harmful content, as shown in Figure 3 The specific steps include the following steps:

[0066] Step 3.1, through the known harmful magnet search site and a plurality of harmful magnet content search words, using an automated testing tool, search box input on the homepage thereof.

[0067] The search box input uses the method of obtaining the input search box with the label name "input" and the type "text" in the attribute. After inputting the search words and the carriage return symbol, the link text and link address in the search return result page can be obtained. The link text is obtained by using the path expression " / / a / text()". The link address is obtained by using the path expression " / / a / @href".

[0068] All link texts and link addresses are marked for harm. For the link texts and link addresses marked as harmful above, the regular expression "[a-zA-Z0-9]{40}" is used to extract the file hash value with a length of 40 bits in the link address string, which is not case sensitive, as a suspected magnet file hash value. The reason why this can be done is that the detail page address of a magnet link on the current magnet search site is mostly in the form of http: / / xxxx.com / <40-bit infoHash>.html, as shown in Figure 4 .

[0069] Step 3.2, for the magnet search site and magnet navigation site obtained in step 2, first search for the search box. If it cannot be found, it is determined that the website is not a magnet search site. If it can be found, a plurality of known search words that can obtain harmful content are input through multiple rounds, the link text and link address in the search return result page are obtained, the file hash value in the link address string is extracted, and the file hash value is compared with the suspected magnet file hash value. If they are consistent, it is determined that the website provides harmful content.

[0070] To sum up, the above is only the preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting harmful magnetic force search site discovery, characterized by, The method comprises the following steps: Step 1, extracting website features of a magnet search website, a magnet navigation website and a common website to establish a training sample data set; training a website classifier by using the training sample data set to obtain a trained website classifier; Step 2, obtaining all links on a magnet navigation website page, removing the main domain name to obtain an intermediate website, inputting website features of the intermediate website into the website classifier to determine a magnet search website or a magnet navigation website, and taking all links on the determined magnet search website or magnet navigation website page as a magnet search website to be detected and a magnet navigation website to be detected; Step 3, inputting a search term into a homepage search box of a harmful magnet search website to obtain a search return result page and mark whether the search return result page is harmful, extracting a harmful link text and a file hash value of a harmful link address as a suspected magnet file hash value, inputting the search term into a search box of the magnet search website to be detected and the magnet navigation website to be detected to obtain a search return result page, extracting a file hash value of a link text and a link address, comparing the file hash value with the suspected magnet file hash value, and determining that the magnet search website provides harmful content if the file hash value is the same as the suspected magnet file hash value, otherwise, the magnet search website does not provide harmful content.

2. The method of claim 1, wherein the method further comprises: The website classifier in step 1 is a website classifier based on a support vector machine.

3. The method of claim 1, wherein the method further comprises: The website features in step 1 are described by using a website multi-feature data structure composed of a first-level domain name, a second-level domain name, an IP address belonging to a place, an Alexa global comprehensive ranking, an Alexa China ranking, a domain name creation time, a website title, website keywords and a website description.

4. The method of claim 3, wherein the harmful magnetic search site discovery detection method is characterized by, The website keywords have a weight TF-TDF, and the weight TF-TDF is calculated in the following manner: TF-TDF = TF w *IDF w wherein, TF w is the TF value of the website keyword calculated by taking the term frequency-inverse document frequency algorithm, IDF w is the IDF value of the website keyword calculated by taking the term frequency-inverse document frequency algorithm.

5. The method of claim 1, wherein the method further comprises: In step 2, the method for obtaining all links on a magnet navigation website page is to collect all links on a known magnet navigation website page by using a breadth-first crawler.

6. The method of claim 1, wherein the method further comprises: In step 2, the website features of the intermediate website are obtained in the following manner: the website IP address belonging to a place is obtained by using a local domain name resolution and a network interface query method, and a domain name that cannot be resolved is marked as 0; the website Alexa ranking and the domain name creation time are obtained by using a network interface query, and a value that cannot be obtained is marked as 0; the website title is obtained by using a " / html / head / title" path expression; the website keywords are obtained by using a " / html / head / meta[@name='description']" path expression; and the website description is obtained by using a " / html / head / meta[@name='keywords']" path expression.

7. The method of claim 1, wherein the method further comprises: determining whether the search query is a harmful magnetic search query. In step 3, the method for inputting a search term into a homepage search box of a harmful magnet search website is to obtain an input search box with a label name input and a type text on a page, input a search term and a carriage return symbol, and then extract a link text by using a " / / a / text()" path expression and extract a link address by using a " / / a / @href" path expression on a return result page.

8. The method of claim 1, wherein the method further comprises: In step 3, the search return result page is obtained by inputting the to-be-detected magnetic search website and the to-be-detected magnetic navigation website search box, and the file hash value of the link text and the link address is extracted in the following manner: the expression "[a-zA-Z0-9]{40}" is used to extract the file hash value of 40 bits in the link address string, which is not case-sensitive.

Images