A network crawler-based illegal domain name detection method, system, device and medium

Abstract

A network crawler-based illegal domain name detection method, system, device and medium, the method is: through the network crawler technology, all external links and their contents under the target website domain name are crawled; all obtained external links are searched in the illegal link database, and whether the external link exists in the illegal link database is detected; if it is detected that the external link has been in the illegal link in the illegal link database, an early warning and feedback link is entered; if it is detected that the external link is not in the illegal link database, the external link is submitted to a webpage content review interface for review; an audit result information is obtained from an audit API interface using a thread different from the audit application, and the returned information is analyzed and processed; the system, device and medium can realize illegal domain name detection based on the network crawler-based illegal domain name detection method; the application realizes detection of the nature of the external link while obtaining all internal links and external links under the website by using the crawler, is flexible and efficient, and has universality.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of Internet security detection, and particularly relates to an illegal domain name detection method, system, device and medium based on a web crawler. BACKGROUND

[0002] In the current website administrator work, it is difficult to clean up the website dead chain and dangerous link in time. The domain names that are disabled but not canceled will be left in various places of the webpage, and will be occupied by the lawbreakers and modified as bad information. This phenomenon is more harmful in the website with large access volume.

[0003] In the implementation of link detection and harmful domain name detection technology, the existing technologies include:

[0004] Patent application CN202011280258 discloses a link detection method and device, which obtains a plurality of classification tags from a classification tag database based on a JAVA proxy object, and stores the classification tags in the classification tag database in a tree-shaped cascading form. At least according to the plurality of classification tags, a target link is generated, and then a data acquisition request is sent to a data server based on the target link. The data acquisition response returned by the data server according to the data acquisition request is received. According to the data acquisition response, it is determined whether the target link is an invalid link. In the case where it is determined that the target link is an invalid link according to the data acquisition response, it is determined whether the number of times of sending the data acquisition request to the data server reaches a preset number of times. In the case where the number of times of sending does not reach the preset number of times, the data acquisition request is sent to the data server again, and the data acquisition response returned by the data server according to the data acquisition request is received. According to the data acquisition response returned by the data acquisition request, it is determined whether the target link is an invalid link. In the case where the number of times of sending reaches the preset number of times, it is determined that the target link is an invalid link. However, this scheme can only identify whether the link is invalid, and cannot identify the security of the link content. SUMMARY

[0005] In order to overcome the deficiencies of the above-mentioned prior art, the purpose of the present application is to provide an illegal domain name detection method, system, device and medium based on a web crawler. The crawler is used to obtain all internal and external links of the website, and the nature of the external link is detected at the same time. Through the website link detection, an illegal link database is automatically generated without manual operation. The multi-threading technology is used to reduce the time difference between sending a network request and obtaining a network data response, and improve the running efficiency. The database can be operated at the same time, which is flexible and efficient.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] A network crawler-based illegal domain name detection method, specifically comprising the following steps:

[0008] S1, through the network crawler technology, all external links and their contents under the target website domain name are crawled;

[0009] S2, all external links obtained are searched in the illegal link database, and whether the external link exists in the illegal link database is detected; if it is detected that the external link has been in the illegal link in the illegal link database, the pre-warning and feedback link is directly entered;

[0010] S3, if it is detected that the external link is not in the illegal link database, the external link detected not in the illegal link database is submitted to the webpage content review interface for review;

[0011] S4, after the review application is submitted in step S3, the review result information is obtained from the review API interface using a thread different from the review application, and the returned information is analyzed and processed.

[0012] The specific method of step S1 is: through the Python network crawler technology, the webpage source code is obtained by using the request library, and all links under the target webpage are obtained by using the re regular expression, the internal link sub-interface in the same domain name is accessed in multiple threads by using the multi-thread thread library, and the child link contained in the sub-interface is obtained, then all the links of the pages represented by the internal links are crawled, and the cycle is repeated until all the external links under the domain name are obtained.

[0013] The specific method of step S2 is: the step S1 is searched in the illegal link database, and the select statement is used to query whether the link exists in the illegal link database; if it is detected that the external link has been in the illegal link in the illegal link database, the target website administrator is alarmed, the target website administrator is informed to remove the dangerous link in the website, and the target website administrator can view the illegal link database content in real time and add or delete content

[0014] The specific method of step S3 is: if it is detected that the external link is not in the illegal link database, the URL of the external link detected not in the illegal link database is submitted to the webpage content review interface in parallel through the thread multi-thread, and whether the external link is a harmful link or an advertisement link is judged through the review API interface provided by the service provider providing the webpage content review interface.

[0015] The step S4 is specifically: after the step S3 of submitting the review application, using a thread different from the step S3 to obtain the review result information returned by the webpage content review interface service provider from the review API interface, the review result information including whether the webpage contains dangerous content and the classification of the content, including yellow, advertisement and other dangerous content; according to the returned review result information, the illegal website containing dangerous content is added to the illegal link database through the database instruction insert, and a pop-up window is used to warn the administrator.

[0016] The illegal link database is a MySQL database established based on the dangerous links fed back by authoritative agencies (including China Telecom) and gradually adding suspicious links and dangerous links obtained through website detection.

[0017] The step S1 uses the network crawler technology to crawl all external links under the target website domain name, and in the case that the target website administrator opens the illegal link database permission, all external links can be obtained by directly performing global search in the illegal link database to replace.

[0018] An illegal domain name detection system based on a network crawler, comprising:

[0019] A database module for storing illegal links and pre-checking based on the database before each link detection, which can improve the detection efficiency;

[0020] A review submission module for submitting all external links crawled by multiple threads to the review interface;

[0021] An audit query module for asynchronously obtaining review information from the review interface and transmitting the result to the database module.

[0022] An illegal domain name detection device based on a network crawler, comprising:

[0023] A memory for storing a computer program;

[0024] A processor for executing the computer program to implement the illegal domain name detection method based on the network crawler in steps S1 to S4.

[0025] A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the illegal domain name detection based on the network crawler according to the detection method in steps S1 to S4.

[0026] The advantages of the present application are:

[0027] 1.The application utilizes a crawler to obtain all internal and external links under a website while achieving detection of the nature of external links: after obtaining all internal links and external links under a certain website through crawler technology, the external links are audited, the audit includes pre-examination in the dangerous link database and submission of suspicious links to the audit interface of the web content service provider and obtaining the audit results, while identifying the link danger, the linked content security is identified.

[0028] 2.The application can automatically generate an illegal link database through website link detection, without manual operation, saving time and being efficient.

[0029] 3.The application uses the dangerous websites fed back by authoritative agencies in the early stage as the basis, uses MySQL to establish an illegal link database, and adds the illegal links obtained after each search and detection to the database, and after a large number of training and accumulation of a certain number of illegal links, the detection efficiency is greatly improved.

[0030] 4.The application utilizes the thread library of Python for multi-threading, and obtains different webpage sub-links and grandchild links in each sub-thread; all external links under the website are submitted to the audit interface in each sub-thread, and the audit results are obtained.

[0031] 5.The application can operate the database while running, and is flexible and efficient: after obtaining the audit results to confirm that a certain external link is a dangerous link, the link can be immediately added to the MySQL database, and a pop-up window alarm is initiated to the administrator to notify him / her to delete the link.

[0032] Compared with the prior art, the application has the following advantages:

[0033] 1.In terms of obtaining all links under the website, the application uses Python crawler technology, uses the requests library to obtain website data responses, adds all links under a single webpage to a multi-threaded queue, and uses the thread library to establish multi-threading, continues to add the child links under the parent links in the queue to the queue, thereby obtaining all internal and external links under the website, and filtering all internal links.

[0034] 2.In terms of harmful link detection, the application uses the deep learning image recognition interface of the network audit service provider (Tencent cloud content security interface) to establish threads, asynchronously submits the audit and obtains the audit results asynchronously, and realizes non-blocking efficient parallel link content audit.

[0035] 3. Utilize multi-threading to reduce the time difference between submitting a network request and obtaining a network data response, and improve the efficiency of obtaining all internal links and external links under the website. Overcome the time difference between submitting a network request using request and obtaining network source code under a single thread. Using multi-threading, multiple different network requests can be initiated during this time, improving the efficiency of obtaining web page source code. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The flowchart of the present application.

[0037] Figure 2 The detection main interface of the present application.

[0038] Figure 3 The illegal link database management interface of the present application. DETAILED DESCRIPTION

[0039] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0040] Referring to the drawings Figure 1 A network crawler-based illegal domain name detection method, specifically comprising the following steps:

[0041] S1, through the Python network crawler technology, using the request library to obtain the web page source code, and through the re regular expression to obtain all the links under the target web page, through the multi-threading thread library, in multiple threads, the internal link sub-interfaces in these links under the same domain name are accessed step by step, and the child links contained by the sub-interfaces are obtained, then all the links of the pages represented by these internal links are crawled, and the cycle is repeated until all the external links under the domain name are obtained. Utilize multi-threading to reduce the time difference between submitting a network request and obtaining a network data response, and improve the efficiency of obtaining all internal links and external links under the website. Overcome the time difference between submitting a network request using request and obtaining network source code under a single thread. Using multi-threading, multiple different network requests can be initiated during this time, improving the efficiency of obtaining web page source code.

[0042] S2, find all external links under the target web page domain name crawled in step S1 in the illegal link database, and query whether the link exists in the illegal link database through the select statement; if it is detected that the external link has been in the illegal link in the illegal link database, an alarm is sent to the target website administrator, notifying the target website administrator to remove the dangerous link in the website, at the same time, the target website administrator can view the content of the illegal link database in real time and add or delete the content; if it is detected that the external link is not in the illegal link database, then enter the next step.

[0043] S3, through thread multithreading, the URL of the external link detected in step S2 and not in the illegal link database is submitted in parallel to a webpage content review interface (such as a Tencent cloud content security interface, an Ali cloud intelligent content security interface, and a Baidu open platform-content review interface), and whether the external link is a harmful link or an advertising link is determined through an audit API interface provided by a service provider providing the webpage content review interface. A deep learning image recognition interface (Tencent cloud content security interface) of a network audit service provider is used to establish a thread, asynchronously submit an audit, and asynchronously complete the audit result, so that the link content review is carried out in parallel without blocking and efficiently.

[0044] S4, after the audit application is submitted in step S3, a thread different from the thread used to submit the audit application is used to obtain the audit result information returned by the webpage content review interface service provider from the audit API interface: whether the webpage contains dangerous content and the classification of the content, including yellow, advertising, and other dangerous content; according to the returned audit result information, the illegal website containing dangerous content is added to the illegal link database through a database instruction insert, and a pop-up window is used to warn the administrator.

[0045] The illegal link database is a MySQL database established based on the dangerous links fed back by China Telecom and gradually adding suspicious links and dangerous links obtained through website detection.

[0046] If the target website administrator opens the illegal link database permission, we can directly obtain all external links in the illegal link database through global search without the need to obtain them through the crawler technology; for improving efficiency. However, the application of the crawler technology makes the scheme more universal, which can be used when the administrator permission is lacking.

[0047] An illegal domain name detection system based on a network crawler, comprising:

[0048] A database module for storing illegal links and performing pre-examination based on the database before each link detection, which can improve the detection efficiency.

[0049] An audit submission module for directly submitting all external links obtained by crawling to an audit interface through multithreading.

[0050] An audit query module for asynchronously obtaining audit information from the audit interface and transmitting the result to the database module.

[0051] An illegal domain name detection device based on a network crawler, comprising:

[0052] A memory for storing a computer program;

[0053] A processor is configured to implement the network crawler-based illegal domain name detection method in steps S1-S4 when executing the computer program.

[0054] A computer readable storage medium stores a computer program, and the computer program is configured to implement the network crawler-based illegal domain name detection method in steps S1-S4 when executed by a processor.

[0055] According to the above embodiments, the crawler is used to obtain all internal and external links of a website, and the nature of the external links is detected; the illegal link database is automatically generated through the website link detection without manual operation; the multi-thread technology is used to reduce the time difference between the network request and the network data response, and improve the operation efficiency; the database can be operated while the operation is performed, and the operation is flexible and efficient.

Claims

1. A method for detecting illegal domain names based on web crawlers, specifically including the following steps: S1. Use web crawling technology to crawl all external links and their content under the target website domain; S2. Search all the obtained external links in the illegal link database and check whether the external link exists in the illegal link database; if the external link is found to be in the illegal link database, proceed directly to the warning and feedback stage. S3. If the external link is not detected in the illegal link database, the external link that is not detected in the illegal link database will be submitted to the web page content review interface for review. The URLs of external links detected in step S2 that are not in the illegal link database are submitted in parallel to the web page content review interface through multi-threading. The review API provided by the service provider that provides the web page content review interface is used to determine whether the external link is a harmful link or an advertising link. S4. After submitting the review application in step S3, use a different thread from the one used to submit the review application to obtain the review result information returned by the webpage content review interface service provider from the review API interface: whether the webpage contains dangerous content and the classification of the content, including dangerous content such as pornography and advertisements; based on the returned review result information, add illegal websites containing dangerous content to the illegal link database using the database command insert, and alert the administrator through a pop-up window.

2. The method for detecting illegal domain names based on web crawlers according to claim 1, characterized in that, The specific method of step S1 is as follows: using Python web crawler technology, the request library is used to obtain the source code of the webpage, and the re regular expression is used to obtain all the links under the target webpage. Using the thread library, the internal link sub-interfaces of these links under the same domain are accessed level by level in multiple threads, and the descendant links contained in the sub-interfaces are obtained. Then, all the links of the pages represented by these internal links are crawled. This process is repeated until all the external links under the domain are obtained.

3. The method for detecting illegal domain names based on web crawlers according to claim 1, characterized in that, The specific method of step S2 is as follows: For all external links crawled in step S1, search in the illegal link database. Use a select statement to query whether the link exists in the illegal link database. If the external link is detected to be in the illegal link database, send an alert to the target website administrator and notify the target website administrator to remove the external link from the website. At the same time, the target website administrator can view the contents of the illegal link database in real time and add or delete content.

4. The method for detecting illegal domain names based on web crawlers according to claim 1, characterized in that, The illegal link database is a MySQL database built upon reports of dangerous links from authoritative organizations, including China Telecom, and gradually supplemented with suspicious and dangerous links detected by various websites.

5. A method for detecting illegal domain names based on web crawlers according to claim 1 or 2, characterized in that, Step S1 uses web crawler technology to crawl all external links under the target website domain. If the target website administrator has granted access to the illegal link database, all external links can be obtained by directly performing a global search in the illegal link database.

6. An illegal domain name detection system based on the method of any one of claims 1 to 4, characterized in that, include: The database module is used to store illegal links and perform pre-screening based on the database before each link detection, which can improve detection efficiency. The review submission module is used to directly submit all crawled external links to the review interface using multiple threads. The audit query module is used to asynchronously retrieve audit information from the audit interface and pass the results to the database module.

7. An illegal domain name detection device based on the method of any one of claims 1 to 4, characterized in that, include: Memory, used to store computer programs; A processor is used to implement the illegal domain name detection method based on web crawlers described in steps S1 to S4 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program. When the computer program is executed by the processor, it can implement the illegal domain name detection method based on web crawler as described in any one of claims 1 to 4.

Images