Web crawler interception method, device, electronic device and readable storage medium

Abstract

The present application discloses a method, device, electronic device and readable storage medium for intercepting web crawlers. Each time an edge node receives an access request from a first terminal device, it generates an access log based on the access request and sends it to a cache. The computing cluster reads the first access log in the message queue in real time, reads multiple access logs containing the domain name from the message queue based on the domain name contained in the access log, and then determines whether the access request corresponding to the first access log is a malicious request based on the multiple access logs. In this process, since the access logs in the message queue come from edge nodes of the entire network, the computing cluster analyzes the data of the entire network and can quickly and accurately identify malicious web crawlers. If there is a new web crawler, it is only necessary to update the analysis model on the computing cluster, without upgrading the protection software of each edge node, which is fast and simple.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and in particular to a network crawler interception method, device, electronic device and readable storage medium. Background Art

[0002] Web crawlers, also called web spiders, usually look for the Uniform Resource Locator (URL) of a web page based on the address of the web page, and then crawl the website content based on the URL.

[0003] In order to prevent web crawlers from crawling website content, the industry regulates the behavior of web crawlers through robot protocols, which are also called crawler protocols, robots protocols, etc. However, some malicious web crawlers do not comply with robot protocols, and traditional robot protocols cannot intercept such malicious web crawlers. For this reason, in the Content Delivery Network (CDN), protection software is deployed at the edge nodes to detect and protect malicious web crawlers. After the edge node receives the Hyper Text Transfer Protocol (HTTP) request from the terminal device, it analyzes the unique features such as the Internet Protocol (IP) address and User-Agent (UA) in the HTTP request to identify malicious web crawlers. At the same time, it combines black and white list management, repeated violation penalties, etc. to intercept malicious web crawlers at the edge node.

[0004] However, in the CDN network, there are many edge nodes, each of which is independent of each other, and the workload of upgrading and maintaining the protection software is large and the operation steps are cumbersome. Summary of the invention

[0005] The present application provides a network crawler interception method, device, electronic device and readable storage medium. The computing cluster collects data from the entire network in real time and quickly analyzes massive data to detect malicious network crawlers. The speed is fast and the upgrade and maintenance workload is small, thereby achieving the purpose of quickly improving network security.

[0006] In a first aspect, an embodiment of the present application provides a network interception method, which is applied to a computing cluster, and the method includes:

[0007] Reading a first access log in a message queue, wherein the message queue is obtained by sorting access logs from at least two edge nodes in a receiving order by a cache;

[0008] According to the domain name included in the first access log, read a plurality of access logs including the domain name from the message queue in sequence;

[0009] According to the multiple access logs, it is determined whether the access request corresponding to the first access log is a malicious request.

[0010] In a second aspect, an embodiment of the present application provides a network interception method, which is applied to an edge node, and the method includes:

[0011] receiving an access request from a first terminal device;

[0012] Generate an access log according to the access request;

[0013] The access log is sent to a cache, so that the cache sorts the access log from at least one edge node according to a receiving order to obtain a message queue.

[0014] In a third aspect, an embodiment of the present application provides a network crawler interception device, including:

[0015] A reading module is used to read a first access log in a message queue, wherein the message queue is obtained by sorting the access logs from at least two edge nodes in a receiving order by a buffer; and read a plurality of access logs containing the domain name from the message queue in sequence according to the domain name contained in the first access log;

[0016] The processing module determines, based on the multiple access logs, whether the access request corresponding to the first access log is a malicious request.

[0017] In a fourth aspect, an embodiment of the present application provides a network crawler interception device, including:

[0018] A receiving module, used for receiving an access request from a first terminal device;

[0019] A processing module, used to generate an access log according to the access request;

[0020] A sending module is used to send the access log to a cache so that the cache sorts the access logs from at least one edge node in the order of reception to obtain a message queue, wherein multiple access logs in the message queue that are located after the first access log and contain the same domain name as the first access log are used to determine whether the access request corresponding to the first access log is a malicious request.

[0021] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the method described in the first aspect or various possible implementation methods of the first aspect.

[0022] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the method described in the second aspect or various possible implementation methods of the second aspect.

[0023] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, they are used to implement the method described in the first aspect or various possible implementation methods of the first aspect.

[0024] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, they are used to implement the method described in the second aspect or various possible implementation methods of the second aspect.

[0025] In a ninth aspect, an embodiment of the present application provides a computer program product comprising a computing program, and when the computer program is executed by a processor, the method described in the first aspect or various possible implementation methods of the first aspect is implemented.

[0026] In the tenth aspect, an embodiment of the present application provides a computer program product comprising a computing program, and when the computer program is executed by a processor, the method described in the second aspect or various possible implementation methods of the second aspect is implemented.

[0027] The web crawler interception method, device, electronic device and readable storage medium provided by the embodiment of the present application, each time the edge node receives an access request from the first terminal device, an access log is generated according to the access request and sent to the buffer, and the buffer sorts the access logs from at least two edge nodes in the order of reception to obtain a message queue. The computing cluster reads the first access log in the message queue in real time, reads multiple access logs containing the domain name from the message queue according to the domain name contained in the access log, and then determines whether the access request corresponding to the first access log is a malicious request according to the multiple access logs. In this process, since the access logs in the message queue come from the edge nodes of the entire network, the computing cluster analyzes the data of the entire network and can quickly and accurately identify malicious web crawlers. If there is a new web crawler, it only needs to update the analysis model on the computing cluster, and there is no need to upgrade the protection software of each edge node, which is fast and simple. Moreover, since the data of the entire network is analyzed in real time, malicious web crawlers will not be missed, thereby achieving the purpose of improving network security. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1A This is a schematic diagram of an implementation environment of the network crawler interception method provided in an embodiment of the present application;

[0030] Figure 1B It is another implementation environment diagram of the network crawler interception method provided in the embodiment of the present application;

[0031] Figure 2 is a flow chart of a network crawler interception method provided in an embodiment of the present application;

[0032] Figure 3 is another flow chart of the web crawler interception method provided in an embodiment of the present application;

[0033] Figure 4 is another flow chart of the web crawler interception method provided in the embodiment of the present application;

[0034] Figure 5 A schematic diagram of a network crawler interception device provided in an embodiment of the present application;

[0035] Figure 6 Another schematic diagram of the network crawler interception device provided in an embodiment of the present application;

[0036] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0038] At present, the robot protocol is a protocol between websites and web crawlers. The robot protocol is usually a simple text in txt format. In the protection based on the robot protocol, the web crawler is told the allowed permissions through a simple txt text format. When a web crawler visits a site, it first checks whether there is a robots.txt file in the root directory of the site. If it exists, the web crawler will determine the scope of access according to the content of the file; if the file does not exist, the web crawler will be able to access all pages on the website that are not protected by passwords.

[0039] However, some malicious web crawlers do not comply with the robot protocol, and the above robot protocol cannot effectively prevent malicious web crawlers from crawling website content. While malicious web crawlers crawl the core content of the website, they occupy a large amount of bandwidth, computing resources, etc., seriously damaging the various interests of the crawled website. For example, malicious web crawlers use Bot tools to crawl a large number of preferential resources on the website, and then resell them for secondary profit, which seriously affects the normal operation of merchants. For another example, malicious web crawlers crawl the core content of the website and copy it to other websites, which greatly affects the ranking of the core website on the search engine, resulting in a decrease in the website's traffic and advertising revenue.

[0040] In the CDN acceleration network, access requests initiated by terminal devices will be dispersed to edge nodes in various regions according to the region where the visitor is located. Malicious crawlers simulate the access requests of normal visitors to crawl website content, and these malicious requests will also be transmitted to edge nodes. In order to intercept malicious web crawlers, CDN manufacturers deploy protection software on edge nodes. While accelerating the website, they use edge nodes to detect and protect malicious web crawlers to achieve security protection for the website. In the process of malicious web crawler protection, edge nodes identify malicious web crawlers by analyzing unique features such as IP, UA, and client identification in HTTP requests. At the same time, edge nodes combine refined strategies such as black and white list management, repeated violation penalties, and access behavior restrictions after exceeding the number of visits to directly identify and intercept malicious crawler information at the edge nodes.

[0041] With the rapid development of the Internet, malicious web crawlers have shown the characteristics of frequent changes in attack features and changeable crawling behaviors. The characteristics analyzed by the traditional interception method based on edge nodes are too single and cannot accurately identify and intercept malicious web crawlers. Moreover, in the interception method based on edge nodes, the protection software is deployed on edge nodes, and the number of edge nodes is huge, which makes the upgrade and maintenance of the protection software more troublesome. The upgrade and maintenance cycle is long. For example, assuming that there are 1,000 edge nodes in the CDN network, when a new web crawler appears, the protection software on each edge node needs to be upgraded and maintained. The upgrade cycle is long, and it is difficult to successfully upgrade and intercept online in a short time.

[0042] In addition, in the interception method based on edge nodes, each edge node can only detect access requests passing through itself. In scenarios where malicious web crawlers are relatively scattered, it is easy to fail to identify and not intercept certain malicious crawlers. For example, within a time window, if an IP address is accessed more than or equal to 10 times, the access request to the IP address is considered a malicious request. However, since the calculation of each edge node is independent, it is very likely that the 10 access requests initiated by malicious web crawlers are sent to 10 different edge nodes. At this time, each edge node believes that there is only one access, and the edge node cannot feel that the access request has reached the threshold, resulting in the inability to identify and intercept malicious web crawlers. In other words, the existing protection method based on edge nodes is prone to miss some malicious web crawlers.

[0043] In addition, for malicious web crawlers that appear less frequently, edge nodes need to analyze data for a longer period of time to detect malicious crawler behavior. However, the computing resources of each edge node are limited, and it is difficult to analyze historical data over a longer period of time. Therefore, the interception method based on edge nodes cannot identify and intercept low-frequency web crawlers.

[0044] To this end, the embodiment of the present application provides a network crawler interception method, which uses common data collection, streaming computing, big data storage and other technologies to detect malicious crawlers. It is not only applicable to CDN, but also to various distributed deployment systems, with good versatility and strong scalability.

[0045] Figure 1A This is a schematic diagram of an implementation environment of the web crawler interception method provided in the embodiment of the present application. Figure 1A The environment of this embodiment includes: a computing cluster 11, an edge node 12, a cache 13, a memory 14, a first terminal device 15 and a second terminal device 16.

[0046] The first terminal device 15 can be hardware or software. When the first terminal device 15 is hardware, the first terminal device 15 is, for example, a mobile phone, tablet computer, personal computer, e-book reader, laptop portable computer, desktop computer, etc. installed with an Android operating system, a Microsoft operating system, a Symbian operating system, a Linux operating system, or an Apple iOS operating system. When the first terminal device 15 is software, it can be installed in the hardware devices listed above. At this time, the first terminal device 15 is, for example, a plurality of software modules or a single software module, etc., which is not limited by the embodiment of the present application. Office applications, instant messaging applications, shopping applications, etc. are installed on the first terminal device 15. In the CDN network, the first terminal device 15 accesses the CDN network through the edge node 12. After the first terminal device 15 initiates an access request, it accesses the edge node 12 according to the principle of proximity, etc. The edge node 12 generates an access log according to the access request initiated by the first terminal device 15 and sends it to the cache 13. The cache 13 sorts the access logs of each edge node to obtain a message queue.

[0047] The computing cluster 11 is also called a stream computing cluster, a streaming computing cluster, etc. It is a cluster composed of multiple computing servers, which is used to perform real-time analysis on access logs in message queues. By performing real-time analysis on access logs of the entire network, malicious crawlers can be quickly detected in massive data and certain malicious web crawlers can be prevented from being missed.

[0048] The cache 13 can be integrated into the computing cluster 11 or set separately to sort the access logs of each edge node to obtain a message queue. Through the first-in-first-out principle, the access logs in the message queue are delivered to the computing cluster 11 in the form of a stream, so that the computing cluster 11 can perform real-time analysis of the entire network data. The access logs stored in the cache 13 are limited. When a message queue is full, once the access log is delivered to the computing cluster 11, the new access log will be added to the message queue and placed at the end.

[0049] The memory 14 can be integrated in the computing cluster 11 or set separately to store access logs within a period of time, for example, access logs within 24 hours, access logs within 48 hours, etc. After receiving each access log, the cache 13 sends the access log to the memory 14. The computing cluster 11 regularly analyzes the access logs in the memory 14 to prevent missing malicious crawlers with low frequency.

[0050] After the computing cluster 11 analyzes the access log in the cache 13 in real time to determine the malicious web crawler, or after the computing cluster 11 regularly performs offline analysis on the access log in the memory 14 to obtain the malicious web crawler, it sends an interception instruction to each edge node 12, and the interception instruction carries at least one of the client identifier, IP address, domain name or URL to be intercepted. After receiving the interception instruction, the edge node 12 intercepts the access request from the first terminal device 15 according to the interception instruction.

[0051] The second terminal device 16 is, for example, a desktop computer or a notebook computer of an operation and maintenance personnel. The second terminal device 16 sends a configuration request to the computing cluster 11 to request a configuration interface. Through the configuration interface, it is possible to flexibly configure which features to be analyzed in the access log, and set some thresholds, such as the preset number of IP addresses, the preset number of URLs, etc. The computing cluster 11 analyzes the access log according to these settings. For example, if the features set through the configuration interface include IP addresses and URLs, the computing cluster 11 extracts the IP addresses and URLs from the access log, and determines the number of IP addresses, the number of URLs, etc.

[0052] Figure 1B This is another implementation environment diagram of the web crawler interception method provided in the embodiment of the present application. Figure 1B , this embodiment is different from the above Figure 1A The difference is that the implementation environment also includes: offline component 17, configuration management center 18, configuration file cluster 19 and zookeeper cluster 20. The offline component 17 is, for example, a server, etc., which is used to perform offline analysis on the access logs in the memory 14 to determine low-frequency malicious web crawlers.

[0053] The configuration management center 18 is used to generate a configuration file based on the malicious web crawler identified by the computing cluster 11 or the offline component 17, and upload the configuration file to the configuration file cluster 19; at the same time, the node information of the zookeeper cluster 20 is updated, and the node information is used to indicate that the configuration file on the configuration file cluster 19 has been updated. The edge node 12 monitors the zookeeper cluster 20, and once the node information is monitored to change, it obtains the configuration file from the configuration file cluster 19, and the configuration file is used to indicate at least one of the client identifier, IP address, domain name or URL to be intercepted. Afterwards, the edge node 12 intercepts the access request from the first terminal device 15 according to the configuration file.

[0054] Below, based on Figure 1A and Figure 1B The architecture shown in the figure is used to describe in detail the web crawler interception method described in the embodiment of the present application. Figure 2 .

[0055] Figure 2 Flowchart of the web crawler interception method provided by the embodiment of the present application. This embodiment is described from the perspective of the interaction between the computing cluster, the cache, and the edge node. This embodiment includes:

[0056] 201. An edge node receives an access request from a first terminal device.

[0057] Exemplarily, the user opens an APP, a browser, etc. on the first terminal device, thereby sending an access request to the edge node.

[0058] 202. The edge node generates an access log according to the access request.

[0059] Exemplarily, the edge node generates an access log according to the time when the access request is initiated, the domain name, IP address, UA, URL, etc. carried in the access request.

[0060] 203. The edge node sends the access log to the cache.

[0061] Correspondingly, the cache receives the access log. Due to the large number of first terminal devices, each first terminal device accesses the edge node according to the principle of proximity. Within a period of time, the cache can receive access logs sent by multiple edge nodes. The cache sorts the access logs according to the reception time of each access log to obtain a message queue. The first access log received is at the front of the message queue. Since the message queue stores the access logs from each edge node, the access logs in the message queue can be understood as the whole network data.

[0062] 204. The computing cluster reads the first access log from the message queue.

[0063] Exemplarily, during the real-time analysis process, the computing cluster reads the access log in the message queue in real time. For example, the computing cluster reads the access log once every 1 millisecond. The access log is referred to as the first access log below. Alternatively, the computing cluster shunts the access log in the message queue according to the domain name to obtain multiple access log streams distinguished by the domain name. For any access log stream, the first access log is any access log in the access log stream.

[0064] 205. The computing cluster sequentially reads multiple access logs containing the domain name from the message queue according to the domain name contained in the first access log.

[0065] 206. Determine, based on the multiple access logs, whether the access request corresponding to the first access log is a malicious request.

[0066] Exemplarily, each access log contains a respective domain name. In order to determine whether the access request corresponding to the first access log is a malicious request, after the computing cluster reads the first access log, it continues to read access logs from the message queue, and determines whether the access request corresponding to the first access log is a malicious request based on multiple access logs containing the same domain name as the first access log.

[0067] Multiple analysis models are pre-deployed on the computing cluster, and the computing cluster can use one or more analysis models to analyze multiple access logs to identify malicious web crawlers. For example, the computing cluster analyzes the client identifier of each access log in multiple access logs to determine whether the access request corresponding to the first access log is a malicious request. For another example, the computing cluster analyzes the UA, IP address, etc. contained in each access log in multiple access logs to determine whether the access request corresponding to the first access log is a malicious request.

[0068] The web crawler interception method provided by the embodiment of the present application is that each time the edge node receives an access request from the first terminal device, an access log is generated according to the access request and sent to the buffer, and the buffer sorts the access logs from at least two edge nodes in the order of reception to obtain a message queue. The computing cluster reads the first access log in the message queue in real time, reads multiple access logs containing the domain name from the message queue according to the domain name contained in the access log, and then determines whether the access request corresponding to the first access log is a malicious request according to the multiple access logs. In this process, since the access logs in the message queue come from the edge nodes of the entire network, the computing cluster performs real-time streaming analysis on the entire network data, and can quickly and accurately identify malicious web crawlers in massive data. If there is a new web crawler, it only needs to update the analysis model on the computing cluster, and there is no need to upgrade the protection software of each edge node, which is fast and simple. Moreover, since the entire network data is analyzed in real time, malicious web crawlers will not be missed, and the purpose of improving network security is achieved.

[0069] In the embodiment of the present application, the computing cluster identifies malicious web crawlers including: real-time analysis of access logs to identify malicious web crawlers, and offline analysis of a large number of historical access logs to identify malicious web crawlers.

[0070] During the real-time analysis process, the computing cluster reads the first access log from the cache, and identifies and intercepts malicious web crawlers based on the domain name, client identifier, etc. contained in the first access log.

[0071] Figure 3 is another flow chart of the web crawler interception method provided by the embodiment of the present application. In this embodiment, the computing cluster analyzes the access log in real time to identify and intercept malicious web crawlers. This embodiment includes:

[0072] 301. The computing cluster distributes access logs according to domain names.

[0073] Exemplarily, the computing cluster continuously reads the access logs in the message queue, and each time the access log is read, the access log is divided according to the domain name contained in the access log. For example, if one access log contains the domain name "google.com" and another access log contains the domain name "developer.android.com", the computing cluster divides the two access logs into different access log streams.

[0074] 302 , the computing cluster reads the first access log and determines whether the first access log contains the first identifier. If the first access log contains the first identifier, execute step 303 ; if the first access log does not contain the first identifier, execute step 306 .

[0075] Exemplarily, each time the edge node receives an access request, it determines whether the access request contains the first identifier or the second identifier. If the access request carries neither the first identifier nor the second identifier, the edge node considers that this is the first time that the first terminal device has initiated an access request. Therefore, the edge node assigns a first identifier to the first terminal device and sends an indication message to the first terminal device, where the indication message carries the first identifier and the target operation. The indication message is used to instruct the first terminal device to perform the target operation on the first identifier and obtain the second identifier before sending an access request carrying the second identifier. Among them, the first identifier and the second identifier are both client identifiers. The first identifier assigned to the first terminal device by the edge node is also called a pre-assigned client identifier (PreClientID), and the second identifier generated by the first terminal device performing the target operation on the first identifier is also called a client identifier (ClientID).

[0076] When the edge node assigns the first identifier to the first terminal device, the edge node generates a first access log for the access request that carries neither the first identifier nor the second identifier. Afterwards, the edge node sends the first access log to the cache. Obviously, if the first access log contains the first identifier, it means that the access request corresponding to the first access log is an initial request, and the access request does not carry the first identifier or the second identifier.

[0077] In addition, when the access request carries the second identifier, it indicates that before the first terminal device initiates the access request, it has received the indication information carrying the first identifier and the target operation, and has performed the target operation on the first identifier to generate the second identifier. At this time, the edge node generates an access request containing the second identifier for the access request containing the second identifier.

[0078] In the embodiment of the present application, the malicious web crawler is usually a piece of code, and the hacker uses the code to crawl the website content. The malicious web crawler will not perform the target operation. For example, the target operation is to multiply the first identifier by 2. When the first identifier is "111", if the first terminal device is a normal user's terminal device, after the first terminal device receives the instruction information carrying the first identifier and the target operation, the target operation can be performed on the first identifier to obtain the second identifier, and the second identifier is "222". After that, the access request initiated by the first terminal device will carry the second identifier, for example, the second identifier is carried in a field of the HTTP request.

[0079] If the first terminal device is a hacker's terminal device and it does not perform the target operation, then after the first terminal device receives the target operation and the first identifier, it cannot perform the target operation on the first identifier to obtain the second identifier. The first terminal device receives the instruction information carrying the first identifier and the target operation and considers that the crawling has failed. At this time, the hacker obtains the second identifier by modifying the malicious crawler code or other means. If the hacker obtains the second identifier, he or she will initiate an access request carrying the second identifier to deceive trust; if the hacker cannot obtain the second identifier, he or she will not be able to crawl the website content.

[0080] According to the above, it can be known that: from the perspective of the computing cluster, each time the computing cluster reads the first access log from the message queue, it diverts the first access log and determines whether the client identifier contained in the first access log is the first identifier or the second identifier. If the first access log contains the first identifier, it means that the access request is an initial access request, and it is impossible to determine whether the access request corresponding to the first access log is a malicious request. At this time, the computing cluster executes step 303; if the first access log does not contain the first identifier but contains the second identifier, the computing cluster believes that: the first terminal device may be a terminal device of a normal user who can perform the target operation, and the access request is a normal access request; or, the first terminal device may also be a terminal device of a hacker who can obtain the second identifier by illegal means, and the access request is a malicious access request. In order to prevent misjudgment, the computing cluster executes step 306.

[0081] 303. The computing cluster determines whether there is a second access log containing the second identifier in the multiple access logs. If there is a second access log containing the second identifier in the multiple access logs, execute step 304; if there is no second access log containing the second identifier in the multiple access logs, execute step 305.

[0082] Exemplarily, after the computing cluster reads the first access log, it parses the first access log to obtain the domain name. After that, it continues to read the access log from the message queue, and divides the access log containing the same domain name as the first access log into the same access log stream. For example, at 9:00, the computing cluster determines that the first access log contains the first identifier, then the computing cluster sets a 10-minute time window, which starts from 9:00 and ends at 9:10, and allocates a memory space for the time window. Within the time window, the computing cluster continues to read the message queue. Once an access log containing the same domain name as the first access log is read, the access log is transferred to the memory space corresponding to the time window to determine whether the access log contains the second identifier. If there is at least one access log containing the second identifier within the entire time window, step 304 is executed; if no access log contains the second identifier after the entire time window ends, step 305 is executed.

[0083] 304. The computing cluster determines that the access request corresponding to the first access log is not a malicious request.

[0084] For example, within the time window from 9:00 to 9:10, at 9:05, the computing cluster reads an access log that contains the same domain name as the first access log and contains the second identifier. The computing cluster believes that the access request corresponding to the first access log is not a malicious request, but an access request initiated by a normal user. In other words, although the first access log does not contain the second identifier, after the first terminal device initiates the initial access request, within 10 minutes, after the first terminal device receives the instruction information carrying the first identifier and the target operation, and performs the target operation on the first identifier to generate the second identifier, the access request subsequently initiated carries the second identifier. In this process, the length of the time window can be set according to network delays, etc., and the embodiments of the present application are not limited.

[0085] 305. The computing cluster determines that the access request corresponding to the first access log is a malicious request.

[0086] With this solution, when the first access log contains the first identifier, the computing cluster determines whether the access request corresponding to the first access log is a malicious request by setting a time window, with high accuracy.

[0087] In addition, when the first access log contains the first identifier, the edge node only carries the first identifier and the target operation in the indication information and sends it to the first terminal device, but does not send the content on the website to the first terminal device, which is equivalent to intercepting a malicious access request.

[0088] 306. The computing cluster extracts a second identifier from the first access log.

[0089] Exemplarily, when the edge node generates an access log, if the access request carries neither the first identifier nor the second identifier, an access log containing the first identifier is generated; if the access request carries the second identifier, an access log containing the second identifier is generated. In this step, the computing cluster parses the first access log to obtain the second identifier.

[0090] 307. The computing cluster constructs a time window according to the second identifier.

[0091] The length of the time window is, for example, 3 minutes, 5 minutes, 10 minutes, etc., which is not limited in the embodiment of the present application.

[0092] 308. The computing cluster incrementally calculates the number of different IP addresses.

[0093] Exemplarily, when an access request carries a second identifier, it is possible that the first terminal device is a normal user's terminal device, which can obtain the first identifier and the target operation, and perform the target operation on the first identifier to obtain the second identifier. Another possibility is that a hacker illegally obtains the second identifier and carries the second identifier in the access request to deceive trust. At this time, the computing cluster continues to analyze the access log based on multi-dimensional features to determine whether the access request corresponding to the first access log is a malicious request. Among them, the multiple dimensions include client identifier and IP address dimensions, IP address dimensions and URL dimensions, etc.

[0094] In this step, when an access request carries the second identifier, in order to prevent misjudgment, the computing cluster continues to read access logs from the message queue, and divides the access logs containing the same domain name as the first access log and the second identifier into the same access log stream. The access log containing the same domain name as the first access log and the second identifier is referred to as the second access log.

[0095] At the same time, the computing cluster determines the IP address contained in each of the plurality of second access logs to obtain the number of different IP addresses. Exemplarily, each time a second access log is read, the IP address is extracted from the second access log. Since some second access logs contain the same IP address, the number of IP addresses is increased by 1 only when different IP addresses are extracted.

[0096] 309. The computing cluster determines whether the number of IP addresses exceeds the preset number. If the number of IP addresses exceeds the preset number, step 305 is executed. If the number of IP addresses does not exceed the preset number, step 304 is executed.

[0097] Exemplarily, within the time window constructed in step 307, the computing cluster continues to read the message queue. Once a second access log containing the same IP address and the same second identifier as the first access log is read, the IP address is extracted from the second access log to determine whether the number of IP addresses exceeds the preset number. If the number of IP addresses within the entire time window does not exceed the preset number, step 304 is executed; if within the time window, after the second access log is read once, the number of IP addresses exceeds the preset number, step 305 is executed.

[0098] For example, the length of the time window is 5 minutes, the start time is 8:00, the end time is 8:05, and the preset number is 10. Assuming that at 8:04, the number of IP addresses is 11, it means that multiple malicious web crawlers carry the same second identifier and initiate access through different IP addresses. At this time, the computing cluster executes step 305; if by 8:05, the number of IP addresses is less than 11, the computing cluster executes step 304.

[0099] With this solution, when the first access log contains the second identifier, the computing cluster can prevent misjudgment by counting the client identifiers, IP addresses, etc. of multiple access logs to determine whether the access request corresponding to the first access log is a malicious request.

[0100] Optionally, in the above embodiment, whether analyzing access logs in real time to identify malicious requests or analyzing access logs offline to identify malicious requests, the computing cluster can identify malicious web crawlers by analyzing multi-dimensional features in the access logs.

[0101] During the real-time multi-dimensional analysis process, the computing cluster reads multiple access logs from the message queue and extracts at least two features from each of the multiple access logs. Afterwards, the computing cluster determines whether the access request corresponding to the first access log is a malicious request based on the at least two features extracted from each of the multiple access logs, wherein the at least two features include any two fields contained in the access request, and the fields contained in the access request include a client identifier, an IP address, a user agent UA, or a uniform resource locator URL. For example, each time the first access log is read, the computing cluster determines the domain name, client identifier, etc. contained in the first access log, and then reads multiple access logs containing the same domain name and client identifier from the message queue, and performs incremental calculations on the IP addresses, etc. in these access logs, thereby determining whether the location request corresponding to the first access log is a malicious request.

[0102] This solution, through real-time, multi-dimensional correlation analysis, can cope with scenarios where malicious web crawler attack characteristics change frequently and crawling behaviors are complex and changeable.

[0103] Below, taking at least two features, IP address and URL, as an example, the above real-time multi-dimensional analysis is described in detail. Figure 4 . Figure 4 This is another flow chart of the web crawler interception method provided in the embodiment of the present application. This embodiment includes:

[0104] 401. The computing cluster distributes access logs according to domain names.

[0105] For details, see the row number Figure 3 The description of step 301 of the embodiment will not be repeated here.

[0106] 402. The computing cluster allocates a time window to the daily access log stream of each domain name.

[0107] Exemplarily, the computing cluster configures a time window for each domain name according to the configuration uploaded by the second terminal device, etc. For example, for the domain name "google.com", the time window is set to 5 minutes, and for the domain name "developer.android.com", the time window is set to 8 minutes.

[0108] 403. For each domain name's access log stream, the computing cluster determines the IP address in each access log.

[0109] Malicious web crawlers often dynamically change features such as IP addresses and UA to bypass the interception of edge nodes. Therefore, the computing cluster can analyze and specify specific features in real time and detect abnormal behavior. For example, the computing cluster can be specified in advance through the second terminal device to analyze the IP address and URL.

[0110] Each time the computing cluster reads a first access log from the message queue, it determines the access log stream to which the first access log belongs, and further determines the IP address contained in each access log in the access log stream.

[0111] 404. For each IP address, the computing cluster determines the number of access logs containing the IP address.

[0112] Exemplarily, the access logs belonging to the same access log stream all contain the same domain name. However, the IP addresses contained in each of these access logs may be the same or different.

[0113] For example, the access log stream including the first access log includes 40 access logs, of which 20 access logs include IP address 1, 10 access logs include IP address 2, and the other 10 access logs include IP address 3.

[0114] 405. For each IP address, the computing cluster determines whether the number of access logs corresponding to the IP address exceeds the preset threshold. If the number of access logs corresponding to the IP address exceeds the preset threshold, execute step 406; if the number of access logs corresponding to the IP address does not exceed the preset threshold, execute step 409.

[0115] Continuing with the example in step 404, assuming that the IP address contained in the first access log is IP address 1, the access log containing IP address 1 and the same domain name as the first access log is referred to as the third access log. The number of third access logs is 20. Assuming that the preset threshold is 15, the computing cluster executes step 406; if the preset threshold is 30, the computing cluster executes step 409.

[0116] 406. The computing cluster extracts URLs from each third access log.

[0117] 407 . The computing cluster determines whether the number of different URLs exceeds the preset number of URLs. If so, the computing cluster executes step 408 . If not, the computing cluster executes step 409 .

[0118] The number of third access logs is 20, and the URLs contained in the 20 third access logs are the same or different. The computing cluster counts the number of URLs. For example, among the 20 third access logs, 10 third access logs contain URL a, 5 third access logs contain URL b, and the remaining 5 third access logs contain URL c, then the number of different URLs is 3. Assuming the preset number of URLs is 5, the computing cluster executes step 409; assuming the preset number of URLs is 2, the computing cluster executes step 408.

[0119] 408. The computing cluster determines that the access request corresponding to the first access log is a malicious request.

[0120] 409. The computing cluster determines that the access request corresponding to the first access log is not a malicious request.

[0121] With this solution, the computing cluster identifies and intercepts malicious web crawlers by analyzing the real-time specified features, which can prevent malicious crawlers from bypassing the interception of edge nodes by dynamically changing IP addresses, UA, etc. The accuracy of crawler identification and interception is high.

[0122] During the offline analysis process, the computing cluster analyzes all or part of the access logs in the memory. During the analysis process, the computing cluster divides the access logs in the memory according to the domain names contained in each access log in the memory to obtain multiple access log streams, and each access log belonging to the same access log stream contains the same domain name. The memory stores the access logs from the cache. Afterwards, at least two features contained in each access log in the target access log stream are determined to determine whether there are malicious requests in the access requests corresponding to the access logs contained in the target access log stream.

[0123] Exemplarily, the memory stores access logs for one day, three days, one week, or one month. The computing cluster regularly performs offline analysis on these access logs to identify web crawlers with low frequency of occurrence. For example, the computing cluster performs offline analysis on the access logs in the memory every day; or, the computing cluster performs offline analysis on the access logs in the memory once a week. During the analysis process, the computing cluster can analyze one or more features such as client identification, IP address, UA, URL, domain name, etc. For details, please refer to Figure 4 The embodiments are not described in detail here.

[0124] It should be noted that, although in this solution, the computing cluster performs offline analysis on the access logs in the memory. However, the embodiment of the present application is not limited to this, and other feasible implementations, such as Figure 1B As shown, an offline component may also be separately provided, and the access log in the memory may be analyzed offline by using the offline component.

[0125] By adopting this solution, the access logs in the storage are analyzed offline regularly, so as to identify malicious web crawlers with low occurrence frequency.

[0126] Optionally, in the above embodiment, before offline analysis or real-time analysis, the features to be analyzed, preset thresholds, preset number of URLs, etc. can be indicated to the computing cluster through the second terminal device. During the indication process, the second terminal device sends a configuration request for requesting a configuration interface to the computing cluster; the corresponding computing cluster receives the configuration request and returns a data stream for displaying the configuration interface to the second terminal device. After receiving the data stream, the second terminal device displays the configuration interface for the operation and maintenance personnel to set at least two features to be analyzed, preset thresholds, preset number of URLs, etc. After that, click the "Submit" button on the configuration interface. The computing cluster determines at least two features to be analyzed according to the settings of the operation and maintenance personnel, and performs real-time analysis or offline analysis based on these features.

[0127] In addition, through the configuration interface, you can also configure which domain name access log streams need offline analysis, which domain name access log streams need real-time online analysis, and which domain name access logs need both online and offline analysis.

[0128] With this solution, the features to be analyzed are specified through the configuration interface, which is highly flexible and has a simple operation process.

[0129] Optionally, in the above embodiment, after the computing cluster performs multi-dimensional real-time analysis or offline analysis on the access log, if a malicious web crawler is identified, an interception instruction is sent to each edge node in at least one edge node, and the interception instruction carries at least one of the client identifier, IP address, domain name or URL to be intercepted.

[0130] For example, the interception instruction carries a client identifier, which is, for example, the second identifier mentioned above. After receiving the interception instruction, the edge node intercepts all access requests carrying the second identifier. For another example, the interception instruction carries an IP address and a client identifier, which is, for example, the second identifier mentioned above. After receiving the interception instruction, the edge node intercepts all access requests carrying the second identifier and the IP address.

[0131] With this solution, after the computing cluster identifies a malicious web crawler, it directly sends an interception instruction to the edge node, so that the edge node intercepts the access request according to the interception instruction, thereby achieving the purpose of quickly intercepting the malicious web crawler.

[0132] Please refer to Figure 1B After the computing cluster identifies a malicious web crawler, it can also issue an interception through the configuration management center. In this implementation environment, the computing cluster sends a task configuration instruction to the management configuration center, which is used to instruct the management configuration center to generate a configuration file and upload the configuration file to the configuration file cluster. At the same time, the computing cluster triggers the zookeeper cluster to update the node information, which is used to indicate that the configuration file on the file cluster has been updated. The configuration file is used to indicate at least one of the client identifier, IP address, domain name or URL to be intercepted.

[0133] Exemplarily, after the computing cluster analyzes the access log in real time and identifies the malicious web crawler, or after the computing cluster performs offline analysis on the access log in the storage and identifies the malicious web crawler, the computing cluster sends a task configuration instruction to the management configuration center, and the task configuration instruction carries the configuration file name, version number, component name, etc. Among them, the component name is, for example, the name of the real-time analysis component and the offline analysis component, and the configuration file name is used to indicate the client identifier, IP address, etc. corresponding to the malicious web crawler.

[0134] The configuration management center maintains a mapping relationship between component names and configuration file paths. After receiving the task configuration instructions, the configuration file generates the configuration file and the mapping relationship. After that, the configuration management file uploads the configuration file to the configuration file cluster. The configuration file cluster stores various versions of the configuration file. At the same time, the configuration management center sends node information to the zookeeper cluster, which includes the name of the updated configuration file, the version number, and the subsequent mapping relationship.

[0135] The edge node monitors the zookeeper cluster. Once the changed node information is monitored, the edge node obtains the latest configuration file from the configuration file cluster and updates it. At the same time, the old version of the configuration file is deleted. After that, the edge node intercepts the access request from the first terminal device according to the configuration file.

[0136] In addition, please refer to Figure 1B ,After the offline analysis component identifies malicious web crawlers with low ,frequency, it can also send task configuration instructions to the ,configuration management center so that the configuration management center generates ,configuration files and uploads them to the configuration file cluster.

[0137] With this solution, the relevant configuration files of malicious web crawlers are stored in the configuration file cluster, which facilitates tracking. If a new edge node is added to the CDN network, the latest configuration file can be directly obtained from the configuration file cluster, avoiding the situation where the new edge node cannot intercept the malicious web crawler.

[0138] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0139] Figure 5 A schematic diagram of a network crawler interception device provided in an embodiment of the present application. The network crawler interception device 500 includes: a reading module 51 and a processing module 52.

[0140] The reading module 51 is used to read a first access log in a message queue, where the message queue is obtained by sorting the access logs from at least two edge nodes in a receiving order by the buffer; and read a plurality of access logs containing the domain name from the message queue in sequence according to the domain name contained in the first access log;

[0141] The processing module 52 determines, based on the multiple access logs, whether the access request corresponding to the first access log is a malicious request.

[0142] In a feasible implementation, the processing module 52 is used to determine whether there is a second access log containing a second identifier in the multiple access logs when the first access log contains a first identifier, the first identifier being assigned by the edge node to the first terminal device when the access request does not carry a client identifier, and the second identifier being generated when the first terminal device performs a target operation on the first identifier; if the second access log does not exist in the multiple access logs, it is determined that the access request corresponding to the first access log is a malicious request.

[0143] In a feasible implementation, the processing module 52 is used to extract at least two features from each access log of the multiple access logs; based on the at least two features extracted from each access log of the multiple access logs, determine whether the access request corresponding to the first access log is a malicious request, the at least two features include any two fields contained in the access request, and the fields contained in the access request include a client identifier, an IP address, a user agent UA or a uniform resource locator URL.

[0144] In a feasible implementation, the processing module 52 is used to determine, from the multiple access logs, a second access log containing the second identifier, to obtain multiple second access logs when the at least two features are a client identifier and an IP address and when the client identifier contained in the first access log is a second identifier; determine the IP address contained in each second access log in the multiple second access logs to obtain the number of different IP addresses; and when the number of different IP addresses exceeds a preset number, determine that the access request corresponding to the first access log is a malicious request.

[0145] In a feasible implementation, the processing module 52 is used to determine, from the multiple access logs, a third access log containing the IP address based on the IP address contained in the first access log when the at least two features are the IP address and the URL; when the number of the third access logs is greater than a preset threshold, extract the URLs in each of the third access logs; and when the number of different URLs exceeds the preset number of URLs, determine that the access request corresponding to the first access log is a malicious request.

[0146] In a feasible implementation, the processing module 52 is also used to divert the access logs in the memory according to the domain names contained in each access log in the memory to obtain multiple access log streams, and each access log belonging to the same access log stream contains the same domain name, and the access logs from the cache are stored in the memory; analyze at least two features contained in each access log in the target access log stream to determine whether there are malicious requests in the access requests corresponding to the access logs contained in the target access log stream.

[0147] Optional, please refer to Figure 5 In a feasible implementation, the network crawler interception device 500 further includes:

[0148] The transceiver module 53 is used to receive a configuration request from a second terminal device before the processing module 52 extracts at least two features from each of the multiple access logs, wherein the configuration request is used to request a configuration interface; send a data stream for displaying the configuration interface to the second terminal device; and receive a configuration request sent by the second terminal device through the configuration interface, wherein the configuration request is used to indicate the at least two features.

[0149] In a feasible implementation, the transceiver module 53 is used to send an interception instruction to each edge node in the at least one edge node after the processing module 52 determines whether the access request corresponding to the first access log is a malicious request based on the multiple access logs. When the access request corresponding to the first access log is a malicious request, the interception instruction carries at least one of the client identifier, IP address, domain name or URL to be intercepted.

[0150] In a feasible implementation, the transceiver module 53 is used to send a task configuration instruction to the management configuration center after the processing module 52 determines whether the access request corresponding to the first access log is a malicious request based on the multiple access logs. The task configuration instruction is used to instruct the management configuration center to generate a configuration file, upload it to the configuration file cluster, and trigger the zookeeper cluster to update the node information. The node information is used to indicate that the configuration file on the file cluster is updated, and the configuration file is used to indicate at least one of the client identifier, IP address, domain name or URL to be intercepted.

[0151] The web crawler interception device provided in the embodiment of the present application can execute the actions of the computing cluster in the above embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0152] Figure 6 Another schematic diagram of a network crawler interception device provided in an embodiment of the present application. The network crawler interception device 600 includes: a receiving module 61 , a processing module 62 and a sending module 63 .

[0153] The receiving module 61 is used to receive an access request from a first terminal device;

[0154] A processing module 62, configured to generate an access log according to the access request;

[0155] A sending module 63 is used to send the access log to a cache so that the cache sorts the access logs from at least one edge node in the order of reception to obtain a message queue. Multiple access logs in the message queue that are located after the first access log and contain the same domain name as the first access log are used to determine whether the access request corresponding to the first access log is a malicious request.

[0156] In a feasible implementation, the processing module 62 is used to generate an access log containing the first identifier when the access request does not carry the first identifier; when the access request carries the second identifier, generate an access log containing the second identifier, and the second identifier is generated by the first terminal device performing a target operation on the first identifier.

[0157] In a feasible implementation, the processing module 62 is further configured to allocate the first identifier to the first terminal device when the access request does not carry the first identifier;

[0158] The sending module 63 is further used to send an instruction message carrying the first identifier and the target operation to the first terminal device, so as to instruct the first terminal device to perform the target operation on the first identifier to obtain the second identifier and then send an access request carrying the second identifier.

[0159] In a feasible implementation, the receiving module 61 is further used to receive an interception instruction from a computing cluster after the sending module 63 sends the access log to the cache, the interception instruction carrying at least one of a client identifier, an IP address, a domain name or a URL to be intercepted;

[0160] The sending module 63 is further configured to intercept the access request initiated by the first terminal device according to the interception instruction.

[0161] In a feasible implementation, the processing module 62 is further used to monitor the zookeeper cluster to determine whether the node information on the zookeeper cluster is updated after the sending module 63 sends the access log to the cache, and the node information is used to indicate whether the configuration file on the file cluster is updated; if the node information update is monitored, the updated configuration file is obtained from the file cluster and the local configuration file is replaced; and access is intercepted according to the configuration file obtained from the file cluster.

[0162] The network crawler interception device provided in the embodiment of the present application can execute the actions of the edge node in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0163] Figure 7This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device 700 is, for example, the computing cluster or edge node mentioned above, and the electronic device 700 includes:

[0164] Processor 71 and memory 72;

[0165] The memory 72 stores computer instructions;

[0166] The processor 71 executes the computer instructions stored in the memory 72, so that the processor 71 executes the web crawler interception method implemented by the computing cluster as above; or, the processor 71 executes the web crawler interception method implemented by the edge node as above.

[0167] The specific implementation process of the processor 71 can be found in the above-mentioned method embodiment. Its implementation principle and technical effects are similar, and will not be repeated here in this embodiment.

[0168] Optionally, the electronic device 700 further includes a communication component 73 , wherein the processor 71 , the memory 72 and the communication component 73 may be connected via a bus 74 .

[0169] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, they are used to implement the network crawler interception method implemented by the computing cluster or edge node as above.

[0170] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the network crawler interception method implemented by the computing cluster or edge node as above.

[0171] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0172] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A network crawler interception method, characterized in that: Applied to computing clusters, including: Reading a first access log in a message queue, wherein the message queue is obtained by sorting access logs from at least two edge nodes in a receiving order by a cache; According to the domain name included in the first access log, read a plurality of access logs including the domain name from the message queue in sequence; Determining, according to the plurality of access logs, whether the access request corresponding to the first access log is a malicious request; The determining, according to the plurality of access logs, whether the access request corresponding to the first access log is a malicious request includes: When the first access log includes a first identifier, determining whether there is a second access log including a second identifier in the multiple access logs, wherein the first identifier is assigned by the edge node to the first terminal device when the access request does not carry a client identifier, and the second identifier is generated by the first terminal device performing a target operation on the first identifier; If the second access log does not exist in the multiple access logs, it is determined that the access request corresponding to the first access log is a malicious request.

2. The method according to claim 1, characterized in that The determining, according to the plurality of access logs, whether the access request corresponding to the first access log is a malicious request includes: extracting at least two features from each of the plurality of access logs; Based on at least two features extracted from each access log of the multiple access logs, determine whether the access request corresponding to the first access log is a malicious request, the at least two features include any two fields contained in the access request, and the fields contained in the access request include a client identifier, an IP address, a user agent UA, or a uniform resource locator URL.

3. The method according to claim 2, characterized in that The at least two features are a client identifier and an IP address, and determining whether an access request corresponding to the first access log is a malicious request based on the at least two features extracted from each access log of the plurality of access logs includes: When the client identifier included in the first access log is a second identifier, determining a second access log including the second identifier from the multiple access logs to obtain multiple second access logs; Determine the IP address included in each of the plurality of second access logs to obtain the number of different IP addresses; When the number of different IP addresses exceeds a preset number, it is determined that the access request corresponding to the first access log is a malicious request.

4. The method according to claim 2, characterized in that: The at least two features are an IP address and a URL, and determining whether an access request corresponding to the first access log is a malicious request based on the at least two features extracted from each access log of the plurality of access logs includes: According to the IP address included in the first access log, determining a third access log including the IP address from the plurality of access logs; When the number of the third access logs is greater than a preset threshold, extracting URLs in each of the third access logs; When the number of different URLs exceeds a preset number of URLs, it is determined that the access request corresponding to the first access log is a malicious request.

5. The method according to any one of claims 1 to 4, characterized in that: Also includes: According to the domain name contained in each access log in the memory, the access log in the memory is split to obtain multiple access log streams, each access log belonging to the same access log stream contains the same domain name, and the memory stores the access logs from the cache; At least two features contained in each access log in the target access log stream are analyzed to determine whether there are malicious requests in the access requests corresponding to the access logs contained in the target access log stream.

6. The method according to any one of claims 2 to 4, characterized in that: Before extracting at least two features from each of the plurality of access logs, the method further includes: Receiving a configuration request from a second terminal device, wherein the configuration request is used to request a configuration interface; Sending a data stream for displaying the configuration interface to the second terminal device; A configuration request is received, which is sent by the second terminal device through the configuration interface, and the configuration request is used to indicate the at least two features.

7. The method according to any one of claims 2 to 4, characterized in that: After determining, according to the plurality of access logs, whether the access request corresponding to the first access log is a malicious request, the method further includes: When the access request corresponding to the first access log is a malicious request, an interception instruction is sent to each edge node of the at least one edge node, and the interception instruction carries at least one of a client identifier, an IP address, a domain name or a URL to be intercepted.

8. The method according to any one of claims 2 to 4, characterized in that: After determining, according to the plurality of access logs, whether the access request corresponding to the first access log is a malicious request, the method further includes: Send a task configuration instruction to the management configuration center, where the task configuration instruction is used to instruct the management configuration center to generate a configuration file, upload it to the configuration file cluster, and trigger the zookeeper cluster to update the node information. The node information is used to indicate that the configuration file on the file cluster is updated. The configuration file is used to indicate at least one of the client identifier, IP address, domain name or URL to be intercepted.

9. A network crawler interception method, characterized in that: Applied to edge nodes, including: receiving an access request from a first terminal device; Generate an access log according to the access request; Sending the access log to a cache so that the cache sorts the access logs from at least one edge node in a receiving order to obtain a message queue, wherein multiple access logs in the message queue that are located after the first access log and contain the same domain name as the first access log are used to determine whether the access request corresponding to the first access log is a malicious request; Generating an access log according to the access request includes: When the access request does not carry the first identifier, generating an access log including the first identifier; When the access request carries a second identifier, an access log including the second identifier is generated, where the second identifier is generated when the first terminal device performs a target operation on the first identifier.

10. The method according to claim 9, characterized in that Also includes: When the access request does not carry the first identifier, assigning the first identifier to the first terminal device; An indication message carrying the first identifier and the target operation is sent to the first terminal device to instruct the first terminal device to perform the target operation on the first identifier to obtain the second identifier and then send an access request carrying the second identifier.

11. The method according to claim 9 or 10, characterized in that: After sending the access log to the cache, the method further includes: Receiving an interception instruction from a computing cluster, the interception instruction carrying at least one of a client identifier, an IP address, a domain name, or a URL to be intercepted; The access request initiated by the first terminal device is intercepted according to the interception instruction.

12. The method according to claim 9 or 10, characterized in that: After sending the access log to the cache, the method further includes: Monitor the zookeeper cluster to determine whether node information on the zookeeper cluster is updated, where the node information is used to indicate whether a configuration file on the file cluster is updated; If the node information is updated, the updated configuration file is obtained from the file cluster and the local configuration file is replaced; Access interception is performed according to the configuration file obtained from the file cluster.

13. A network crawler interception device, characterized in that: include: A reading module is used to read a first access log in a message queue, wherein the message queue is obtained by sorting the access logs from at least two edge nodes in a receiving order by a buffer; and read a plurality of access logs containing the domain name from the message queue in sequence according to the domain name contained in the first access log; A processing module, which determines, based on the multiple access logs, whether the access request corresponding to the first access log is a malicious request; When the processing module determines whether the access request corresponding to the first access log is a malicious request based on the multiple access logs, the processing module is used to determine whether there is a second access log containing a second identifier in the multiple access logs when the first access log contains a first identifier, the first identifier being assigned by the edge node to the first terminal device when the access request does not carry a client identifier, and the second identifier being generated when the first terminal device performs a target operation on the first identifier; if the second access log does not exist in the multiple access logs, it is determined that the access request corresponding to the first access log is a malicious request.

14. A network crawler interception device, characterized in that: include: A receiving module, used for receiving an access request from a first terminal device; A processing module, used to generate an access log according to the access request; a sending module, configured to send the access log to a cache, so that the cache sorts the access logs from at least one edge node in a receiving order to obtain a message queue, wherein multiple access logs in the message queue that are located after the first access log and contain the same domain name as the first access log are used to determine whether the access request corresponding to the first access log is a malicious request; When the processing module generates an access log based on the access request, it is used to generate an access log containing the first identifier when the access request does not carry the first identifier; when the access request carries the second identifier, it is used to generate an access log containing the second identifier, and the second identifier is generated by the first terminal device performing a target operation on the first identifier.

15. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the electronic device implements the method according to any one of claims 1 to 12.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images