A method and server for determining web crawler

Abstract

Embodiments of the invention provide a network crawler determination method and a server. The method comprises the steps of obtaining access information sent by a user terminal; according to the access information and a first preset rule, allocating a target user interface to the user terminal, wherein the target user interface comprises a modified specified tag; and if the user terminal continuesto access a specified access address after reading the modified specified tag, obtaining an access frequency of the access address, and according to the access frequency and a second preset rule, determining whether the user terminal is a network crawler or not. The server executes the method. According to the network crawler determination method and the server, the user terminal reads the modified specified tag; and according to the obtained frequency of continuing to access the specified access address by the user terminal, whether the user terminal is the network crawler or not can be accurately determined, so that the security of real data is protected.

Description

technical field [0001] The embodiment of the present invention relates to the technical field of network data security, in particular to a method and server for determining a web crawler. Background technique [0002] The current Internet is full of various web crawlers, stealing data from servers all the time. [0003] In the prior art, one method restricts the access rights of suspected web crawlers with more access times by obtaining the number of visits of web crawlers per unit time; the other method uses a certain algorithm to identify non-browsing requests, and then passes Restrict the access rights of user terminals that do not request browsing to ensure that data is not obtained illegally. But for the first method: the threshold of the number of visits is difficult to reasonably set, and sometimes it will affect the visits of normal users, for example: when the network condition is not good, multiple clicks by normal users will result in multiple visits per unit tim...

AI Technical Summary

Problems solved by technology

But for the first method: the threshold of the number of visits is difficult to set reasonably, and sometimes it will affect the visits of normal users.

For the second method: Due to the development of simulated browsing technology, such as: selenium, phantomjs, it also causes the problem that the identification of web crawlers is not accurate enough

Images