Anti-crawler method and system based on operation environment feature identification

Abstract

The invention discloses an anti-crawler method and system based on operation environment feature identification, and relates to the field of anti-crawler technology, and addresses the technical problems of crawler program identification and anti-crawling strategy implementation. The method herein includes the following steps: generating a new strategy packet and an option code which is for operating the new strategy packet, using the new strategy packet to update the current strategy packet of a server and establishing a feature type white list about the operation feature data of the new strategy packet; transmitting the option code to a client, then transmitting a response request to the client; the client, based on the option code, optionally operating the strategy packet for performing responding by the server, acquiring the feature data corresponding to the option code and returning the feature data to the server; analyzing the option code and the feature data corresponding to the option code, computing the feature type of the client and determining whether the feature type is on the feature type white list, and performing access control on the client which is not on the feature type white list.

Description

technical field [0001] The invention relates to the technical field of reptile identification and anti-reptile, in particular to an anti-reptile method and system based on operating environment feature recognition. Background technique [0002] With the advent of the era of big data, data becomes more and more important. To analyze valuable data from data, a large amount of data is first required. The data on the Internet has attracted the attention of enterprises and individuals due to its openness, massiveness and extensiveness. Many web crawlers have been developed and utilized to collect various data on the Internet. However, web crawlers can have many negative effects. The web crawler program will request a large number of servers in a short period of time, resulting in a decrease in server performance; some crawlers will maliciously collect a large amount of public data, aggregate and sell it, and infringe copyright. There are also some websites whose data has high...

AI Technical Summary

Problems solved by technology

[0003] In view of the above-mentioned prior art, the purpose of the present invention is to provide an anti-crawler method and system based on operating environment feature recognition, so as to solve the problem of slow running speed of the server and the existence of information resources caused by the crawler implementers in the prior art constantly accessing the server and obtaining a large amount of information. Technical issues such as batch acquisition and theft

Images