Crawler recognition and processing method and related device

Abstract

The embodiment of the invention provides a crawler recognition and processing method and a related device. The method includes the steps that webpage access record data is read, and feature fields in the access record data are counted according to user identifiers to generate a detection dataset; the generated detection dataset and a pre-marked training dataset are input into multiple classifiers for classification learning, and multiple classification results corresponding to first detection data in the detection dataset in the multiple classifiers are obtained; according to the number of classification results belonging to crawlers in the multiple classification results and the total number of the classification results, the classification probability that the first user identifier corresponding to the first detection data belongs to crawlers is determined. According to the method and the device, crawlers are recognized through learning with a lot of statistical data, and the adverse effect of real-time recognition on system throughput is avoided; besides, when crawlers are updated, the crawlers can be effectively recognized by adjusting the feature fields, and the expandability of crawler recognition is improved.

Description

technical field [0001] The embodiments of the present invention relate to the field of computer networks, in particular to a crawler identification and processing method and related devices. Background technique [0002] In related technologies, the identification of crawlers usually identifies and defends against real-time access of users and crawlers through whitelists and blacklists. For example, the access requests of all IPs are stored in real time through local memory or redis, memcache, etc., and each IP is accessed once, and the corresponding counter is incremented by 1. When the predetermined threshold is exceeded, the relevant IP will be restricted, for example, a verification code will be returned. If the verification fails many times, the IP will be identified as a crawler and added to the blacklist to prohibit access. [0003] In related technologies, the recognition of reptiles is convenient for real-time processing, and the recognition and processing of rept...

AI Technical Summary

Problems solved by technology

[0004] The embodiment of the present invention provides a crawler identification and processing method and related devices, which are used to solve the development and maintenance costs caused by the fact that the crawler identification and processing in the prior art are partial to real-time processing, which affects the overall throughput and the high degree of coupling between identification and processing. higher question

Images