A low-frequency crawler identification method and device

Abstract

Disclosed herein are a low-frequency crawler identification method, a device, a readable storage medium and an equipment, the method comprising: computing a behavior feature vector of each user IP within a preset time slot according to a network application log of each user IP; clustering the behavior feature vector of each user IP to acquire a plurality of clusters; and determining an inspection rule, determining a cluster that meets the corresponding inspection rule, and determining each user IP in the cluster as a crawler. The embodiments of the present invention may effectively identify low frequency crawlers, and may solve group threats, low-frequency threats, associated threats and persistent threats, which traditional security products cannot identify. Public cloud or private cloud deployment is supported, and threat identification and blocking may be performed without changing network topology and without embedding any code; the joining of custom blocking interfaces is supported, and a deployment environment being completely switched off under extreme cases will not influence the normal operation of an original service.

Description

technical field [0001] The invention relates to the technical field of the Internet, in particular to a low-frequency crawler identification method and device. Background technique [0002] The Internet is filled with a large number of crawlers, and in the process of anti-crawlers, crawlers are also constantly evolving. The evolution process of crawlers includes the following three stages: primary crawlers, browser crawlers, and low-frequency crawlers. Among them, the primary crawler crawls the target page without disguising itself, and can be accurately identified by features such as user agent (User-agent), frequency, etc.; the browser crawler will use the User-agent it uses through Firefox, opera , chrome and other types of browsers are disguised, and their behavior will be similar to normal users. Browser crawlers can be identified by features such as access frequency and timeline; low-frequency crawlers use a large number of proxy IP pools to imitate ordinary users for...

AI Technical Summary

Problems solved by technology

[0004] (1) The identification recall rate is limited by the coverage rate of the proxy IP library. Currently, there are hundreds of millions of Internet proxy IPs, and the mobile phone proxy IP library can only cover a small part;

[0005] (2) The proxy IP is not static, so it is necessary to update the proxy IP library frequently. Customers generally have a resistance to online updates, and offline updates will face the problem of update delays;

[0006] (3) The proxy IP obtained by using ADSL community broadband disconnection replay and multicast is more concealed, and this IP will be used by many real users, and the proxy IP library will face problems such as false sealing and inability to accurately identify

Images