Unlock instant, AI-driven research and patent intelligence for your innovation.

Crawler identification encryption string generation method and crawler identification method and device

An encrypted string and crawler technology, which is applied in the field of anti-crawler systems, can solve the problems that search engines cannot include website content, affect website promotion, and are prone to misjudgment.

Inactive Publication Date: 2018-08-21
GUANGDONG INTELL VISION TECH CO LTD
View PDF7 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, this method has the following disadvantages: 1) The user agent can be set to be simulated, so when the crawler is identified and judged only through the user agent, it is easy to have a misjudgment, such as judging a normal user as a crawler or a machine crawler. It is a normal user, etc.; 2) The method of discarding the search engine's machine crawler will cause the search engine to fail to include the content of the website, which will affect the promotion of the website

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Crawler identification encryption string generation method and crawler identification method and device
  • Crawler identification encryption string generation method and crawler identification method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0030] Below, the present invention will be further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, on the premise of not conflicting, the various embodiments described below or the technical features can be combined arbitrarily to form new embodiments. .

[0031] The present invention optimizes the method for identifying crawlers (a crawler is a program for automatically obtaining webpage content, which is an important part of a search engine), not only judges through user agents, but uses user agents (useregent), The cache (cookie) on the client side is combined to realize the crawler judgment on the access request, classify the judgment results, and then perform corresponding access resource allocation processing according to different access types, so as to ensure normal user access.

[0032] like figure 1 As shown, the crawler identification in the present invention includes two parts of judging and pr...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a crawler identification encryption string generation method. The method is applied to a server. The method comprises the following steps: a receiving step of receiving an encryption string returned by the server when an access request is the first-time access request, and storing the encryption string to a cache of a user side; and a sending step of reading the encryptionstring in the cache of the user side when the access request is not the first-time access request, and sending the encryption string to the server. The invention also provides a webpage crawler identification method and a computer readable memory medium. According to the methods and the device, the misjudgment problem for the crawler identification in the prior art can be solved.

Description

technical field [0001] The invention relates to an anti-reptile system, in particular to a method for generating an encrypted string for identifying a crawler, a method for identifying a crawler and a storage medium. Background technique [0002] At present, in the web system (including all websites and API interfaces), in the case of limited service resources, when there are a large number of web crawlers, a large amount of server resources will be consumed, which will affect normal user access. And existing anti-reptile system basically judges whether it is a crawler by the user-agent (user-agent, which refers to browser or search engine, etc.) in the Web request, and when it is considered to be a crawler, the access request of the crawler is discarded. However, this method has the following disadvantages: 1) The user agent can be set to be simulated, so when the crawler is identified and judged only through the user agent, it is easy to have a misjudgment, such as judging...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08H04L29/06G06F17/30
CPCH04L63/0428H04L63/0478H04L67/34G06F16/951H04L67/5683H04L67/60
Inventor 王新林
Owner GUANGDONG INTELL VISION TECH CO LTD