Intelligent anti-shielding web crawler system

A web crawler and intelligent technology, applied in the computer field, can solve problems such as not being able to use crawlers to obtain, and achieve the effect of intelligent collection

Inactive Publication Date: 2016-12-07
安徽天达网络科技有限公司
View PDF2 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Then, due to the commercial exclusive behavior of many websites, a crawler mechanism will be set up on the website, resulting in pages that users can normally access cannot be obtained by crawlers

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Intelligent anti-shielding web crawler system
  • Intelligent anti-shielding web crawler system
  • Intelligent anti-shielding web crawler system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0032] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0033] Depend on figure 1 As shown, the present invention provides an intelligent anti-shielding web crawler system, which includes an intelligent agent module 111, a user behavior simulation module 112, an information crawling module 113, an information sorting and storage module 114, and an information analysis unit 120 . A shielding rule base 115 , an agent information base 116 , a user account base 117 , and a user behavior rul...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an intelligent anti-shielding web crawler system. The intelligent anti-shielding web crawler system comprises an intelligent agent module, a user behavior analog module, an information crawling module, an information arranging and storing module, an information analysis unit, a shielding rule base, an agent information base, a user account base and a user behavior rule base. A method comprises the steps that the intelligent agent module judges whether IP agent is started or not in an active triggering mode; the user behavior analog module evades a shielding mechanism of a target website by simulating the Internet surfing behavior of people, and then the permission of having access to information of the target website is acquired. The intelligent anti-shielding web crawler system has the advantages that by means of the active triggered shielding judgment mechanism, the intelligent agent processing module and strict user behavior simulating, an anti-shielding effect is achieved, and all public information which can be visited by normal users can be collected.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to an intelligent anti-shielding web crawler system. Background technique [0002] A web crawler is a program for "automatically browsing the web" and an automatic retrieval tool that can automatically collect the content of all website pages it can access, and then store the accessed content for analysis. [0003] Then, due to the commercial exclusive behavior of many websites, a crawler mechanism will be set up on the website, so that the pages that users can normally access cannot be obtained by crawlers. Contents of the invention [0004] In view of the above problems, the present invention is proposed to provide an intelligent anti-screening web crawler system that overcomes the above problems or at least partially solves the above problems. [0005] According to one aspect of the present invention, an intelligent anti-shielding web crawler system is provided [0006] The...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): H04L29/08H04L12/24
CPCH04L41/145H04L67/56
Inventor 李让剑
Owner 安徽天达网络科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap