Distributed reptile cluster system

A cluster system and distributed technology, applied in special data processing applications, instruments, electrical and digital data processing, etc., can solve problems such as ineffective high-speed crawler systems, and achieve the effect of solving the problem of possession conflicts and improving the speed of crawling and grasping.

Inactive Publication Date: 2009-08-05
BEIJING UNIV OF POSTS & TELECOMM
View PDF0 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

At present, there is no systematic and effective high-speed crawler system

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed reptile cluster system
  • Distributed reptile cluster system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0018] Specific embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0019] figure 1 is a block diagram of a system according to an embodiment of the present invention. 101 represents a web crawler, 102 represents a hyperlink locker, 103 represents a webpage locker, and 104 represents a hyperlink giver.

[0020] The webpage crawler 101 is used for downloading webpages and processing webpages. Each web crawler 101 is independent of each other. At the same time, if only one web crawler is crawling web pages, it is obviously inefficient and cannot meet the real-time requirements. In order to improve the crawling speed, multiple webpage crawlers 101 are used to work simultaneously at the same time, thus greatly improving the crawling speed. The number of web crawlers 101 is usually determined by the ability of the hardware and the network environment conditions. A specific example is in figure 2 shown in . ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a distribution type reptile assembly system which includes components: a web page scrambing device, a hyperlink lock memorizer, a web page lock memorizer and a hyperlink giver. The system can solve problem of resource appropriation contradiction in the distribution type system efficiently; provide an universal design architecture for developing the distribution type reptile system; realize the reptile assembly system conveniently and rapidly and increase scrambing speed of the reptile with great extent.

Description

technical field [0001] The invention relates to a network data collection system, in particular to a distributed crawler cluster system. Background technique [0002] With the advent of the 21st century, information has grown explosively, and people are submerged in information garbage. In this situation, people propose and implement search engines in order to quickly extract useful information and improve work and study efficiency. As the basis of search engines and the only source of data processed by search engines, the status and importance of crawler systems are gradually highlighted. However, today's information updates are too fast, which requires increasing the crawling speed of crawlers to maintain a certain real-time search, and the speed of the current crawler system is far from meeting the needs of information updates. Therefore, improving the speed of the crawler system has become a focus of the current search field. At present, there is no systematic and effe...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 杨溥郭军徐蔚然
Owner BEIJING UNIV OF POSTS & TELECOMM
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap