Distributed crawler system architecture, data crawling method and computer equipment
A crawler system and distributed technology, applied in the field of data collection, can solve the problems of low development and maintenance efficiency of developers, poor stability and versatility of mining construction, etc., to achieve improved stability and expansion capabilities, reliable and efficient operation and maintenance management, and reduce The effect of capacity needs
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment Construction
[0039] In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.
[0040] refer to figure 1 , the present application first proposes a distributed crawler system architecture. The design of the architecture uses HTTP service registration to isolate different modules. Different modules use message queues to access each other. The architecture includes :
[0041] Task publishing module 10, for publishing crawler tasks;
[0042] The crawler service module 20 is used to store different crawler services in the form of services, and different crawler services complete different crawler tasks;
[0043] The crawler module 30 is used to receive...
PUM
Abstract
Description
Claims
Application Information
- R&D Engineer
- R&D Manager
- IP Professional
- Industry Leading Data Capabilities
- Powerful AI technology
- Patent DNA Extraction
Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.
© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com