Method and system for parallel square crossing network data collection
A vertical cross, network data technology, applied in the Internet field, can solve the problems of network server blocking, server router paralysis, single page hierarchy, etc., to achieve the effect of reducing potential risks and costs, and being easy to implement and understand
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Example Embodiment
[0055] Figure 1A It is a flow chart of the parallel vertical cross network data acquisition method of the present invention. Such as Figure 1A As shown, the method includes:
[0056] S1. Determine the collection method according to the configuration parameters. If it is a common URL list collection method, download the web page data and its metadata directly according to the URL list. If it is a vertical cross collection method, use the cross keyword list to retrieve the page to be downloaded and download it The web page data and its metadata;
[0057] S2, a regular update task, scan and check all the web pages from the first layer to the current layer in the downloaded web page. When the last modification time of the newly downloaded page is greater than the last modification time of the downloaded page, the newly downloaded page Perform data collection and update web data records.
[0058] Further, in the S1:
[0059] When the webpage to be downloaded is a text file in doc or pd...
PUM
Abstract
Description
Claims
Application Information
- R&D Engineer
- R&D Manager
- IP Professional
- Industry Leading Data Capabilities
- Powerful AI technology
- Patent DNA Extraction
Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.
© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap